1: \begin{abstract}

2: $Q$-learning is one of the most fundamental reinforcement learning algorithms.

3: Previously,

4: it is widely believed that $Q$-learning with linear function approximation (i.e., linear $Q$-learning) suffers from possible divergence.

5: This paper instead establishes the first $L^2$ convergence rate of linear $Q$-learning to a bounded set.

6: Notably,

7: we do not make any modification to the original linear $Q$-learning algorithm, do not make any Bellman completeness assumption,

8: and do not make any near-optimality assumption on the behavior policy.

9: All we need is an $\epsilon$-softmax behavior policy with an adaptive temperature.

10: The key to our analysis is the general result of stochastic approximations under Markovian noise with fast-changing transition functions.

11: As a side product,

12: we also use this general result to establish the $L^2$ convergence rate of tabular $Q$-learning with an $\epsilon$-softmax behavior policy,

13: for which we rely on a novel pseudo-contraction property of the weighted Bellman optimality operator. 

14: \end{abstract}

15: