1: \begin{abstract}%   <- trailing '%' for backward compatibility of .sty file

2:   This paper establishes the first almost sure convergence rate and the first maximal concentration bound with exponential tails for general contractive stochastic approximation algorithms with Markovian noise.

3:   As a corollary,

4:   we also obtain convergence rates in $L^p$.

5:   Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length.

6:   As applications,

7:   we provide the first almost sure convergence rate for $Q$-learning with Markovian samples without count-based learning rates.

8:   We also provide the first concentration bound for off-policy temporal difference learning with Markovian samples.

9:   \let\svthefootnote\thefootnote

10:   \let\thefootnote\relax\footnotetext{$^*$ indicates equal contribution.}

11:   \let\thefootnote\svthefootnote

12: \end{abstract}

13: