1: \begin{abstract}

2: We propose a new simple and natural algorithm for learning the optimal $Q$-value function of a discounted-cost  Markov Decision Process (MDP) when the transition kernels are unknown. Unlike the classical learning algorithms for MDPs, such as $Q$-learning and `actor-critic' algorithms, this algorithm doesn't depend on a stochastic approximation-based method. We  show that our algorithm, which we call the \textit{empirical $Q$-value iteration} (EQVI) algorithm, converges almost surely to the optimal $Q$-value function. To the best of our knowledge, this is the first algorithm for learning in MDPs that guarantees an almost sure convergence without using  stochastic approximations. We also give a rate of convergence or a non-aymptotic sample complexity bound, and also show that an asynchronous (or online) version of the algorithm will also work. Preliminary experimental results suggest a faster rate of convergence to a ball park estimate for our algorithm compared to stochastic approximation-based algorithms. In fact, the asynchronous setting EQVI vastly outperforms the popular and widely-used Q-learning algorithm.

3: 

4: \textbf{Keywords:} Dynamic programming; Empirical methods; Simulation; Stochastic approximations.

5: 

6: \textbf{AMS Subject Classification:} 49L20, 90C39, 37M05, 62C12, 47B80, 37H99

7: 

8: \textbf{History:} Submitted: \today

9: \end{abstract}

10: