1: \begin{abstract}

2: %Eligibility trace is an important mechanism to speed up the convergence of reinforcement learning.

3: %It has been shown that temporal-difference learning algorithms combining with eligibility traces can increase the efficiency and performance of estimation and control.

4: Recently, a new multi-step temporal learning algorithm, called $Q(\sigma)$, unifies $n$-step Tree-Backup (when $\sigma=0$) and $n$-step Sarsa (when $\sigma=1$) by introducing a sampling parameter $\sigma$. However, similar to other multi-step temporal-difference learning algorithms, $Q(\sigma)$ needs much memory consumption and computation time. 

5: Eligibility trace is an important mechanism to transform the off-line updates into efficient on-line ones which consume less memory and computation time.

6: In this paper, we further develop the original $Q(\sigma)$, combine it with eligibility traces and propose a new algorithm, called $Q(\sigma ,\lambda)$, in which $\lambda$ is trace-decay parameter. This idea unifies Sarsa$(\lambda)$ (when $\sigma =1$) and $Q^{\pi}(\lambda)$  (when $\sigma =0$). Furthermore, we give an upper error bound of $Q(\sigma ,\lambda)$ \emph{policy evaluation} algorithm. We prove that $Q(\sigma,\lambda)$ \emph{control} algorithm can converge to the optimal value function exponentially.

7: We also empirically compare it with conventional temporal-difference learning methods. Results show that, with an intermediate value of $\sigma$, $Q(\sigma ,\lambda)$ creates a mixture of the existing algorithms that can learn the optimal value significantly faster than the extreme end ($\sigma=0$, or $1$).

8: \end{abstract}

9: