1: \begin{abstract}

2: 	Evolution Strategies such as \texttt{CMA-ES} (covariance matrix adaptation evolution strategy) and \texttt{NES} (natural evolution strategy) have been widely 

3: 	used in machine learning applications, where an objective function is optimized without using its derivatives.   

4: 	However, the convergence behaviors of these algorithms have not been carefully studied. In particular, there is no rigorous analysis for the convergence of the estimated covariance matrix, and it is unclear how does the estimated covariance matrix help the converge of the algorithm. 

5: 	The relationship between Evolution Strategies and derivative free optimization algorithms is also not clear.

6: 	In this paper, we propose a new algorithm closely related to \texttt{NES}, which we call \texttt{MiNES} (mirror descent natural evolution strategy), for which we can establish rigorous convergence results.

7: 	We show that the estimated covariance matrix of \texttt{MiNES} converges to the inverse of Hessian  matrix of the objective function with a sublinear convergence rate.

8: 	Moreover, we show that some derivative free optimization algorithms are special cases of \texttt{MiNES}.

9: 	Our empirical studies demonstrate that \texttt{MiNES} is a query-efficient optimization algorithm competitive to classical algorithms including \texttt{NES} and \texttt{CMA-ES}.

10: \end{abstract}

11: