1: \begin{abstract}

2: 

3: %[[\textbf{make sure new title is submitted to arXiv!}]]

4: 

5: %[[\textbf{make sure citing documents use new name}]]

6: 

7: %[[\textbf{TODO: move material in first big convergence proof to the appendix.  Since we don't actually need to use CALEs anymore to actually compute the required bounds, all of that discussion about them is less interesting}]]

8: 

9: 

10: Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent.  In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix used to compute the natural gradient direction can be viewed as an approximation of the Hessian.  This perspective turns out to have significant implications for how to design a practical and robust version of the method.  Among our various other contributions is a thorough analysis of the convergence speed of natural gradient descent and more general stochastic methods, a critical examination of the oft-used ``empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by the method, which we show still holds for certain other choices of the curvature matrix, but notably not the Hessian.

11: 

12: \end{abstract}

13: