1: \begin{abstract}

2: % This document provides a basic paper template and submission guidelines.

3: % Abstracts must be a single paragraph, ideally between 4--6 sentences long.

4: % Gross violations will trigger corrections at the camera-ready phase.

5: 

6: Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training parameters is huge. In \citet{goldfarb2020practical}, practical quasi-Newton methods were proposed that approximate the Hessian of a multilayer perceptron (MLP) model by a layer-wise block diagonal matrix where each layer's block is further approximated by a Kronecker product corresponding to the structure of the Hessian restricted to that layer. Here, we extend these methods to enable them to be applied to convolutional neural networks (CNNs), by analyzing the Kronecker-factored structure of the Hessian matrix of convolutional layers. Several improvements to the methods in \citet{goldfarb2020practical} are also proposed that can be applied to both MLPs and CNNs. These new methods have memory requirements comparable to first-order methods and much less per-iteration time complexity than those in \citet{goldfarb2020practical}. Moreover, convergence results are proved for a variant under relatively mild conditions. Finally, we compared the performance of our new methods against several state-of-the-art (SOTA) methods on MLP autoencoder and CNN problems, and found that they outperformed the first-order SOTA methods and performed comparably to the second-order SOTA methods.

7: 

8: 

9: 

10: 

11: 

12: \end{abstract}

13: