1: \begin{abstract}

2: In this paper, we describe a phenomenon, which we named ``super-convergence'', where neural networks can be trained  an order of magnitude faster than with standard training methods. 

3: The existence of super-convergence is relevant to understanding why deep networks generalize well.

4: One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate.

5: A primary insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance.

6: %Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. 

7: We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.

8: Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures.

9: In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.

10: The architectures to replicate this work will be made available upon publication.

11: %The architectures and code to replicate the figures in this paper are  available at github.com/lnsmith54/super-convergence.

12: \end{abstract}

13: