1: \begin{abstract}  

2: Restart techniques are common in gradient-free optimization to deal 

3: with multimodal functions. 

4: Partial warm restarts are also gaining popularity in gradient-based optimization 

5: to improve the rate of convergence in accelerated gradient schemes  

6: to deal with ill-conditioned functions.

7: In this paper, we propose a simple warm restart technique for stochastic gradient descent 

8: to improve its anytime performance when training deep neural networks. 

9: We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets,   

10: where we demonstrate new state-of-the-art results at 3.14\% and 16.21\%, respectively. 

11: We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset.  

12: Our source code is available at \\ \url{https://github.com/loshchil/SGDR}

13: \end{abstract}