1: \begin{abstract}

2: Learning rate is one of the most important hyper-parameters that has significant influence for neural network training. 

3: Learning rate schedules are widely used in real practice to adjust the learning rate according to pre-defined schedules for the fast convergence and good generalization.

4: However, existing learning rate schedules are all heuristic algorithms and lack theoretical support. 

5: Therefore, people usually choose the learning rate schedules through multiple ad-hoc trial, and the obtained learning rate schedules are sub-optimal. 

6: To boost the performance of the obtained sub-optimal learning rate schedule, we propose a generic learning rate schedule plugin, called \textbf{LEA}rning Rate \textbf{P}erturbation (LEAP), which can be applied to various learning rate schedules to improve the model training by introducing a certain perturbation to the learning rate.

7: We found that, with such simple yet effective strategy, training processing exponentially favors flat minima rather than sharp minima with guaranteed convergence, which leads to better generalization ability.  

8: In addition, we conduct extensive experiments which show that training with LEAP can improve the performance of various

9: deep learning models on diverse datasets using various learning rate schedules (including constant learning rate). 

10: \end{abstract}

11: