1: \begin{abstract}

2: In this paper we demonstrate a simple heuristic adaptive restart technique that can

3: dramatically improve the convergence rate of accelerated gradient schemes.  The

4: analysis of the technique relies on the observation that these schemes exhibit

5: two modes of behavior depending on how much momentum is applied. In what we

6: refer to as the `high momentum' regime the iterates generated by an accelerated

7: gradient scheme exhibit a periodic behavior, where the period is

8: proportional to the square root of the local condition number of the objective

9: function.  This suggests a restart technique whereby we reset the momentum

10: whenever

11: we observe periodic behavior. We provide analysis to show that in many cases

12: adaptively restarting allows us to recover the optimal rate of convergence

13: with no prior knowledge of function parameters.

14: \iffalse

15: We also demonstrate that functions that are well approximated by a quadratic

16: around their optima can enter a region of linear convergence, even if they

17: are not globally strongly convex.

18: It has been shown that periodically restarting an accelerated smooth first

19: order convex optimization algorithm can improve the convergence rate.  In this

20: paper we present two \emph{adaptive} restarting techniques for such

21: algorithms.  We refer to the techniques as adaptive because restarts are made

22: not at fixed intervals but when some condition on the iterates is satisfied.

23: We demonstrate that within a region near the optimum an adaptive restart

24: algorithm can achieve linear convergence, much better than that guaranteed by

25: the accelerated algorithms alone. In addition, the restart technique is able to

26: automatically exploit local function properties and therefore can 

27: sometimes obtain a better convergence rate than that predicted by global

28: function parameters.

29: \fi

30: \end{abstract}

31: