1: \begin{abstract}

2: The classical AdaGrad method adapts the learning rate by dividing

3: by the square root of a sum of squared gradients. Because this sum

4: on the denominator is increasing, the method can only decrease step

5: sizes over time, and requires a learning rate scaling hyper-parameter to

6: be carefully tuned. To overcome this restriction, we introduce GradaGrad,

7: a method in the same family that naturally grows or shrinks the learning rate based on a different accumulation in the denominator, 

8: one that can both increase and decrease.

9: We show that it obeys a similar convergence rate as AdaGrad and demonstrate its non-monotone adaptation capability with experiments.

10: \end{abstract}

11: