1: \begin{abstract}

2:     \noindent \uclip{} is a simple amendment to gradient clipping

3:     that can be applied to any iterative gradient optimization

4:     algorithm. Like regular clipping, \uclip{} involves using

5:     gradients that are clipped to a prescribed size

6:     (e.g.~with component wise or norm based clipping) 

7:     but instead of discarding the clipped portion of the gradient,

8:     \uclip{} maintains a buffer of these values that is added to the gradients

9:     on the next iteration (before clipping).

10:     We show that the cumulative bias of the~\uclip{} updates is bounded by a constant.

11:     This implies that the clipped updates are unbiased \emph{on average}.

12:     Convergence follows via a lemma that guarantees

13:     convergence with updates $u_i$ as long as

14:     $\sum_{i=1}^t (u_i - g_i) = o(t)$ where $g_i$ are the gradients.

15:     Extensive experimental exploration is performed on CIFAR10 with further validation

16:     given on ImageNet.

17: \end{abstract}

18: