1: \begin{abstract}

2: In online continual learning, a neural network incrementally learns from a non-i.i.d. data stream.

3: % without training to convergence before observing new data.

4: % and networks are not trained to convergence before observing new training data.

5: Nearly all online continual learning methods employ experience replay to simultaneously prevent catastrophic forgetting and underfitting on past data.

6: % maximize the approximate likelihood over all observed data.

7: Our work demonstrates a limitation of this approach:

8: networks trained with experience replay tend to have unstable optimization trajectories, impeding their overall accuracy.

9: Surprisingly, these instabilities persist even when the replay buffer stores all previous training examples,

10: suggesting that this issue is orthogonal to catastrophic forgetting.

11: % However, less attention is placed on how to optimize this approximate likelihood such that the optimization process is stable.

12: We minimize these instabilities through a simple modification of the optimization geometry.

13: % In this work, we highlight the importance of experience replay objective optimization on the model's predictive performance and introduce Layerwise Proximal Replay (LPR) as a possible solution.

14: Our solution, Layerwise Proximal Replay (LPR), balances learning from new and replay data while only allowing for gradual changes in the hidden activation of past data.

15: We demonstrate that LPR consistently improves replay-based online continual learning methods across multiple problem settings, regardless of the amount of available replay memory.

16: \end{abstract}

17: