1: \begin{abstract}%   <- trailing '%' for backward compatibility of .sty file

2: We propose a learning framework based on stochastic Bregman iterations\revision{, also known as mirror descent,} to train sparse neural networks with an inverse scale space approach.

3: We derive a baseline algorithm called \emph{LinBreg}, an accelerated version using momentum, and \emph{AdaBreg}, which is a Bregmanized generalization of the \emph{Adam} algorithm.

4: In contrast to established methods for sparse training the proposed family of algorithms constitutes a regrowth strategy for neural networks that is solely optimization-based without additional heuristics. 

5: Our Bregman learning framework starts the training with very few initial parameters, successively adding only significant ones to obtain a sparse and expressive network.

6: The proposed approach is extremely easy and efficient, yet supported by the rich mathematical theory of inverse scale space methods.

7: We derive a statistically profound sparse parameter initialization strategy and provide a rigorous stochastic convergence analysis of the loss decay and additional convergence proofs in the convex regime.

8: Using only $3.4\%$ of the parameters of ResNet-18 we achieve $90.2\%$ test accuracy on CIFAR-10, compared to $93.6\%$ using the dense network.

9: Our algorithm also unveils an autoencoder architecture for a denoising task.

10: The proposed framework also has a huge potential for integrating sparse backpropagation and resource-friendly training.

11: \end{abstract}

12: