1: \begin{abstract}

2: Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable.

3: Despite this, few researchers dare to train their models from scratch.

4: Most work builds on one of a handful of ImageNet pre-trained models, and fine-tunes or adapts these for specific tasks.

5: This is in large part due to the difficulty of properly initializing these networks from scratch.

6: A small miscalibration of the initial weights leads to vanishing or exploding gradients, as well as poor convergence properties.

7: In this work we present a fast and simple data-dependent initialization procedure, that sets the weights of a network such that all units in the network train at roughly the same rate, avoiding vanishing or exploding gradients.

8: Our initialization matches the current state-of-the-art unsupervised or self-supervised pre-training methods on standard computer vision tasks, such as image classification and object detection, while reducing the pre-training time by three orders of magnitude.

9: When combined with pre-training methods, our initialization significantly outperforms prior work, narrowing the gap between supervised and unsupervised pre-training.

10: \let\thefootnote\relax\footnote{Code available: \url{https://github.com/philkr/magic_init}}

11: %

12: 

13: %

14: %

15: %

16: \end{abstract}

17: