1: \begin{abstract}

2: 	In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster.

3: 	We consider the setting in which all workers sample from the same dataset,

4:  	and communicate over a sparse graph (decentralized).

5: 	In this setting, current theory fails to capture important aspects of real-world behavior.

6: 	First, the `spectral gap' of the communication graph is not predictive of its empirical performance in (deep) learning.

7: 	Second, current theory does not explain that collaboration enables \emph{larger} learning rates than training alone.

8: 	In fact, it prescribes \emph{smaller} learning rates, which further decrease as graphs become larger, failing to explain convergence in infinite graphs.

9: 	This paper aims to paint an accurate picture of sparsely-connected distributed optimization when workers share the same data distribution.

10: 	We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives.

11: 	Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies.

12: 	Code: \texttt{\href{https://github.com/epfml/topology-in-decentralized-learning}{github.com/epfml/topology-in-decentralized-learning}}

13: \end{abstract}

14: