1: \begin{abstract}

2: In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between \emph{consensus} and \emph{optimality}. 

3: In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the~\textit{incremental consensus}-based distributed SGD (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. 

4: Second, we propose the~\textit{generalized consensus}-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). 

5: We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. 

6: We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.

7: \end{abstract}

8: