1: \begin{abstract}

2: Training large neural networks is time consuming. To speed up the process, distributed training is often used. One of the largest bottlenecks in distributed training is communicating gradients across different nodes. Different gradient compression techniques have been proposed to alleviate the communication bottleneck, including top$_K$ gradient sparsification, which truncates the gradient to the largest $K$ components before sending it to other nodes. While some authors have investigated top$_K$ gradient sparsification in the parameter-server framework by applying top$_K$ compression in both the worker-to-server (uplink) and server-to-worker (downlink) direction, the currently accepted belief says that adding extra compression degrades the convergence of the model. We demonstrate, on the contrary, that adding downlink compression can potentially improve the performance of top$_K$ sparsification: not only does it reduce the amount of communication per step, but also, counter-intuitively, can improve the upper bound in the convergence analysis. To show this, we revisit non-convex convergence analysis of top$_K$ stochastic gradient descent (SGD) and extend it from the unidirectional to the bidirectional setting. We also remove a restriction of the previous analysis that requires unrealistically large values of $K$. We experimentally evaluate bidirectional top$_K$ SGD against unidirectional top$_K$ SGD and show that  models trained with bidirectional top$_K$ SGD will perform as well as models trained with unidirectional top$_K$ SGD while yielding significant communication benefits for large numbers of workers.

3: \end{abstract}

4: