1: \begin{abstract}

2: To efficiently train large-scale models, low-bit gradient communication compresses full-precision gradients on local GPU nodes into low-precision ones for higher gradient synchronization efficiency among GPU nodes. However, it often degrades training quality due to compression information loss. To address this, we propose the Low-bit Communication Adaptor (\method{}), which compensates gradients on local GPU nodes before compression, ensuring efficient synchronization without compromising training quality. Specifically, \method{} designs a moving average of historical compensation errors to stably estimate concurrent compression error and then adopts it to compensate for the concurrent gradient compression, yielding a less lossless compression. This mechanism allows it to be compatible with general optimizers like Adam and sharding strategies like FSDP. Theoretical analysis shows that integrating \method{} into full-precision optimizers like Adam and SGD does not impair their convergence speed on nonconvex problems. Experimental results  show that across large-scale model training frameworks like Megatron-LM and PyTorch's FSDP,  \method{} significantly improves communication efficiency, e.g., 

3: improving Adam's training speed by 14\% to 40\%  without performance degradation on large language models like LLAMAs and MoE.

4: \end{abstract}

5: