1: \begin{abstract}

2: 

3: Batch Normalization (\BN) is widely used in {centralized} deep learning to improve convergence and generalization.

4: %

5: However, in {federated} learning (FL) with decentralized data, 

6: prior work has observed that training with \BN could hinder performance and suggested replacing it with Group Normalization (\GN).

7: %

8: In this paper, we revisit this substitution by expanding the empirical study conducted in prior work. Surprisingly, we find that \BN outperforms \GN in many FL settings. The exceptions are high-frequency communication and extreme non-IID regimes. We reinvestigate

9: factors that are believed to cause this problem, including the mismatch of \BN statistics across clients and the deviation of gradients during local training. We empirically identify a simple practice that could reduce the impacts of these factors while maintaining the strength of \BN. Our approach, which we named \Ours, is fairly easy to implement, without any additional training or communication costs, and performs favorably across a wide range of FL settings. We hope that our study could serve as a valuable reference for future practical usage and theoretical analysis in FL.

10: 

11: 

12: 

13: \end{abstract}

14: