1: \begin{abstract}

2: \thispagestyle{empty}

3: {

4: Federated learning (FL) is a promising approach that enables distributed clients to collaboratively train a global model while preserving their data privacy.

5: However, FL often suffers from data heterogeneity problems, which can significantly affect its performance.

6: To address this, clustered federated learning (CFL) has been proposed to construct personalized models for different client clusters.

7: One effective client clustering strategy is to allow clients to choose their own local models from a model pool based on their performance.

8: However, without pre-trained model parameters, such a strategy is prone to clustering failure, in which all clients choose the same model.

9: Unfortunately, collecting a large amount of labeled data for pre-training can be costly and impractical in distributed environments.

10: To overcome this challenge, we leverage self-supervised contrastive learning to exploit unlabeled data for the pre-training of FL systems.

11: Together, self-supervised pre-training and client clustering can be crucial components for tackling the data heterogeneity issues of FL.

12: Leveraging these two crucial strategies, we propose contrastive pre-training--based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems.

13: In this work, we demonstrate the effectiveness of CP-CFL through extensive experiments in heterogeneous FL settings, and present various interesting observations.

14: }

15: 

16: \end{abstract}

17: