1: \begin{abstract}

2: Federated Learning (FL) is a promising technology that enables multiple participants to train a joint model collaboratively without sharing their local data. Owing to its privacy protection nature, FL has attracted interest from the industry, leading to its deployment across diverse domains such as smartphones, institutions, and the Internet of Things (IoTs). While multiple FL algorithms have been proposed to enhance FL performance from different perspectives, the evaluation method for FL algorithms is typically based on a single metric, like accuracy, failing to account for the unique demands of different use cases. Thus, how to comprehensively evaluate an FL algorithm and determine the most suitable candidate for a designated use case remains an open question. To mitigate this research gap, we introduce the Holistic Evaluation Metrics (HEM) in this work. Specifically, we identify the application scenarios of IoT, smartphones (smart devices), and institutions as the most represented FL use cases. We first identify the components of the evaluation metric, which encompass accuracy, convergence, computational efficiency, fairness, and personalization. Then, we determine the respective importance vector for each use case, considering each scenario has its distinct performance requirements and priorities. The HEM index is finally generated by integrating these metric components with their importance vectors. By evaluating various FL algorithms in three identified use cases, our experimental results demonstrate that HEM can effectively evaluate and select the appropriate FL algorithms tailored to specific use cases.

3: \end{abstract}

4: