1: \begin{abstract}

2: Large Language Models (LLMs) have become indispensable in numerous real-world applications. Unfortunately, fine-tuning these models at scale, especially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing methods often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typically comes at the cost of model accuracy. To address these limitations, we propose \textit{\fontfamily{qpl}\selectfont federated full-parameter tuning at scale for LLMs} (\ours{}), the first first-order method with shared randomness to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. \ours{} accomplishes this through three aspects: \textit{(1)} it employs widely applied first-order methods for efficient local updates; \textit{(2)} it projects these updates into a low-dimensional space to considerably reduce communication overhead; and \textit{(3)} it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that \ours{} significantly enhances the scalability of existing federated full-parameter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accuracy. Our implementation is available at \textcolor{red}{\url{https://github.com/allen4747/Ferret}}.

3: % This therefore makes \ours{} a highly desirable solution to deploy LLMs in large-scale federated environments.

4: 

5: \end{abstract}

6: