1: \begin{abstract}

2: Fine-tuning has become a popular approach for adapting large deep learning models to specific tasks. As the size of models and datasets grows, parameter-efficient fine-tuning techniques are increasingly important. One of the most widely used methods is Low-Rank Adaptation (LoRA), where the adaptation update is expressed as the product of two low-rank matrices. While LoRA has shown strong performance in fine-tuning, it often underperforms compared to full-parameter fine-tuning. Although LoRA-based methods have been extensively studied empirically, their theoretical optimization analysis remains largely unexplored. In this work, we demonstrate that LoRA and its extensions, Asymmetric LoRA and Chain of LoRA, can encounter convergence issues. To address these challenges, we propose a general optimization framework that rigorously analyzes the convergence rates of LoRA-based methods. This framework also serves as an interpolation between full-parameter fine-tuning and low-rank adaptation. We provide provable guarantees of convergence to the same solution as full-parameter fine-tuning, along with the rate of convergence. Additionally, we present a convergence analysis for smooth, non-convex loss functions, covering gradient descent, stochastic gradient descent, and federated learning settings. Our theoretical findings are supported by experimental results.

3: 

4: \end{abstract}

5: