1: \begin{abstract}

2: Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks \citep{lu2020meanfield} that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only \textit{partial homogeneity} and \textit{local Lipschitz smoothness}. These new techniques are of independent interest.

3: \end{abstract}

4: