1: \begin{abstract}

2: We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance.  In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality.

3: This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data.  %Our analysis holds for general loss functions as well as non-isotropic covariance, and it illustrates the role of noise in the SGD dynamics.  

4: This framework allows us to obtain learning rate thresholds for stability of SGD as well as convergence guarantees.  

5: In addition to the deterministic equivalent, 

6: we introduce an SDE with a simplified diffusion coefficient (homogenized SGD)

7: %stochastic version of the limiting process (referred to as homogenized SGD), 

8: which allows us to analyze the dynamics of general statistics of SGD iterates.  

9: Finally, we illustrate this theory on some standard examples

10: and show numerical simulations which give an excellent match to the theory.

11: \end{abstract}

12: