1: \begin{abstract}

2: We develop two fundamental stochastic sketching techniques \textemdash \\ \textit{Penalty Sketching} (PS) and \textit{Augmented Lagrangian Sketching} (ALS) for solving consistent linear systems. The proposed PS and ALS techniques extend and generalize the scope of \textit{Sketch \& Project} (SP) method by introducing Lagrangian penalty sketches. In doing so, we recover SP methods as special cases and furthermore develop a family of new stochastic iterative methods. By varying sketch parameters in the proposed PS method, we recover novel stochastic methods such as \textit{Penalty Newton Descent}, \textit{Penalty Kaczmarz}, \textit{Penalty Stochastic Descent}, \textit{Penalty Coordinate Descent}, \textit{Penalty Gaussian Pursuit}, and \textit{Penalty Block Kaczmarz}. Furthermore, the proposed ALS method synthesizes a wide variety of new stochastic methods such as \textit{Augmented Newton Descent}, \textit{Augmented Kaczmarz}, \textit{Augmented Stochastic Descent}, \textit{Augmented Coordinate Descent}, \textit{Augmented Gaussian Pursuit}, and \textit{Augmented Block Kaczmarz} into one framework. Moreover, we show that the developed PS and ALS frameworks can be used to reformulate the original linear system into equivalent stochastic optimization problems namely \textemdash \textit{Penalty Stochastic Reformulation} and \textit{Augmented Stochastic Reformulation}. We prove global convergence rates for the PS and ALS methods as well as sub-linear $\mathcal{O}(\frac{1}{k})$ rates for the Cesaro average of iterates. The proposed convergence results hold for a wide family of distributions of random matrices, which provides the opportunity of fine-tuning the randomness of the method suitable for specific applications. Finally, we perform computational experiments that demonstrate the efficiency of our methods compared to the existing SP methods.

3: \end{abstract}

4: