1: \begin{abstract}

2: This work presents stochastic optimization methods targeted at least-squares problems involving Monte Carlo integration. While the most common approach to solving these problems is to apply stochastic gradient descent (SGD) or similar methods such as AdaGrad~\cite{adagrad} and Adam~\cite{adam}, which involve estimating a stochastic gradient from a small number of Monte Carlo samples computed at each iteration, we show that for this category of problems it is possible to achieve faster asymptotic convergence rates using an \textit{increasing} number of samples per iteration instead, a strategy we call \textit{increasing precision} (IP). We then improve pre-asymptotic convergence by introducing a \textit{hybrid approach} that combines the qualities of increasing precision and otherwise ``constant'' precision, resulting in methods such as the IP-SGD hybrid and IP-AdaGrad hybrid, essentially by modifying their gradient estimators to have an equivalent effect to increasing precision. Finally, we observe that, in some problems, incorporating a \textit{Gauss-Newton preconditioner} to the IP-SGD hybrid method can provide much better convergence than employing a Quasi-Newton approach or covariance-preconditioning as in AdaGrad or Adam.

3: \end{abstract}

4: