7560cb246ffe58c5.tex
1: \begin{abstract}
2: This paper studies an acceleration technique for incremental aggregated gradient 
3: ({\sf IAG}) method through the use of \emph{curvature} information for 
4: solving strongly convex finite sum optimization problems. 
5: These optimization problems of interest arise in large-scale learning applications. 
6: Our technique utilizes a curvature-aided gradient tracking
7: step to produce accurate gradient estimates incrementally using Hessian information. 
8: We propose and analyze two methods utilizing the new technique,
9: the curvature-aided IAG ({\sf CIAG}) method and the accelerated CIAG ({\sf A-CIAG}) method, 
10: which are analogous to gradient method and Nesterov's accelerated gradient method, respectively.
11: Setting $\kappa$ to be the condition number of the objective function, we prove the $R$ linear convergence rates of $1 - \frac{4c_0 \kappa}{(\kappa+1)^2}$ for the {\sf CIAG} method, and $1 - \sqrt{\frac{c_1}{2\kappa}}$ for the {\sf A-CIAG} method, where $c_0,c_1 \leq 1$ are constants inversely proportional to the distance between the initial point and the optimal solution. When the initial iterate is close to the optimal solution, the $R$ linear convergence rates match with the gradient and accelerated gradient method, albeit {\sf CIAG} and {\sf A-CIAG} operate in an incremental setting with strictly lower computation complexity.
12: Numerical experiments confirm our findings.
13: The source codes used for this paper can be found on \url{http://github.com/hoitowai/ciag/}.
14: \end{abstract}
15: