1: \begin{abstract}

2: We propose a novel limited-memory stochastic block BFGS update for incorporating enriched curvature information in stochastic approximation methods.

3: In our method, the estimate of the inverse Hessian matrix that is maintained by it, is updated at each iteration using a sketch of the Hessian, i.e., a randomly generated compressed form of the Hessian.

4:  We propose several sketching strategies, present a new quasi-Newton method that uses stochastic block BFGS updates combined with the variance reduction approach SVRG  to compute batch stochastic gradients, and prove linear convergence of the resulting method. Numerical tests on large-scale logistic regression problems  reveal that our method is more robust and substantially outperforms current state-of-the-art methods.

5: 

6: 

7: %, especially in the common regime with the number of features  not greatly exceeding the number of data points.

8: % as compared to standard SVRG and the variant proposed by Moritz, Nishihara and Jordan.

9: %

10: %is able to gradually learn the curvature of the objective function on the backend of first order method.

11: %

12: % In our setting the desired inverse matrix is the inverse Hessian of the underlying objective function. By sampling the action of the Hessian matrix, that is a few Hessian-vector products, and by maintaining an estimate of the previous inverse Hessian....

13: %

14: %

15: %through a learning model the inverse Hessian of the underlying objective function when only samples of the Hessians action, that is a few Hessian-vector products, are made available. This learning model is inspired on the recent stochastic sketch-and-project methods (Gower and Richt\'arik 2016)

16: %

17: %

18: %the new block BFGS methods allow for a varied amount of curvature information

19: %

20: %adaptively learn the curvature of the underlying objective function.

21: %

22: %The new methods arise from a sketch-and-project framework...

23: 

24: \end{abstract}

25: