1: \begin{abstract}

2: We introduce TrustVI, a fast second-order algorithm for black-box variational

3: inference based on trust-region optimization and the ``reparameterization trick.''

4: At each iteration, TrustVI proposes and assesses a step based on

5: minibatches of draws from the variational distribution.

6: The algorithm provably converges to a stationary point.

7: %To prove convergence to a stationary point, we show that 1) in expectation,

8: %TrustVI's iterates improve the value of a surrogate objective function in

9: %proportion to the trust region size, and 2) the trust region radius cannot

10: %remain arbitrarily small indefinitely if the gradient of the objective at the

11: %iterates is bounded away from zero.

12: We implemented TrustVI in the Stan framework and compared it to two alternatives: Automatic Differentiation Variational Inference (ADVI) and

13: Hessian-free Stochastic Gradient Variational Inference (HFSGVI).

14: The former is based on stochastic first-order optimization.

15: The latter uses second-order information, but lacks convergence guarantees.

16: TrustVI typically converged at least one order of magnitude faster than ADVI,

17: demonstrating the value of stochastic second-order information.

18: TrustVI often found substantially better variational distributions than

19: HFSGVI, demonstrating that our convergence theory can matter in practice.

20: \end{abstract}

21: