1: \begin{abstract}

2: The stochastic variational inference (SVI) paradigm, which combines variational inference, natural gradients, and stochastic updates, was recently proposed for large-scale data analysis in conjugate Bayesian models and demonstrated to be effective in several problems. 

3: This paper studies a family of Bayesian latent variable models with two levels of hidden variables but without any conjugacy requirements, making several contributions in this context. 

4: %

5: The first is observing that SVI, with an improved structured variational approximation, 

6: is applicable under more general conditions than previously thought with the only requirement being that the approximating variational distribution be in the same family as the prior. 

7: The resulting approach, \algfull{} (\alg), significantly extends the scope of SVI, enabling large-scale learning in non-conjugate models. 

8: %

9: %

10: %

11: %

12: %

13: %

14: %

15: %

16: For models with latent Gaussian variables we propose a hybrid algorithm, using both standard and natural gradients, which is shown to improve stability and convergence.

17: Applications in mixed effects models, sparse Gaussian processes, probabilistic matrix factorization and correlated topic models demonstrate the generality of the approach and the advantages of the proposed algorithms.

18: %

19: %

20: %

21: %

22: \end{abstract}

23: