1: \begin{abstract}

2: Variational Bayes (VB) has become a widely-used tool

3: for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB 

4: algorithms is so far generally restricted to

5: the case where the variational parameter space is Euclidean,

6: which hinders the potential broad application of VB methods.

7: This paper extends the scope of VB to

8: the case where the variational parameter space is a Riemannian manifold.

9: We develop an efficient manifold-based VB algorithm

10: that exploits both the geometric structure of the constraint parameter space

11: and the information geometry of the manifold of VB approximating probability distributions.

12: Our algorithm is provably convergent and achieves a convergence rate of order $\mathcal O(1/\sqrt{T})$ and $\mathcal O(1/T^{2-2\epsilon})$ for a non-convex evidence lower bound function and a strongly retraction-convex evidence lower bound function, respectively. 

13: We develop in particular two manifold VB algorithms, Manifold Gaussian VB and Manifold Wishart VB, and demonstrate through numerical experiments that 

14: the proposed algorithms are stable, less sensitive to initialization and compares favourably to existing VB methods.\\

15: \newline

16: \noindent\textbf{Keywords.} Marginal likelihood, variational Bayes, natural gradient, stochastic approximation, Riemannian manifold.

17: \end{abstract}

18: