1: \begin{abstract}

2: The interpretation of numerical methods, such as finite difference methods for differential equations, as point estimators suggests that formal uncertainty quantification can also be performed in this context. 

3: Competing statistical paradigms can be considered and Bayesian probabilistic numerical methods (PNMs) are obtained when Bayesian statistical principles are deployed.

4: Bayesian PNM have the appealing property of being closed under composition, such that uncertainty due to different sources of discretisation in a numerical method can be jointly modelled and rigorously propagated.

5: Despite recent attention, no exact Bayesian PNM for the numerical solution of ordinary differential equations (ODEs) has been proposed.

6: This raises the fundamental question of whether exact Bayesian methods for (in general nonlinear) ODEs even exist.

7: The purpose of this paper is to provide a positive answer for a limited class of ODE.

8: To this end, we work at a foundational level, where a novel Bayesian PNM is proposed as a proof-of-concept. 

9: Our proposal is a synthesis of classical Lie group methods, to exploit underlying symmetries in the gradient field, and non-parametric regression in a transformed solution space for the ODE.

10: The procedure is presented in detail for first and second order ODEs and relies on a certain strong technical condition -- existence of a solvable Lie algebra -- being satisfied.

11: Numerical illustrations are provided.

12: \end{abstract}

13: