1: \begin{abstract}

2: Recent work has shown that the \emph{prior} over functions induced by a deep Bayesian neural network (BNN) behaves as a Gaussian process (GP) as the width of all layers becomes large.

3: However, many BNN applications are concerned with the BNN function space \emph{posterior}.

4: While some \emph{empirical} evidence of the posterior convergence was provided in the original works of \citet{neal1996} and \citet{matthews2018gaussian}, it is limited to small datasets or architectures due to the notorious difficulty of obtaining and verifying exactness of BNN posterior approximations.

5: We provide the missing theoretical proof that the \emph{exact} BNN posterior converges (weakly) to the one induced by the GP limit of the prior.

6: For empirical validation, we show how to generate \emph{exact} samples from a finite BNN on a small dataset via rejection sampling.

7: % For empirical verification, we show how to generate \emph{exact} samples from the BNN posterior of finite width networks for small datasets.

8: \end{abstract}

9: