1: \begin{abstract}

2: In a recent work~\cite{Halverson:2021:NeuralNetworksQuantum}, Halverson, Maiti and Stoner proposed a description of neural networks in terms of a Wilsonian effective field theory.

3: The infinite-width limit is mapped to a free field theory while finite $N$ corrections are taken into account by interactions (non-Gaussian terms in the action).

4: In this paper, we study two related aspects of this correspondence.

5: First, we comment on the concepts of locality and power-counting in this context.

6: Indeed, these usual space-time notions may not hold for neural networks (since inputs can be arbitrary), however, the renormalization group provides natural notions of locality and scaling.

7: Moreover, we comment on several subtleties, for example, that data components may not have a permutation symmetry: in that case, we argue that random tensor field theories could provide a natural generalization.

8: Second, we improve the perturbative Wilsonian renormalization from~\cite{Halverson:2021:NeuralNetworksQuantum} by providing an analysis in terms of the non-perturbative renormalization group using the Wetterich-Morris equation.

9: An important difference with usual non-perturbative RG analysis is that only the effective (IR) 2-point function is known, which requires setting the problem with care.

10: Our aim is to provide a useful formalism to investigate neural networks behavior beyond the large-width limit (i.e.~far from Gaussian limit) in a non-perturbative fashion.

11: A major result of our analysis is that changing the standard deviation of the neural network weight distribution can be interpreted as a renormalization flow in the space of networks.

12: We focus on translations invariant kernels and provide preliminary numerical results.

13: \end{abstract}

14: