1: \begin{abstract}

2: We study the computational complexity of (deterministic or randomized) algorithms

3: based on point samples for approximating or integrating functions

4: that can be well approximated by neural networks.

5: Such algorithms (most prominently stochastic gradient descent and its variants)

6: are used extensively in the field of deep learning.

7: One of the most important problems in this field concerns the question of whether it is possible

8: to realize theoretically provable neural network approximation rates by such algorithms.

9: We answer this question in the negative by proving hardness results

10: for the problems of approximation and integration on a novel class

11: of neural network approximation spaces.

12: In particular, our results confirm a conjectured

13: and empirically observed theory-to-practice gap in deep learning.

14: We complement our hardness results by showing that approximation rates

15: of a comparable order of convergence are (at least theoretically) achievable.

16: \end{abstract}

17: