1: \begin{abstract}

2: \bigskip

3: \noindent

4: {\bf Abstract}

5: 

6: \medskip

7: \noindent

8: The goal of this survey is to present an explanatory review of the

9: approximation properties of deep neural networks. Specifically, we aim

10: at understanding how and why deep neural networks outperform other

11: classical linear and nonlinear approximation methods. This survey

12: consists of three chapters. 

13: In Chapter 1 we review the key ideas

14: and concepts underlying deep networks and their compositional

15: nonlinear structure. We formalize the neural network problem by

16: formulating it as an optimization problem when solving regression and

17: classification problems. We briefly discuss the stochastic gradient

18: descent algorithm and the back-propagation formulas used in solving

19: the optimization problem and address a few issues related to the

20: performance of neural networks, including the choice of activation

21: functions, cost functions, overfitting issues, and

22: regularization. 

23: In Chapter 2 we shift our focus to the

24: approximation theory of neural networks. We start with an introduction

25: to the concept of density in polynomial approximation and in

26: particular study the Stone-Weierstrass theorem for real-valued

27: continuous functions. Then, within the framework of linear

28: approximation, we review a few classical results on the density

29: and convergence rate of feedforward networks, followed by more recent

30: developments on the complexity of deep networks in approximating

31: Sobolev functions. 

32: In Chapter 3, utilizing nonlinear approximation theory,

33: we further elaborate on the power of depth and approximation superiority

34: of deep ReLU networks over other classical methods of nonlinear approximation.

35: 

36: 

37: \bigskip

38: %\end{abstract}

39: