1: \begin{abstract}

2: Interpretation of Deep Neural Networks (DNNs) training as an optimal control problem with

3: nonlinear dynamical systems has received considerable attention recently, yet the algorithmic development remains relatively limited.

4: In this work, we make an attempt along this line

5: by reformulating the training procedure from the trajectory optimization perspective.

6: We first show that most widely-used algorithms for training DNNs can be linked to the

7: Differential Dynamic Programming (DDP),

8: a celebrated second-order method

9: rooted in the Approximate Dynamic Programming.

10: In this vein, we propose a new class of optimizer, DDP Neural Optimizer (DDP-NOpt),

11: for training {feedforward} and convolution networks.

12: DDPNOpt features layer-wise feedback policies which improve convergence and

13: reduce sensitivity to hyper-parameter over existing methods.

14: It outperforms other optimal-control inspired training methods in both convergence and complexity,

15: and is competitive against state-of-the-art first and second order methods.

16: We also observe DDPNOpt has surprising benefit in preventing gradient vanishing.

17: Our work opens up new avenues for principled algorithmic design built upon the optimal control theory.

18: \end{abstract}

19: