1: \begin{abstract}

2: In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces.

3: The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program.

4: The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at  pre-specified grid points by solving convex optimization problems in each iteration.

5: We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions.

6: We also propose an \emph{interpolation-free} design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer.

7: When a nonlinear control-affine system is considered, 

8: the convex optimization approach provides an approximate  policy with a provable suboptimality bound. 

9: For general cases,

10:  the proposed convex formulation of dynamic programming operators can be  modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

11: \end{abstract}

12: