1: % 31.07.2006: New version: approximations etc. (Carlo)

2: % 24.08.2006: Changed introduction (Carlo)

3: % 26.08.2006: Abstract rewritten (Carlo)

4: % 31.08.2006: minor corrections (Carlo)

5: % 11.10.2006: added figures of Sergei (Carlo)

6: % 07.06.2007: changed title and introduction (CM)

7: % xx.06.2008: referees' corrections

8: 

9: \documentclass[a4paper,11pt]{article}

10: \usepackage{amsfonts,amsmath,amsthm}

11: \usepackage{graphicx}

12: %\usepackage{showkeys}

13: \usepackage{color}

14: 

15: \newcommand{\A}{\mathcal{A}}

16: \newcommand{\E}{\mathbb{E}}

17: \newcommand{\erre}{\mathbb{R}}

18: 

19: \newcommand{\cp}[2]{\langle#1,#2\rangle}

20: \newcommand{\ds}{\displaystyle}

21: \newcommand{\tr}{\mathop{\mathrm{Tr}}\nolimits}

22: 

23: \newtheorem{prop}{Proposition}[section]

24: \newtheorem{thm}[prop]{Theorem}

25: \newtheorem{coroll}[prop]{Corollary}

26: \newtheorem{lemma}[prop]{Lemma}

27: \theoremstyle{definition}

28: \newtheorem{rmk}[prop]{Remark}

29: 

30: \newsymbol\lesssim 132E

31: 

32: \newcommand{\carlo}[1]{\textcolor{red}{#1}}

33: 

34: \title{On controlled linear diffusions with delay in a model of

35:   optimal advertising under uncertainty with memory effects}

36: 

37: \author{Fausto Gozzi\thanks{Facolt\`a di Economia, Libera Universit\`a

38:     degli Studi Sociali ``Guido Carli'', 00198 Roma, Italy. e-mail:

39:     \texttt{fgozzi@luiss.it}.} \and Carlo Marinelli\thanks{Institut

40:     f\"ur Angewandte Mathematik, Universit\"at Bonn, Wegelerstr. 6,

41:     D-53115 Bonn, Germany. URL:

42:     \texttt{http://www.uni-bonn.de/$\sim$cm788}.} \and Sergei

43:   Savin\thanks{Graduate School of Business, Columbia University, New

44:     York, NY 10027, USA. e-mail: \texttt{svs30@columbia.edu}.}}

45: 

46: \date{\normalsize June 7, 2007. Revised July 21, 2008.}

47: 

48: \addtolength{\hoffset}{-1.5cm}

49: \addtolength{\textwidth}{3cm}

50: \addtolength{\voffset}{-1.75cm}

51: \addtolength{\textheight}{3.5cm}

52: 

53: \begin{document}

54: \maketitle

55: 

56: \begin{abstract}

57:   We consider a class of dynamic advertising problems under

58:   uncertainty in the presence of carryover and distributed forgetting

59:   effects, generalizing the classical model of Nerlove and Arrow

60:   \cite{NA}. In particular, we allow the dynamics of the product

61:   goodwill to depend on its past values, as well as previous

62:   advertising levels. Building on previous work (\cite{levico}), the

63:   optimal advertising model is formulated as an infinite dimensional

64:   stochastic control problem. We obtain (partial) regularity as well

65:   as approximation results for the corresponding value function. Under

66:   specific structural assumptions we study the effects of delays on

67:   the value function and optimal strategy. In the absence of carryover

68:   effects, since the value function and the optimal advertising policy

69:   can be characterized in terms of the solution of the associated HJB

70:   equation, we obtain sharper characterizations of the optimal policy.

71:   

72:   \medskip\par\noindent \emph{Keywords:} stochastic control problems

73:   with delay, dynamic programming, infinite dimensional Bellman

74:   equations, optimal advertising.

75: \end{abstract}

76: 

77: 

78: 

79: \section{Introduction}

80: This paper is devoted to the study of a class of optimal control

81: problems for linear stochastic differential equations with delay both

82: in the state and the control term, and is a natural continuation of

83: \cite{levico}. These problems arise in the theory of optimal

84: advertising under uncertainty with memory structures. We approach the

85: problem using stochastic control techniques in infinite dimensions.

86: 

87: In particular, in \cite{levico} we considered a controlled stochastic

88: differential equation (SDE) with delay entering both the state and the

89: control variable as an extension of the dynamic advertising model of

90: Nerlove and Arrow \cite{NA}. The results of \cite{levico} are the

91: following: we construct a controlled infinite dimensional SDE that is

92: equivalent to the controlled SDE with delay, we prove a verification

93: theorem, and we exhibit a simple example for which the Bellman

94: equation associated to the control problem admits a sufficiently

95: regular solution, hence the verification theorem can be applied. 

96: %

97: In the present manuscript, we extend \cite{levico} by

98:   developing several new sets of results.  On the one hand, we provide

99:   qualitative characterization of the first- and second-order

100:   properties of the optimal value function (Section

101:   \ref{sec:reformul}.1). In particular, we show that, under natural

102:   restrictions, the monotonicity of the optimal value function with

103:   respect to the initial goodwill profile still holds even in the

104:   presence of the state and control-related delay terms in the

105:   advertising dynamics (Proposition \ref{prop:Vincr}). In addition, we

106:   establish that the decreasing marginal influence of the attained

107:   goodwill levels on the primal profit components is retained by the

108:   optimal profit function (Proposition \ref{prop:Vconv}). As is well

109:   known, this last property is important in reducing the computational

110:   load required to solve the time- and space-discretized version of

111:   our problem by dynamic programming methods.

112: %

113:   On the other hand, in the view of intractability of the general

114:   variant of our problem, we propose approximation schemes for the

115:   optimal value function (Theorem \ref{prop:approx}) and for the

116:   optimal advertising policy (Propositions 3.11 and 3.12). The latter

117:   result is of particular importance since it suggests a

118:   computationally feasible approach to constructing asymptotically

119:   optimal advertising trajectories.

120: %

121:   In addition, we provide a complete characterization of the optimal

122:   advertising policy in the case when the cost function is quadratic

123:   and the reward function is linear in goodwill level (Section

124:   3.3). For a specific instance of this case we conduct a numerical

125:   study aimed at demonstrating the importance of proper accounting of

126:   the delay effects in calculating the optimal advertising policy.

127: 

128:   Finally, we are able to provide sharper characterization of the

129:   optimal policies in the case when the influence of advertising on

130:   the goodwill evolution is instantaneous and the delay effects are of

131:   the state-only type (Section \ref{sec:ex}). The key result in this

132:   section is Theorem \ref{thm:L2} which formulates sufficient

133:   conditions ensuring that the optimal advertising policy is of the

134:   feedback type. In particular, in the case of linear cost function

135:   the optimal control takes a particularly simple ``bang-bang'' form

136:   (Corollary \ref{cor:bang}).

137: 

138: % Using the setting of \cite{levico}, here we prove qualitative

139: % properties of the value function (monotonicity, concavity,

140: % regularity), and obtain approximation results for the value function

141: % and the optimal strategy.  In particular, the value function is

142: % characterized as the limit of a sequence of solutions of approximating

143: % ``regularized'' Bellman equations. The reason to look for such results

144: % is that the Bellman equation associated to the control problem cannot

145: % be solved directly, i.e. it is not covered by any of the available

146: % techniques (see \cite{DZ02}, \cite{FT1}, \cite{GolGoz}), with the

147: % possible exception of the viscosity approach, which nonetheless

148: % requires an ad hoc treatment and does not allow one to construct

149: % optimal strategies.  A study of the effects of delay with respect to a

150: % model without delay is in general not feasible without making specific

151: % structural assumptions. This is due to the fact that a controlled SDE

152: % with delay is essentially an infinite dimensional object, for which

153: % hardly anything can be computed explicitly. However, the special case

154: % of quadratic cost and linear reward, for which the Bellman equation

155: % admits an explicit solution, provides us with a realistic example

156: % where the effects of delay can be observed. Moreover, in the case of

157: % delay in the state only, we show that in some examples of interest one

158: % can give qualitative characterizations of the optimal strategy.

159: 

160: Optimal control problems for stochastic systems with delay in the

161: state term admit alternative, more traditional treatments: for

162: instance, see \cite{elsanosi} and \cite{larssen} for a more direct

163: application of the dynamic programming principle without appealing to

164: infinite dimensional analysis, and \cite{KSlibro} for the

165: linear-quadratic case. However, we would like to point out that none

166: of the methods just mentioned apply to the control of stochastic

167: differential equations with delay in the control term.

168: 

169: Analysis of advertising policies has always been occupying a

170: front-and-center place in the marketing research. The sheer size of

171: the advertising market (over \$143 billion in the US in 2005

172: \cite{admarket}) and the strong body of evidence of systematic

173: over-advertising by firms across many industries (see e.g.

174: \cite{overad1}, \cite{overad4}, \cite{overad2}, \cite{overad3})

175: has caused a renewal of attention to the proper accounting for the

176: so-called ``carryover'' or ``distributed lag'' advertising effects.

177: The term ``carryover'' designates an empirically observed

178: advertising feature under which the advertising influence on product

179: sales or goodwill level is not immediate, but rather is spread over

180: some period of time: according to a survey of recent empirical

181: ``carryover'' research by Leone \cite{leone}, delayed advertising effects

182: can last between 6 and 9 months in different settings.

183: 

184: On the theoretical front, pioneering work of \cite{VW} and \cite{NA}

185: has paved the way for the development of a number of models dealing

186: with the optimal distribution of advertising spending over time in

187: both monopolistic and competitive settings. A comprehensive review of

188: the state of the advertising control literature in \cite{sethi} points

189: out that the majority of these models operate under deterministic

190: assumptions and do not capture some of the most essential

191: characteristics of real-world advertising phenomena. On the empirical

192: side, one of the first and most important substreams of advertising

193: literature was formed by the papers focused on the studies of

194: distributed advertising lag (see e.g. \cite{bass2},

195: \cite{bassparsons}, \cite{griliches}). An important early empirical

196: result was obtained by Bass and Clark \cite{bassclark}, who

197: established that the initially adopted models with monotone decreasing

198: lags (see \cite{Koyck}) are often inferior in their explanatory power

199: to the models with more general lag distributions.

200: 

201: Despite the wide and growing empirical literature on the measurement

202: of carryover effects, there are practically no analytical studies that

203: incorporate distributed lag structure into the optimal advertising

204: modeling framework in the stochastic setting. The only papers dealing

205: with optimal dynamic advertising with distributed lags we are aware of

206: are \cite{bassparsons} (which provides a numerical solution to a

207: discrete-time deterministic example), and \cite{hartl}, \cite{HS}

208: (which applies a version of the maximum principle in the deterministic

209: setting). The creation of models which incorporate the treatment of

210: ``carryover'' effects in the stochastic settings have long been

211: advocated in the advertising modeling literature (see e.g.

212: \cite{hartl}, \cite{sethi} and references therein).

213: 

214: As mentioned above, in this work we study a class of stochastic models

215: deriving from that of Nerlove and Arrow \cite{NA}, incorporating both

216: the advertising lags as well as distributed ``churn''

217: (\cite{sethi-VWcomp}), or ``forgetting'', effects.  More precisely,

218: we formulate an optimization program that seeks to maximize the

219: goodwill level at a given time $T>0$ net of the cumulative cost of

220: advertising until $T$. This optimization problem is studied using

221: techniques of stochastic optimal control in infinite dimensions, using

222: the modeling approach of \cite{levico}: in particular, we specify the

223: goodwill dynamics in terms of a controlled stochastic delay

224: differential equation (SDDE), that can be rewritten as a stochastic

225: differential equation (without delay) in a suitable Hilbert space.

226: This allows us to associate to the original control problem for the

227: SDDE an equivalent infinite dimensional control problem for the

228: ``lifted'' stochastic equation.

229: 

230: The paper is organized as follows: in section \ref{sec:formul} we

231: formulate the optimal advertising problem as an optimal control

232: problem for an SDE with delay, and we recall the equivalence result of

233: \cite{levico}. In section \ref{sec:reformul} we prove the above

234: mentioned results about the value function and approximate strategies

235: in the general case, together with a detailed discussion of the effect

236: of delays in a specific situation. Section \ref{sec:ex} treats the

237: case of distributed forgetting in the absence of advertising

238: carryover.

239: 

240: Let us conclude this introduction fixing notation and recalling some

241: notions that will be needed. Given a lower semicontinuous convex

242: function $f:E\to\bar{\erre}:=\erre\cup\{+\infty\}$ on a Hilbert space

243: $E$ with inner product $\cp{\cdot}{\cdot}$, we denote its conjugate by

244: $f^*(y):=\sup_{x\in E}(\cp{x}{y}-f(x))$. Recall also that $D^-

245: f^*(y)=\arg\max_{x\in E} (\cp{x}{y}-f(x))$, where $D^-$ stands for the

246: subdifferential operator (see e.g. \cite{barbu-v}, p.~103).

247: %

248: Throughout the paper, $X$ will be the Hilbert space defined as

249: $$

250: X = \erre \times L^2([-r,0],\erre),

251: $$

252: with inner product

253: $$

254: \langle x,y\rangle = x_0y_0 + \int_{-r}^0 x_1(\xi)y_1(\xi)\,d\xi

255: $$

256: and norm

257: $$

258: |x| = \left( |x_0|^2 + \int_{-r}^0 |x_1(\xi)|^2\,d\xi\right)^{1/2},

259: $$

260: where $r>0$, $x_0$ and $x_1(\cdot)$ denote the $\erre$-valued and the

261: $L^2([-r,0],\erre)$-valued components, respectively, of the generic

262: element $x$ of $X$. Given $f:X\to\erre$, $k\in\{0,1\}$, we shall

263: denote by $\partial_k f$, $D^-_k f$, respectively, the partial

264: derivative and the subdifferential of $f$ with respect to the $k$-th

265: component.  We shall use mollifiers in a standard way: for $\zeta \in

266: C^\infty(\erre^d,\erre_+)$, equal to zero for $|x|>1$ and such that

267: $\int_{\erre^d}\zeta(x)\,dx=1$, we shall set

268: $\zeta_\lambda(x)=\lambda^{-d}\zeta(\lambda^{-1}x)$ for $\lambda\neq 0$.

269: %

270: By $a \lesssim b$ we mean that there exists a constant $N$ such that

271: $a \leq Nb$. If $N$ depends on some parameter of interest $p$, we

272: shall write $N(p)$ and $a \lesssim_p b$.

273: 

274: 

275: \section{The model}\label{sec:formul}

276: We consider a monopolistic firm preparing the market introduction of a

277: new product at some time $T$ in the future. In defining the state

278: descriptor for a firm to follow we use the Nerlove-Arrow framework and

279: consider the product's ``goodwill stock'' $y(t)$, $0\leq s \leq t \leq

280: T$. The firm directly influences the rate of advertising spending

281: $z(t)$ to induce the following trajectory for the goodwill stock:

282: \begin{equation}

283: \label{eq:SDDE}

284: \left\{\begin{array}{l}

285: dy(t) = \ds \left[a_0 y(t) + \int_{-r}^0 a_1(\xi)y(t+\xi)\,d\xi

286:         + b_0 z(t) +

287:             \int_{-r}^0b_1(\xi)z(t+\xi)\,d\xi\right]dt \\[10pt]

288: \ds \qquad\qquad  + \sigma\, dW_0(t), \quad s \leq t\leq T \\[10pt]

289: y(s)=x_0; \quad y(s+\xi)=x_1(\xi),\;

290: z(s+\xi)=\delta(\xi),\;\; \xi\in[-r,0],

291: \end{array}\right.

292: \end{equation}

293: where the Brownian motion $W_0$ is defined on a filtered probability

294: space $(\Omega,\mathcal{F},\mathbb{F}=(\mathcal{F}_t)_{t\geq 0},%

295: \mathbb{P})$, with $\mathbb{F}$ being the completion of the filtration

296: generated by $W_0$. We assume that the advertising spending rate

297: $z(t)$ is constrained to remain in the set $\mathcal{U}$, the space of

298: $\mathbb{F}$-adapted processes taking values in a compact

299:   interval $U\subseteq\erre_+$. In addition, we assume that the

300: following conditions hold:

301: \begin{itemize}

302: \item[(i)] $a_0 \leq 0$;

303: \item[(ii)] $a_1(\cdot) \in L^2([-r,0],\erre)$;

304: \item[(iii)] $b_0 \geq 0$;

305: \item[(iv)] $b_1(\cdot) \in L^2([-r,0],\erre_+)$;\label{iv}

306: \item[(v)] $x_0 \geq 0$;

307: \item[(vi)] $x_1(\cdot) \geq 0$, with $x_1(0)=x_0$;

308: \item[(vii)] $\delta(\cdot) \geq 0$.

309: \end{itemize}

310: %

311: Here $a_0$ and $a_1(\cdot)$ describe the process of goodwill

312: deterioration when the advertising stops, and $b_0$ and $b_1(\cdot)$

313: provide the characterization of the effect of the current and the past

314: advertising rates on the goodwill level. The values of $x_0$,

315: $x_1(\cdot)$ and $\delta(\cdot)$ reflect the ``initial'' goodwill and

316: advertising trajectories.  Note that we recover the model of Nerlove

317: and Arrow from (\ref{eq:SDDE}) in the deterministic setting

318: ($\sigma=0$) in the absence of delay effects

319: ($a_1(\cdot)=b_1(\cdot)=0$).

320: 

321: Setting $X \ni x:=(x_0,x_1(\cdot))$ and denoting by $y^{s,x,z}(t)$,

322: $t\in[0,T]$, a solution of (\ref{eq:SDDE}), we define the objective

323: functional

324: \begin{equation}

325: \label{eq:obj-orig}

326: J(s,x;z) =

327: \E\left[\varphi_0(y^{s,x,z}(T))-\int_s^T h_0(z(t))\,dt \right],

328: \end{equation}

329: where $\varphi_0:\erre\to\erre$ and $h_0:\erre_+\to\erre_+$ are

330: measurable utility and cost functions, respectively, satisfying the

331: conditions

332: \begin{equation}\label{eq:hphipol}

333: |\varphi_0(x)| \leq K(1+|x|)^m

334: \qquad \forall x\in\erre,

335: \end{equation}

336: and

337: \begin{equation}

338:   \label{eq:schip}

339:   |h(x)| \leq K \qquad \forall x \in U,

340: \end{equation}

341: for some $K>0$ and $m\geq 0$. In the sequel we shall often move the

342: superscripts $s,x,z$ to the expectation sign, with obvious meaning of

343: the notation.

344: %

345: Let us also define the value function $V$ for this problem as follows:

346: $$

347: V(s,x) = \sup_{z\in\mathcal{U}} J(s,x;z).

348: $$

349: We shall say that $z^*\in\mathcal{U}$ is an optimal strategy if it

350: is such that

351: $$

352: V(s,x)=J(s,x;z^*).

353: $$

354: %

355: The problems we will deal with are the maximization of the objective

356: functional $J$ over all admissible strategies $\mathcal{U}$, and the

357: characterization of the value function $V$ and of the optimal strategy

358: $z^*$.

359: 

360: Throughout the paper we will always assume that the assumptions of

361: this section hold true. In particular the constants $T$, $m$ and $K$

362: are fixed from now on.

363: 

364: 

365: \subsection{An equivalent infinite dimensional Markovian

366:   representation}

367: We shall recall a representation result (proposition \ref{prop:equiv}

368: below) proved in \cite{levico}, generalizing a corresponding

369: deterministic result due to Vinter and Kwong \cite{VK}.

370: 

371: \noindent

372: Let us define an operator $A:D(A)\subset X\to X$ as

373: follows:

374: \begin{eqnarray*}

375: A: (x_0,x_1(\xi)) &\mapsto& \Big(a_0x_0 + x_1(0),a_1(\xi)x_0

376:                            -{dx_1(\xi)\over d\xi}\Big) \quad%

377: \textrm{a.e.}\ \xi\in [-r,0], \\

378: D(A) &=& \left\{ x \in X:

379:        x_1 \in W^{1,2}([-r,0];\erre),\; x_1(-r)=0\right\}.

380: \end{eqnarray*}

381: Moreover, define the bounded linear control operator $B:U \to X$ as

382: \begin{equation}

383: B: u \mapsto \Big(b_0u,b_1(\cdot)u\Big),

384: \end{equation}

385: and finally the operator $G:\erre \to X$ as $G: x_0 \mapsto (\sigma

386: x_0, 0)$. Sometimes it will be useful to identify the operator $B$

387: with the element $(b_0,b_1)\in X$.

388: %

389: \begin{prop}\label{prop:equiv}

390:   Let $Y(\cdot)$ be the weak solution of the abstract evolution

391:   equation

392: \begin{equation}

393: \label{eq:abstract}

394: \left\{\begin{array}{l}

395: dY(t) = (A Y(t)+Bz(t))\,dt + G\,dW_0(t) \\[8pt]

396: Y(s) = \bar{x} \in X,

397: \end{array}\right.

398: \end{equation}

399: with arbitrary initial datum $\bar{x} \in X$ and control

400: $z\in\mathcal{U}$. Then, for $t\geq r$, one has,

401: $\mathbb{P}$-a.s.,

402: $$

403: Y(t) = M(Y_0(t),Y_0(t+\cdot),z(t+\cdot)),

404: $$

405: where

406: \begin{eqnarray*}

407: M: X \times L^2([-r,0],\erre) &\to& X \\

408: (x_0,x_1(\cdot),v(\cdot)) &\mapsto& (x_0,m(\cdot)),

409: \end{eqnarray*}

410: $$

411: m(\xi) := \int_{-r}^\xi a_1(\zeta) x_1(\zeta-\xi)\,d\zeta

412: + \int_{-r}^\xi b_1(\zeta) v(\zeta-\xi)\,d\zeta.

413: $$

414: %

415: Moreover, let $\{y(t),\; t\geq -r\}$ be a continuous solution of the

416: stochastic delay differential equation (\ref{eq:SDDE}), and $Y(\cdot)$

417: be the weak solution of the abstract evolution equation

418: (\ref{eq:abstract}) with initial condition

419: $$

420: \bar{x} = M(x_0,x_1,\delta(\cdot)).

421: $$

422: Then, for $t\geq 0$, one has, $\mathbb{P}$-a.s.,

423: $$

424: Y(t) = M(y(t),y(t+\cdot),z(t+\cdot)),

425: $$

426: hence $y(t)=Y_0(t)$, $\mathbb{P}$-a.s., for all $t\geq 0$.

427: \end{prop}

428: %

429: Using this equivalence result, we can now give a Markovian

430: reformulation on the Hilbert space $X$ of the problem of maximizing

431: (\ref{eq:obj-orig}), as in \cite{levico}.

432: In particular, denoting by $Y^{s,\bar{x},z}(\cdot)$ a mild solution of

433: (\ref{eq:abstract}), (\ref{eq:obj-orig}) is equivalent to

434: \begin{equation}

435: \label{ex1jbis}

436: J(s,x;z) =

437: \E\left[ \varphi(Y^{s,\bar{x},z}(T))+\int_s^T h(z(t))\,dt \right],

438: \end{equation}

439: with the functions $h:U\to\erre$ and $\varphi:X\to\erre$ defined by

440: \begin{eqnarray*}

441: h(z) &=& -h_0(z) \\

442: \varphi(x_0,x_1) &=& \varphi_0(x_0).

443: \end{eqnarray*}

444: Hence also $V(s,x)=\sup_{z\in \mathcal{U}} J(s,x;z)$.

445: 

446: 

447: \section{The case of delay in the state and the control term}

448: \label{sec:reformul}

449: The aims of this section are the following: to prove regularity

450: properties of the value function, to develop an approximation scheme

451: for the value function and the optimal strategy, and to illustrate in

452: a numerical example the effects of the delay structures in our model.

453: In particular, we prove that, under natural assumptions, the value

454: function is continuous in both arguments, and monotone concave with

455: respect to the initial goodwill profile. As already remarked, this

456: property is essential in order to obtain computationally tractable

457: discrete-time and discrete-state-space dynamic programming versions of

458: our problem.

459: 

460: Moreover, since we cannot guarantee that the Bellman equation

461: associated to our control problem admits a solution in general (nor do

462: we have any information about its uniqueness and regularity), it is of

463: primary interest to obtain approximation schemes for the optimal value

464: function and for the optimal advertising policy. The latter result is

465: of particular importance since it suggests a computationally feasible

466: approach to constructing asymptotically optimal advertising

467: trajectories.

468: 

469: In addition, in the last subsection we provide a complete

470: characterization of the optimal advertising policy in the case when

471: the cost function is quadratic and the reward function is linear in

472: goodwill level, and for a specific instance of this case we

473: conduct a numerical study aimed at demonstrating the importance of

474: proper accounting of the delay effects in calculating the optimal

475: advertising policy.

476: 

477: \subsection{Qualitative properties of the value function}

478: Let us first show that the value function is finite.

479: \begin{prop}\label{prop:Vfin}

480:   There exists a

481:   constant $N=N(T,m,K)$ such that $|V(s,x) |\leq N(1+|x|)^m$ for all

482:   $s\in[0,T]$, $x\in X$.

483: \end{prop}

484: \begin{proof}

485:   The estimate from below simply follows by taking a constant

486:   deterministic control. For the estimate from above we have,

487:   recalling that $h(x) \leq 0$ for all $x\in U$,

488:   \begin{eqnarray*}

489:     V(s,x) &\leq& \sup_{z\in\mathcal{U}} \E_{s,x}^z \Big[

490:     \int_s^T h(z(t))\,dt + |\varphi(Y(T))| \Big]\\

491:     &\leq& K \sup_{z\in\mathcal{U}} \E_{s,x}^z (1+|Y(T)|)^m.

492:   \end{eqnarray*}

493:   Moreover, we have

494:   \begin{align*}

495:     \E|Y(T)|^m &\lesssim |e^{(T-s)A}x|^m 

496:          + \E\bigg|\int_0^{T-s} e^{(T-s-r)A}Bz(r)\,dr\bigg|^m

497:     + \E\bigg|\int_0^{T-s} e^{(T-s-r)A}G\,dW(r)\bigg|^m\\

498:     &\lesssim_T M_T^m |x|^m + |B|^mM_T^m \bar{z}^m + \E|W_A(T)|^m,

499:   \end{align*}

500:   where $\bar{z}:=\max\{z:\,z\in U\}$, $W_A(t):=\int_0^{t-s}

501:   e^{(t-s-r)A}G\,dW(r)$, and

502:   $M_T:=\sup_{t\in[0,T]}|e^{tA}|$. Recalling that $GG^*$ is of trace

503:   class, hence

504:   \[

505:   \E|W_A(T)|^2 = \mathrm{Tr}\,\int_0^{T-s} e^{tA}GG^*e^{tA^*}\,dt <

506:   \infty

507:   \]

508:   (see e.g. \cite{DP-K}, Proposition 2.2), i.e. $W_A(T)$ is a

509:   well-defined Gaussian random variable on $X$, we also get that

510:   $\E|W_A(T)|^m<\infty$. The proof is completed observing that the

511:   upper bound on $\E|Y(T)|^m$ is uniform over $z$.

512: \end{proof}

513: %

514: %\begin{rmk}

515: %  Assuming that admissible strategies are Markovian of the type

516: %  $z(t)=f(t,Y(t))$ with $f(t,x)$ Lipschitz in $x$ uniformly over $t$

517: %  and $|f(t,x)|<K(1+|x|)$ with $K$ independent of $f$, we do not need

518: %  to assume that $U$ nor $h$ are bounded, and a polynomial growth

519: %. Here is a sketch of the argument: if we

520: %  write the state equation (\ref{eq:abstract}) in mild form,

521: %  \[

522: %  Y(t) = e^{tA}x + \int_0^t e^{(t-s)A}Bf(s,Y(s))\,ds + W_A(t),

523: %  \]

524: %  then we get, using elementary inequalities,

525: %  \[

526: %  \E\sup_{t\in[0,T]}|Y(t)|^m \leq K_1\Big( 1 + |x|^m + T

527: %  \E\sup_{t\in[0,T]}|Y(t)|^m + \E\sup_{t\in[0,T]}|W_A(t)|^m \Big),

528: %  \]

529: %and we obtain the required result for a sufficiently small $T$

530: %recalling that, e.g. by Theorem 2.9 of \cite{DP-K}, there exists a

531: %constant $C=C(m,T)$ such that $\E\sup_{t\in[0,T]} |W_A(t)|^m \leq C$.

532: %The result for general $T$ follows by considering the equation on

533: %partitions $[0,t_0]$, $[t_0,2t_0]$, etc. with $t_0$ small enough.

534: %\end{rmk}

535: 

536: 

537: We establish now some qualitative properties of the value function

538: that do not require studying an associated Bellman equation. The

539: following simple result, typical of control problems with linear

540: dynamics, asserts that the value function inherits the concavity with

541: respect to the space variable from the reward and cost functions.

542: %

543: \begin{prop}\label{prop:Vconv}

544:   If $\varphi$ and $h$ are concave,

545:   then the value function $V(s,x)$ is proper concave with respect to

546:   $x$.

547: \end{prop}

548: \begin{proof}

549: Properness follows by the previous proposition. Moreover, let $x^1$,

550: $x^2 \in X$. Then

551: \begin{eqnarray*}

552: \lambda V(s,x^1) + (1-\lambda)V(s,x^2) &=&

553: \lambda \sup_{z\in\mathcal{U}}

554: \E\Big[\int_s^T h(z(t))\,dt + \varphi(y^{s,x^1,z}(T))\Big] \\

555: && + (1-\lambda) \sup_{z\in\mathcal{U}}

556: \E\left[\int_s^T h(z(t))\,dt + \varphi(y^{s,x^2,z}(T))\right] \\

557: &=& \sup_{z^1,z^2\in\mathcal{U}}

558: \E\bigg[\int_s^T [\lambda h(z^1(t))+(1-\lambda)h(z^2(t))]\,dt \\

559: && \phantom{\sup_{z^1,z^2\in\mathcal{Z}}\E\bigg[}

560: +\lambda\varphi(y^{s,x^1,z^1}(T)) + (1-\lambda)\varphi(y^{s,x^2,z^2}(T))\bigg].

561: \end{eqnarray*}

562: Since $\mathcal{U}$ is a convex set and $h$ is concave, then

563: $z_\lambda:=\lambda z^1 + (1-\lambda) z^2$ is admissible for any

564: choice of $z^1$, $z^2\in\mathcal{U}$, and one has

565: \begin{equation}

566:   \label{eq:hc}

567:   h(z_\lambda(s)) \geq \lambda h(z^1(s))+(1-\lambda)h(z^2(s)).

568: \end{equation}

569: Moreover, by linearity

570: of the state equation, it is easy to prove that

571: $$

572: y^{s,x_\lambda,z_\lambda}(T) = \lambda y^{s,x^1,z^1}(T) +

573: (1-\lambda) y^{s,x^2,z^2}(T),

574: $$

575: hence, by the concavity of $\varphi$,

576: \begin{equation}

577:   \label{eq:phic}

578:   \varphi(y^{s,x_\lambda,z_\lambda}(T)) \geq

579:   \lambda \varphi(y^{s,x^1,z^1}(T)) +

580:   (1-\lambda) \varphi(y^{s,x^2,z^2}(T)).

581: \end{equation}

582: Therefore, as a consequence of (\ref{eq:hc}) and (\ref{eq:phic}), we obtain

583: \begin{eqnarray*}

584: \lambda V(s,x^1) + (1-\lambda)V(s,x^2)

585: &\leq&

586: \sup_{z_\lambda\in\mathcal{U}} \E\bigg[\int_s^T h(z_\lambda(t))\,dt +

587: \varphi(y^{s,x_\lambda,z_\lambda}(T)) \bigg] \\

588: &\leq&

589: \sup_{z\in\mathcal{U}} \E\bigg[\int_s^T h(z(t))\,dt +

590: \varphi(y^{s,x_\lambda,z}(T)) \bigg] \\

591: &=& V(s,\lambda x^1+(1-\lambda)x^2),

592: \end{eqnarray*}

593: which proves the claim.

594: \end{proof}

595: %

596: As a consequence of the previous propositions we obtain the following

597: regularity result. Of course it would be ideal to obtain a result

598: guaranteeing that $V \in C^{0,1}([0,T]\times X)$, so that a

599: verification theorem could be proved. Unfortunately we have not been

600: able to obtain such result. We shall prove though that $V$ is locally

601: Lipschitz continuous in the $X$-valued variable.

602: 

603: \begin{coroll}

604:   Under the hypotheses of Proposition

605:   \ref{prop:Vconv}, the value function $V(s,x)$ is locally Lipschitz continuous with

606:   respect to $x\in X$. Moreover, the subgradient $\partial V(s,x)$

607:   with respect to $x$ exists for all $x\in X$ and is locally bounded.

608: \end{coroll}

609: \begin{proof}

610:   The first assertion comes from the fact that a concave locally

611:   bounded function is continuous in the interior of its effective

612:   domain (see e.g. Theorem 2.1.3 in \cite{barbu-v}) and $V(s,x)$ is

613:   finite for all $x\in X$.  Corollary 2.4 in \cite{ET} and the fact

614:   that $D(\phi)^\circ \subset D(\partial \phi)$ for any concave

615:   function $\phi$, where $A^\circ$ denotes the interior of a set $A$,

616:   imply that $V(s,x)$ is locally Lipschitz in $x$, so the assertion on

617:   $\partial V$ follows.

618: \end{proof}

619: %

620: Since $V(s,x)\equiv V(s,x_0,x_1)$ is a concave function of $x_0$ for

621: fixed $s$ and $x_1$, one can also say that $V$ is twice differentiable

622: almost everywhere with respect to $x_0$, as it follows by the

623: Busemann-Feller theorem. A similar statement is not true regarding

624: differentiability with respect to $x_1$, as the Alexandrov theorem is in

625: general no longer true in infinite dimensions.

626: %

627: We now prove that the value function is continuous with respect to

628: the time variable. It is possible to prove local Lipschitz

629: continuity of $V(s,\cdot)$ without appealing to concavity, but

630: assuming local Lipschitz continuity of $\varphi$.

631: \begin{prop} The value function $V(s,x)$ is continuous in $s$. Moreover, if

632: $|\varphi_0(x)-\varphi_0(y)| \leq K(1+R)^m|x-y|$ for all

633:   $|x|$, $|y| \leq R$, then the function $V(s,x)$ is locally Lipschitz

634:   continuous with respect to $x$. Furthermore, there exists a constant

635:   $N=N(K,m)$ such that

636:   \begin{equation}

637:     \label{eq:Vlip}

638:   |V(s,x)-V(s,y)| \leq N(1+R)^m |x-y|

639:   \end{equation}

640:   for all $|x|$, $|y| \leq R$.

641: \end{prop}

642: \begin{proof}

643: Recalling that the difference of two suprema is less or equal to the

644: supremum of the difference, we have

645: \begin{eqnarray*}

646: |V(s,x) - V(s,y)| &\leq&

647: \sup_{z\in\mathcal{U}}|J(s,x;z)-J(s,y;z)|\\

648: &\leq& \sup_{z\in\mathcal{U}} \E|\varphi(Y^{z,s,x}) - \varphi(Y^{z,s,y})|,

649: \end{eqnarray*}

650: and, by Cauchy-Schwarz' inequality,

651: \begin{eqnarray*}

652: && \E|\varphi(Y^{z,s,x}(T)) - \varphi(Y^{z,s,y}(T))| \leq \\

653: && \qquad K\Big(\E(1 + |Y^{z,s,x}(T)|^m + |Y^{z,s,y}(T)|^m)^2\Big)^{1/2}

654:    \Big(\E|Y^{z,s,x}(T) - Y^{z,s,y}(T)|^2\Big)^{1/2}.

655: \end{eqnarray*}

656: Arguing as in the proof of Proposition \ref{prop:Vfin}, there exists a

657: constant $N_1=N_1(K,m)$ such that $\E|Y^{z,s,x}(T)|^m \leq N_1(1+|x|^m)$.

658: Furthermore, a simple calculation reveals that

659: $|Y^{z,s,x}(t)-Y^{z,s,y}(t)|=|e^{(t-s)A}(x-y)|$, hence

660: $$

661: \E|Y^{z,s,x}(T) - Y^{z,s,y}(T)|^2 \leq M_T^2 |x-y|^2,

662: $$

663: and the second claim is proved.

664: %

665: Let us now prove that $V(s,x)$ is continuous in $s$ for a fixed $x$.

666: Let $s_n \uparrow s$ be a given sequence (the case $s_n \downarrow s$

667: is completely similar).

668: Bellman's principle yields

669: \begin{equation}

670:   \label{eq:bp}

671: V(s_n,x) = \sup_{z\in\mathcal{U}} \E\left[

672: \int_{s_n}^s h(z(r))\,dr + V(s,Y^{z,s_n,x}(s)) \right],

673: \end{equation}

674: and choosing a $1/n$-optimal strategy $z_n$ in (\ref{eq:bp}),

675: %i.e. such

676: %that $|V(s_n,x)-J(s_n,x;z_n)|\leq 1/n$,

677: we have

678: \begin{equation}

679: \label{eq:tano}

680: \begin{array}{rcl}

681: \ds \limsup_{n\to\infty}

682: V(s_n,x)-V(s,x) &\leq&

683: \ds \limsup_{n\to\infty} \E\int_{s_n}^s h(z_n(r))\,dr\\[10pt]

684: && \ds + \limsup_{n\to\infty} \E|V(s,Y^{z_n,s_n,x}(s))-V(s,x)|.

685: \end{array}

686: \end{equation}

687: The first term on the right-hand side in (\ref{eq:tano}) is zero

688: because $h$ is bounded on $U$. Let us show the also the second term on

689: the right-hand side of (\ref{eq:tano}) is zero: in fact we have

690: \begin{eqnarray*}

691: \E|Y^{z_n,s_n,x}(s) - x|^2 &\leq& K \Big(

692: \E\int_{s_n}^s |e^{(s-r)A}h(z_n(r))|^2\,dr

693: + \E\Big|\int_0^{s-s_n} e^{(s-s_n-r)A}G\,dW(r)\Big|^2 \Big) \\

694: &\leq& K_1(s-s_n) + q_n \to 0

695: \end{eqnarray*}

696: as $n\to\infty$, hecause $h$ is bounded on $U$ and the stochastic

697: convolution is a Gaussian random variable with covariance operator

698: going to $0$ as $n\to\infty$. Therefore we also have $Y^{z_n,s_n,x}(s)

699: \to x$ in probability. By (\ref{eq:Vlip}), $V(s,x)$ is

700: continuous in $x$ uniformly with respect to $s$, hence

701: $$

702: V(s,Y^{z_n,s_n,x}(s)) \to V(s,x)

703: $$

704: in probability by the continuous mapping theorem. Moreover, recalling

705: that $|V(s,x)|\leq N(1+|x|^m)$ and $\E\sup_{t\in[0,T]}

706: |Y(t)|^m<\infty$, we have

707: \begin{equation}

708: \label{eq:porcon}

709:   \E|V(s,Y^{z_n,s_n,x}(s)) - V(s,x)|^m < \infty,

710: \end{equation}

711: hence Vitali's theorem implies

712: $$

713: \E|V(s,Y^{z_n,s_n,x}(s)) - V(s,x)| \to 0,

714: $$

715: hence $V(s_n,x)-V(s,x) \to 0$ as $n\to \infty$. Furthermore, taking

716: any $z_0 \in U$, we have from (\ref{eq:bp}),

717: $$

718: V(s,x)-V(s_n,x) \leq V(s,x) - (s_n-s)h(z_0) - \E V(s,Y^{z_0,s_n,x}),

719: $$

720: which goes to zero as $n \to \infty$ by (\ref{eq:porcon}), and the

721: claim is proved.

722: \end{proof}

723: %

724: \begin{rmk}

725:   Notice that, by the Rademacher theorem in infinite dimensions, the

726:   previous proposition implies that the value function $V(s,x)$ is

727:   differentiable in a dense subset of $X$. The local Lipschitz

728:   continuity of $V$ also implies that (Clarke's) generalized gradient

729:   of $V(s,x)$ with respect to $x$ is defined everywhere on $X$.

730: \end{rmk}

731: %

732: For the following proposition, which establishes a monotonicity

733: property of the value function, we need to define the natural ordering

734: in $X$: we shall write $x^1\geq x^2$ if $x^1_0 \geq x^2_0$ and $x^1_1

735: \geq x^2_1$ almost everywhere. Similarly, $x^1>x^2$ if the previous

736: inequalities hold with the strict inequality sign.

737: %

738: \begin{prop}\label{prop:Vincr}

739: If $a_1 \geq 0$ and $\varphi_0$ is increasing, then the value

740:   function $V(s,x)$ is increasing with respect to $x$ in the sense just defined.

741: \end{prop}

742: \begin{proof}

743:   The proof is completely analogous to that of proposition

744:   \ref{prop:V-incr} below if we prove that $A$ generates a positivity

745:   preserving semigroup. This is indeed the case: in fact, a direct

746:   calculation shows that $A$ is the adjoint of the operator $\A$

747:   defined in section \ref{sec:ex}. Since $a_1\geq 0$, $\A$

748:   generates a positivity preserving semigroup $S(t)$. It is well known

749:   that $A$ is the generator of the adjoint semigroup $S(t)^*$. Let

750:   $x$, $y$ be arbitrary positive elements of $X$. Then

751: $$

752: 0 \leq \cp{S(t)x}{y} = \cp{x}{S(t)^*y}.

753: $$

754: By the arbitrariness of $x$ and $y$, $S(t)^*$ is positivity preserving.

755: \end{proof}

756: 

757: 

758: \subsection{Approximating the value function and the optimal strategy}

759: Let us now consider the Bellman equation on $X$ associated to the problem of

760: maximizing (\ref{ex1jbis}), which can be written as

761: \begin{equation}

762: \label{eq:HJB0}

763: \left\{\begin{array}{l}

764: \ds v_t + {1\over 2}\tr (GG^*v_{xx}) + \cp{A x}{v_x} + H_0(v_x) = 0,

765: \quad 0 \leq t \leq T\\[8pt]

766: v(T) = \varphi,

767: \end{array}\right.

768: \end{equation}

769: where $H_0(p) = \sup_{z\in U} (\cp{Bz}{p}+h(z))$.

770: 

771: The main problem with (\ref{eq:HJB0}) is that it is not solvable with

772: any of the techniques currently available, with the possible exception

773: of the theory of viscosity solutions. In particular, as of now, one

774: cannot characterize the value function as the (unique) solution, in a

775: suitable sense, of equation (\ref{eq:HJB0}). 

776: %

777: As a consequence, we cannot obtain an optimal strategy for the

778: optimization problem at hand. As a (partial) remedy we develop a

779: method to approximate the value function and to construct suboptimal

780: feedback strategies that are asymptotically optimal, in the sense of

781: Proposition \ref{prop:asop} below.  Let us also briefly recall that,

782: if we know a priori that a smooth solution to the Bellman equation

783: exists, then we can apply the verification theorem proved in

784: \cite{levico}, which in turns allows to obtain precise

785: characterizations of the optimal strategy (see subsection

786: \ref{subsec:exex} and section \ref{sec:ex}).

787: 

788: \smallskip

789: 

790: Let us begin proving some approximation results for the value

791: function $V$. Let $\varepsilon\in]0,1]$ and define $G_\varepsilon:

792: \erre^2\to X$ as

793: $$

794: G_\varepsilon = \left[\begin{array}{cc}

795:                          \sigma_0 & 0 \\

796:                          0        & \varepsilon b_1

797:                       \end{array}

798:                 \right].

799: $$

800: Let $W_1$ be a standard real Wiener process independent of $W_0$, set

801: $W=(W_0,W_1)$, and denote by $\tilde{\mathbb{F}}$ the filtration

802: generated by $W$. Let $\tilde{\mathcal{U}}$ be the set of

803: $\tilde{\mathbb{F}}$-adapted processes taking values in $U$.

804: %

805: \par\noindent

806: %

807: Consider the following approximating SDE on $X$:

808: \begin{equation}

809: \label{eq:eps}

810: dY(t) = [A Y(t) + Bz(t)]\,dt + G_\varepsilon\,dW(t),

811: \quad Y(s)=\bar{x},

812: \end{equation}

813: where $z\in\tilde{\mathcal{U}}$ and $0 \leq s \leq t \leq T$.

814: \par\noindent

815: %

816: For a fixed $z$, let $Y$ and $Y_\varepsilon$ be, respectively,

817: solutions of (\ref{eq:abstract}) and of (\ref{eq:eps}).  Moreover, let

818: us define $\varphi_\varepsilon(x)=(\varphi_{0\varepsilon}(x),0)$,

819: $\varphi_{0\varepsilon}=\tilde{\varphi}_{0\varepsilon}\ast\zeta_\varepsilon$,

820: $\tilde{\varphi}_{0\varepsilon}(x) =

821: \varphi_0(x)\chi_{[-1/\varepsilon,1/\varepsilon]}(x)$, and

822: $h_\varepsilon(x)=(-h_{0\varepsilon}(x),0)$,

823: $h_{0\varepsilon}(x)=h_0\ast\zeta_\varepsilon(x)$. In particular

824: $\varphi_{0\varepsilon}\in C^2_c(\erre)$, $h_{0\varepsilon}\in

825: C^2(\erre)$.  Finally, the approximate objective function and value

826: function are defined as

827: $$

828: J_{\varepsilon_1,\varepsilon_2}(s,x;z) =

829: \E_{s,\bar{x}}^z\Big[\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))

830: + \int_s^T h_{\varepsilon_2}(z(t))\,dt \Big],

831: \qquad

832: V_{\varepsilon_1,\varepsilon_2}(s,x) =

833: \sup_{z\in\tilde{\mathcal{U}}} J_{\varepsilon_1,\varepsilon_2}(s,x;z).

834: $$

835: In the following we shall set

836: $\varepsilon=(\varepsilon_1,\varepsilon_2)$ and

837: $\lim_{\varepsilon\to 0}:= \lim_{\varepsilon_2\to

838: 0}\lim_{\varepsilon_1\to 0}$. Moreover, for $R>0$ we set

839: $C_R=\left\{x\in X: \;|x_0|\le R \right\}$.

840: %

841: \begin{thm}\label{prop:approx}

842:   One has $V_\varepsilon(s,x) \to V(s,x)$ as $\varepsilon \to 0$

843:   uniformly over $s\in[0,T]$, $x\in C_R$, for all $R>0$.

844: \end{thm}

845: \begin{proof}

846: Setting $\eta_{\varepsilon_1}(t)=Y_{\varepsilon_1}(t)-Y(t)$, one has

847: $$

848: \eta_{\varepsilon_1}(t) = \varepsilon_1 \int_s^t e^{(t-r)A}B_1\,dW_1(r)

849: = :\varepsilon_1 \eta(T),

850: $$

851: with $B_1:\erre\to X$, $B_1:x \to (0,b_1(\cdot)x)$,

852: and (suppressing the subscripts on the expectation sign for simplicity)

853: \begin{eqnarray}

854: |J_\varepsilon(s,x;z)-J(s,x;z)| &\leq&

855: |\E[\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T)) - \varphi(Y(T))]|\nonumber\\

856: && + \left|\E\Big[\int_s^T h_{\varepsilon_2}(z(t))\,dt

857:               - \int_s^T h(z(t))\,dt\Big]\right|\nonumber\\

858: &\leq& |\E[\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))

859:            - \varphi_{\varepsilon_2}(Y(T))]|

860: + |\E[\varphi_{\varepsilon_2}(Y(T)) - \varphi(Y(T))]|\nonumber\\

861: \label{eq:Je}

862: && + \left|\E\Big[\int_s^T h_{\varepsilon_2}(z(t))\,dt

863:               - \int_s^T h(z(t))\,dt\Big]\right|

864: \end{eqnarray}

865: Since

866: $$

867: \E|Y_{\varepsilon_1}(T)-Y(T)| =

868: \varepsilon_1 \E\Big|\int_s^t e^{(t-r)A}B_1\,dW_1(r)\Big| \to 0,

869: $$

870: then $Y_{\varepsilon_1}(T) \to Y(T)$ in probability uniformly over

871: $x\in X$ as $\varepsilon_1\to 0$, and by the continuous mapping

872: theorem $\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T)) \to

873: \varphi_{\varepsilon_2}(Y(T))$ in probability. Moreover

874: $\varphi_{\varepsilon_2}(Y(T)) \to \varphi(Y(T))$ in probability

875: uniformly over $x\in C_R$, for all $R>0$, as $\varepsilon_2\to 0$

876: because $\varphi_{0\varepsilon_2}(x)\to\varphi_0(x)$ $dx$-a.e. in

877: $\erre$.

878: %

879: Let us now prove that $\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))$ is

880: uniformly integrable with respect to $\varepsilon$. First let us

881: observe, as it is immediate to show, that there exists $\bar{K}$,

882: independent of $\varepsilon_2$, such that

883: $\varphi_{\varepsilon_2}(x)\leq\bar{K}(1+|x|)^m$.

884: Then we can write

885: \begin{eqnarray}

886: \sup_{\varepsilon\in]0,1]^2} \E|\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))| &\leq&

887: \sup_{\varepsilon_1\in]0,1]} \bar{K}\E(1+|Y(T)+\eta_{\varepsilon_1}(T)|)^m\nonumber \\

888: &\leq& K_1 + K_2 \sup_{\varepsilon_1\in]0,1]}

889:                  (\E|Y(T)|^m + {\varepsilon_1}^m\E|\eta(T)|^m)\nonumber\\

890: &=& \label{eq:ui1}

891:  K_1 + K_2 (\E|Y(T)|^m + \E|\eta(T)|^m) < \infty,

892: \end{eqnarray}

893: where we used twice the inequality $|x+y|^m\leq 2^m(|x|^m+|y|^m)$ and

894: Burkholder-Davis-Gundy's inequality.

895: Furthermore,

896: \begin{equation}

897: \label{eq:ui2}

898: \sup_{\varepsilon\in]0,1]^2} \E[|\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))|\chi_A]

899: \leq \E[(K_1 + K_2(|Y(T)|^m + |\eta(T)|^m))\chi_A] \to 0

900: \end{equation}

901: as $\mathbb{P}(A)\to 0$, because $K_1 + K_2(|Y(T)|^m + |\eta(T)|^m$

902: has finite expectation.  Then (\ref{eq:ui1}) and (\ref{eq:ui2}) imply

903: that $\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T))$ is uniformly

904: integrable (see e.g. \cite{kall}, lemma 3.10), hence

905: $$

906: | \E\varphi_{\varepsilon_2}(Y_{\varepsilon_1}(T)) - \E\varphi(Y(T)) | \to 0

907: $$

908: as $\varepsilon\to 0$ (see e.g. \cite{kall}, proposition 3.12).

909: %

910: \par\noindent

911: %

912: Similarly, since

913: $$

914: |h_{\varepsilon_2}(z(t,\omega)) - h(z(t,\omega))| \leq

915: K_1 + K_2 |z(t,\omega)|^m

916: $$

917: for all $t\in[0,T]$, $\omega\in\Omega$ and $\E\int_0^T |z(t)|^m <

918: \infty$ (because $U$ is compact), by the dominated convergence theorem

919: we have

920: $$

921: \E\int_0^T |h_{\varepsilon_2}(z(t)) - h(z(t)|\,dt \to 0

922: $$

923: as $\varepsilon_2 \to 0$.

924: 

925: In view of (\ref{eq:Je}) we have thus proved that

926: $|J_\varepsilon(s,x;z)-J(s,x;z)| \to 0$ uniformly over $s\in[0,T]$,

927: $x\in C_R$, for all $R>0$ and $z\in\tilde{\mathcal{U}}$, hence also

928: that $V_\varepsilon(s,x) \to V(s,x)$.

929: \end{proof}

930: %

931: %

932: If the cost function $h_0$ is continuous, one can use a different

933: regularization, without requiring compactness of $U$.

934: \begin{prop}

935:   If $h$ is continuous, then the assertion of theorem \ref{prop:approx}

936:   holds.

937: \end{prop}

938: \begin{proof}

939:   Let $h_{\varepsilon,\delta}$ be the sup-inf convolution of $h$ (in

940:   the sense of \cite{LasLio}), that is

941:   $$

942:   h_{\varepsilon,\delta}(x) = \sup_{z\in X}\inf_{y\in X}

943:      \Big(\frac{|z-y|^2}{2\varepsilon} - \frac{|z-x|^2}{2\delta} + h(y) \Big), \qquad 0<\delta<\varepsilon.

944:   $$

945:   It is known that $h_{\varepsilon,\delta}$ is differentiable with

946:   continuous derivative, that $\inf_{x \in X} h(x) \le

947:   h_{\varepsilon,\delta}(x) \le h(x)$ for all $x \in X$ and that

948:   $\lim_{\varepsilon,\delta \to 0^+}h_{\varepsilon,\delta}(x) \rightarrow h(x)$

949: uniformly over $x\in C_R$ (see \cite{LasLio}). Setting

950: $h_{\varepsilon_2}=h_{\varepsilon_2,\varepsilon_2/2}$, proposition

951: \ref{prop:Vfin} and the dominated convergence theorem yield

952:   $$

953:   \E\int_0^T h_{\varepsilon_2}(z(t))\,dt

954:   \stackrel{\varepsilon_2\downarrow 0}{\longrightarrow}

955:   \E\int_0^T h(z(t))\,dt,

956:   $$

957:   which implies that the third term on the right-hand side of

958:   (\ref{eq:Je}) converges to 0 as $\varepsilon \to 0$.

959: \end{proof}

960: %

961: %

962: Theorem \ref{prop:approx} (or its variant), together with the

963: following result, allow one to approximate the value function $V(s,x)$

964: in terms of the solutions of a sequence of Bellman equations.

965: \begin{prop}

966:   Assume that the hypotheses of theorem \ref{prop:approx} are

967:   verified. Assume moreover that $h_0$ is strictly convex and

968:   $\varphi_0$ is concave.  Then the approximate value function

969:   $V_\varepsilon$ is the unique mild solution (in the sense of

970:   \cite{FT05}) of the Bellman equation

971:   $$

972:   v_t + \frac12 \tr(G_{\varepsilon_1} G^*_{\varepsilon_1} v_{xx})

973:   + \cp{Ax}{v_x} + H_{0\varepsilon_2}(v_x) = 0,

974:   \qquad v(T)=\varphi_{\varepsilon_2},

975:   $$

976:   where $H_{0\varepsilon_2}(p)=\sup_{z\in U}(\cp{Bz}{p}+h_{\varepsilon_2}(z))$.

977: \end{prop}

978: \begin{proof}

979: Setting

980: $$

981: \tilde{B}_{\varepsilon_1}:\erre\to\erre^2, \qquad

982: \tilde{B}_{\varepsilon_1} = \left[\begin{array}{c} b_0/\sigma_0 \\

983: \varepsilon_1^{-1}

984:                               \end{array}

985:                         \right],

986: $$

987: the approximating equation (\ref{eq:eps}) can be rewritten as

988: \begin{equation}

989: \label{eq:epss}

990: dY(t) = [AY(t) + G_{\varepsilon_1} \tilde{B}_{\varepsilon_1} z(t)]\,dt

991: + G_{\varepsilon_1}\,dW(t),

992: \quad Y(s)=\bar{x}.

993: \end{equation}

994: The state equation (\ref{eq:epss}), hence also (\ref{eq:eps}), is

995: covered by the FBSDE approach to semilinear PDEs in Hilbert spaces

996: (see e.g. \cite{FT-corso}). In order to prove the statement, we

997: shall verify that hypothesis 7.1 in \cite{FT05} holds true. In

998: particular, $G_{\varepsilon_1}$ is Hilbert-Schmidt because $b_1\in

999: L^2([-r,0],\erre_+)$; $\varphi_{\varepsilon_2}$ is Lipschitz because

1000: $\varphi_{0\varepsilon_2} \in C^2_c(\erre)$;

1001: $|\tilde{B}_{\varepsilon_1} z|_X$ is bounded for $z\in U$ because

1002: $b_1\in L^2([-r,0],\erre)$ and $U$ is compact; finally, since $h_0$

1003: is proper and positive, it is immediate to find

1004: $\varepsilon_0\in]0,1]$, $C\geq 0$ such that $h_{\varepsilon_2}(x)

1005: \geq -C$ and $\inf_U h_{\varepsilon_2} \leq C$, for all positive

1006: $\varepsilon_2<\varepsilon_0$.

1007: 

1008: Since $h$ is strictly convex, for a sufficiently small

1009: $\varepsilon_2$ also $h_{\varepsilon_2}$ is strictly convex.

1010: Therefore we have

1011: $$

1012: g_{\varepsilon_2}(p)= \arg\max_{z\in

1013: U}(\cp{Bz}{p}+h_{\varepsilon_2}(z))=

1014: (h_{\varepsilon_2}')^{-1}(B^*p).

1015: $$

1016: The claim now follows from \cite{FT05}, theorem 7.2 provided we

1017: prove that the closed loop equation

1018: \begin{equation}

1019: \label{eq:CLE} dY(t) = [AY(t) + B g(v_x(t,Y(t))]\,dt +

1020: G_{\varepsilon_1}\,dW(t), \quad Y(s)=\bar{x}.

1021: \end{equation}

1022: admits a solution. In fact this follows as in Theorem 7.2 of

1023: \cite{FT1}.

1024: \end{proof}

1025: %

1026: \begin{rmk}

1027:   In fact the convexity of $U$ implies that $H_{0\varepsilon_2}\in

1028:   C^1(\erre)$, and hence that $V_\varepsilon \in C^{0,1}([0,T],X)$, as

1029:   in corollary \ref{cor:Vdiff} below.

1030: \end{rmk}

1031: 

1032: The above approximations do not give a way to construct approximately

1033: optimal strategies for the original problem. In fact, it is well known

1034: that the problem of constructing approximately optimal controls from

1035: the knowledge of an approximate value function is very hard, and in

1036: general unsolved. However, it is possible to construct a (suboptimal)

1037: feedback control for which we have some error control, in the sense

1038: defined below. For a map $f:[0,T]\times X \to U$ such that the

1039: equation

1040: $$

1041: dY(t) = AY(t)\,dt + Bf(t,Y(t))\,dt + G\,dW(t),\qquad Y(s)=x,

1042: $$

1043: admits a mild solution $Y(t)$, let us set $u_f(t)=f(t,Y(t))$ and

1044: $V^f(s,x)= J(s,x;u_f)$. Similarly we define $V_\varepsilon^f(s,x)$.

1045: Let us suppose that we can obtain a feedback law $f:[0,T]\times X

1046: \to U$, which is approximately optimal for the regularized problem,

1047: and let us write $V_\varepsilon \approx V_\varepsilon^f$ to mean

1048: that the two values differ by a small constant. Moreover, recall

1049: that $V \approx V_\varepsilon \approx V_\varepsilon^f$.

1050: \begin{prop}

1051:   Let $f(t,x)$ be Lipschitz in $x$ uniformly over $t$. Then

1052:   $V_\varepsilon^f(s,x) \to V^f(s,x)$ as $\varepsilon\to 0$.

1053: \end{prop}

1054: %

1055: \begin{proof}

1056: Denote by $Y^f$ and $Y^f_\varepsilon$, respectively, the

1057: solutions of the equations

1058: \begin{eqnarray*}

1059:   dY^f(t) &=& AY^f(t)\,dt + Bf(t,Y^f(t))\,dt + G\,dW(t) \\

1060:   dY^f_\varepsilon(t) &=& AY^f_\varepsilon(t)\,dt

1061:                           + Bf(t,Y^f_\varepsilon(t))\,dt

1062:                           + G_\varepsilon\,dW(t),

1063: \end{eqnarray*}

1064: with $Y^f(s)=Y^f_\varepsilon(s)=x$. Let us assume, without loss of

1065: generality, $s=0$.  Let us show that $Y^f_\varepsilon(t) \to Y^f(t)$

1066: in $L^1(\Omega,\mathbb{P})$, hence in probability, for all

1067: $t\in[0,T]$: by variation of constants we have

1068: \begin{eqnarray*}

1069: |Y^f_\varepsilon(t) - Y^f(t)| &\leq&

1070: \int_0^t |e^{(t-s)A}B(f(s,Y^f_\varepsilon(s))-f(s,Y^f(s)))|\,ds\\

1071: && + \varepsilon \Big|\int_0^t e^{(t-s)A}B_1\,dW_1(s)\Big|\\

1072: &\leq& \int_0^t m(s)|Y^f_\varepsilon(s) - Y^f(s)|\,ds

1073:        + \varepsilon \Big|\int_0^t e^{(t-s)A}B_1\,dW_1(s)\Big|,

1074: \end{eqnarray*}

1075: where $m(s)=|e^{(t-s)A}|\,|B|\,|f|_{\mathrm{Lip}}$. Taking expectation

1076: on both sides and recalling that the stochastic convolution has finite

1077: mean, Gronwall's lemma yields

1078: \begin{equation}

1079:   \label{eq:gro}

1080: \E|Y^f_\varepsilon(t) - Y^f(t)| \leq \varepsilon N e^{\int_0^T m(s)\,ds}

1081: \to 0

1082: \end{equation}

1083: as $\varepsilon \to 0$, hence $Y^f_\varepsilon(t) \to Y^f(t)$ in

1084: probability for all $t\in[0,T]$.

1085: %

1086: \par\noindent

1087: %

1088: By the same arguments used in the proof of theorem \ref{prop:approx}

1089: we obtain that

1090: \begin{equation}

1091:   \label{eq:lim1}

1092: \lim_{\varepsilon_2\to 0}\lim_{\varepsilon_1\to 0}

1093: \E\varphi_{\varepsilon_2}(Y^f_{\varepsilon_1}(T)) = \E\varphi(Y^f(T)).

1094: \end{equation}

1095: Similarly,

1096: \begin{eqnarray*}

1097: |h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))-h(f(t,Y^f(t)))|

1098: &\leq&

1099: |h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))

1100:     - h_{\varepsilon_2}(f(t,Y^f(t)))| \\

1101: && + |h_{\varepsilon_2}(f(t,Y^f(t)))-h(f(t,Y^f(t)))|

1102: \end{eqnarray*}

1103: and

1104: $$

1105: h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))

1106: \to h_{\varepsilon_2}(f(t,Y^f(t)))

1107: $$

1108: in probability as $\varepsilon_1\to 0$ for all $t\in[0,T]$, because

1109: $Y^f_{\varepsilon_1}(t) \to Y^f(t)$ in probability and

1110: $h_{\varepsilon_2}\circ f(t,\cdot)$ is continuous. Furthermore,

1111: $h_{\varepsilon_2}(f(t,Y^f(t))) \to h(f(t,Y^f(t)))$ $\mathbb{P}$-a.s.

1112: for all $t\in[0,T]$ as $\varepsilon_2\to 0$ because $h_{\varepsilon_2}

1113: \to h$ a.e. on $\erre$.  Since

1114: $|h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))| \leq \sup_{x\in

1115:   U}h_{\varepsilon_2}(x) < \infty$, then

1116: $h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))$ is uniformly

1117: integrable with respect to $\varepsilon_1$, hence

1118: \begin{equation}

1119:   \label{eq:lim2}

1120: \E\int_0^T \Big|

1121: h_{\varepsilon_2}(f(t,Y^f_{\varepsilon_1}(t)))                                 - h_{\varepsilon_2}(f(t,Y^f(t)))

1122: \Big|\,dt \to 0

1123: \end{equation}

1124: as $\varepsilon_1\to 0$.

1125: Finally,

1126: \begin{equation}

1127:   \label{eq:lim3}

1128: \E\int_0^T \Big|h_{\varepsilon_2}(f(t,Y^f(t)))\,dt

1129: - h(f(t,Y^f(t)))\Big|\,dt \to 0

1130: \end{equation}

1131: as $\varepsilon_2\to 0$ by the dominated convergence theorem, taking

1132: into account that

1133: $$

1134: \E \int_0^T |h_{\varepsilon_2}(f(t,Y^f(t)))-h(f(t,Y^f(t)))|\,dt

1135: < \infty,

1136: $$

1137: because $f$ is bounded, as follows by the compactness of $U$.  The

1138: claim now follows by (\ref{eq:lim1}), (\ref{eq:lim2}) and

1139: (\ref{eq:lim3}).

1140: \end{proof}

1141: %

1142: The previous proposition does not obviously allow one to say that

1143: $u(t)=f(t,Y(t))$ is an approximately optimal feedback map for the

1144: original problem, as $f$ itself in general depends on $\varepsilon_1$,

1145: $\varepsilon_2$.

1146: %

1147: The next proposition gives quantitative estimates on

1148: $|V^f(s,x)-V^f_\varepsilon(s,x)|$.

1149: %

1150: \begin{prop}     \label{prop:asop}

1151:   Assume that $f(t,x)$ is Lipschitz in $x$ uniformly over $t$,

1152:   and that $\varphi$, $h$ are Lipschitz continuous. Then there exist

1153:   constants $N=N(|f|_\mathrm{Lip})$ and $\delta=\delta(\varepsilon_2)$

1154:   such that

1155: $$

1156: |V^f(s,x)-V^f_\varepsilon(s,x)| \leq N\varepsilon_1 + \delta(\varepsilon_2)

1157: $$

1158: with $\lim_{\varepsilon_2\to 0} \delta(\varepsilon_2) = 0$.

1159: \end{prop}

1160: %

1161: \begin{proof}

1162: Let us write

1163: \begin{eqnarray*}

1164: |V^f-V^f_\varepsilon| &\leq&

1165: \E| \varphi_{\varepsilon_2}(Y^f_{\varepsilon_1}(T)) - \varphi(Y^f(T))| \\

1166: && + \E\int_0^T \Big|h_{\varepsilon_2}(f(t,Y^f(t)))\,dt                           - h(f(t,Y^f(t)))\Big|\,dt \\

1167: &=& I_1+I_2,

1168: \end{eqnarray*}

1169: and

1170: \begin{eqnarray*}

1171: I_1 &\leq& |\E[\varphi_{\varepsilon_2}(Y^f_{\varepsilon_1}(T))

1172:            - \varphi_{\varepsilon_2}(Y^f(T))]|

1173: + |\E[\varphi_{\varepsilon_2}(Y^f(T)) - \varphi(Y^f(T))]|\\

1174: &=& I_{11} + I_{12}.

1175: \end{eqnarray*}

1176: Then

1177: $$

1178: I_{11} \leq |\varphi_{\varepsilon_2}|_{\mathrm{Lip}}

1179: \E|Y^f_{\varepsilon_1}(T) - Y^f(T)| \leq

1180: |\varphi|_{\mathrm{Lip}} \, \varepsilon_1 e^{\int_0^T m(s)\,ds},

1181: $$

1182: where we used the fact that mollification does not increase the

1183: Lipschitz constant.

1184: We also have

1185: \begin{eqnarray*}

1186: I_{12} &\leq& \E\Big[

1187: |\varphi_{0\varepsilon_2}(Y_0^f(T)) - \varphi_0(Y_0^f(T))|;

1188: |Y_0^f(T)| \leq \varepsilon_2^{-1}

1189: \Big]\\

1190: && + \E\Big[

1191: |\varphi_{0\varepsilon_2}(Y_0^f(T)) - \varphi_0(Y_0^f(T))|;

1192: |Y_0^f(T)| > \varepsilon_2^{-1}

1193: \Big]\\

1194: &\leq& \delta_1(\varepsilon_2) + \delta_2(\varepsilon_2),

1195: \end{eqnarray*}

1196: where

1197: $$

1198: \delta_1(\varepsilon_2) = \sup_{|x|\leq1/\varepsilon_2}

1199: |\varphi_{0\varepsilon_2}(x) - \varphi_0(x)| < \infty,

1200: $$

1201: as $\varphi_{0\varepsilon_2}$ converges to $\varphi_0$ uniformly

1202: on compact sets, and $\delta_2(\varepsilon_2)$ is defined as follows:

1203: there exist $K_1$, $K_2\geq 0$ such that

1204: $$

1205: \E\Big[

1206: |\varphi_{0\varepsilon_2}(Y_0^f(T)) - \varphi_0(Y_0^f(T))|;

1207: |Y_0^f(T)| > \varepsilon_2^{-1}

1208: \Big]

1209: \leq

1210: \E\Big[ K_1+K_2|Y_0^f(T)|^m; |Y_0^f(T)| > \varepsilon_2^{-1} \Big],

1211: $$

1212: and

1213: \begin{eqnarray*}

1214: Y^f(T) &\leq& e^{TA}x

1215: + \int_0^T e^{(T-t)A}Bf(t,Y^f(t))\,dt

1216: +  \int_0^T e^{(T-t)A}G\,dW(t) \\

1217: &\leq& e^{TA}x

1218: + \int_0^T e^{(T-t)A}R\,dt

1219: + \int_0^T e^{(T-t)A}G\,dW(t) =: \mu_2 + Z_1.

1220: \end{eqnarray*}

1221: Similarly,

1222: $$

1223: Y^f(T) \geq e^{TA}x

1224: + \int_0^T e^{(T-t)A}r\,dt

1225: + \int_0^T e^{(T-t)A}G\,dW(t) =: \mu_1 + Z_1,

1226: $$

1227: where $\mu_1$, $\mu_2\in X$, $U \subseteq [r,R]$, and

1228: $Z_1$ is a centered $X$-valued Gaussian random variable.

1229: Denoting by $Z$ the $\erre$-valued components of $Z_1$, we have

1230: $$

1231: Y^f_0(T) - Z \leq (\mu_2)_0,

1232: \qquad

1233: Y^f_0(T) - Z \geq (\mu_1)_0,

1234: $$

1235: hence $|Y^f_0(T) - Z| \leq |(\mu_1)_0| \vee |(\mu_2)_0|=:\mu$, or

1236: equivalently $|Y_0^f(T)| \leq \mu + |Z|$.  In particular, $Z$ is a

1237: centered Gaussian random variable. Then

1238: \begin{eqnarray*}

1239: \E\Big[ K_1+K_2|Y_0^f(T)|^m; |Y_0^f(T)| > \varepsilon_2^{-1} \Big]

1240: &\leq& K_1 \mathbb{P}(|Z|+\mu > \varepsilon_2^{-1}) \\

1241: && + K_2\E\Big[ (|Z|+\mu)^m; |Z|+\mu > \varepsilon_2^{-1} \Big]\\

1242: &=:& \delta_2(\varepsilon_2).

1243: \end{eqnarray*}

1244: Note that $\delta_2(\varepsilon_2) \to 0$ as $\varepsilon_2\to 0$

1245: since $\E(|Z|+\mu)^m < \infty$.

1246: 

1247: We have

1248: \begin{eqnarray*}

1249: I_2 &\leq& \Big|

1250: h_{\varepsilon_2}(f(\cdot,Y_{\varepsilon_1}))

1251: - h_{\varepsilon_2}(f(\cdot,Y))

1252: \Big|_{L^1_T}

1253: + \Big|

1254: h_{\varepsilon_2}(f(\cdot,Y))

1255: - h(f(\cdot,Y))

1256: \Big|_{L^1_T}\\

1257: &=& I_{21}+I_{22},

1258: \end{eqnarray*}

1259: where $L^1_T$ stands for $L^1(\Omega\times[0,T],d\mathbb{P}\times

1260: dt)$.  Recalling again that mollification does not increase the

1261: Lipschitz constant, we also have

1262: $$

1263: I_{21} \leq |h|_{\mathrm{Lip}} |f|_{\mathrm{Lip}}

1264: |Y_{\varepsilon_1} - Y|_{L^1_T}

1265: \leq |h|_{\mathrm{Lip}} |f|_{\mathrm{Lip}}

1266: \varepsilon_1 N \int_0^T e^{\int_0^t m(s)\,ds}\,dt.

1267: $$

1268: Finally, using again the uniform convergence on compact sets of

1269: mollified continuous functions,

1270: \[

1271: I_{22} \leq T \delta_3(\varepsilon_2).

1272: \qedhere

1273: \]

1274: \end{proof}

1275: 

1276: 

1277: 

1278: \subsection{An example with explicit solutions}     \label{subsec:exex}

1279: In this subsection we study in detail the optimal advertising problem

1280: with linear reward and quadratic cost. In particular, we shall assume

1281: $h(z)=-\beta z_0^2$ and $\varphi(x) = \gamma x_0$, with $\beta$,

1282: $\gamma>0$. In \cite{levico} we proved that a solution (in integral

1283: sense) of the HJB equation (\ref{eq:HJB0}) is of the type

1284: $$

1285: v(t,x) = \cp{w(t)}{x} + c(t), \qquad t \in [0,T], \; x \in X,

1286: $$

1287: where $w=(w_0,w_1):[0,T]\to X$ and $c:[0,T]\to\erre$ are given by

1288: \begin{equation}\label{eq:HJBf3}

1289: \left\{

1290: \begin{array}{ll}

1291: \ds w_0'(t) + a_0w_0(t) + \int_{-r}^0 a_1(\xi)w_1(t,\xi)\,d\xi = 0, &

1292: t \in [0,T[ \\[8pt]

1293: w_0(T)=\gamma,\\[8pt]

1294: w_1(t,\xi) = w_0(t-\xi)\chi_{[0,T]}(t-\xi),\\[8pt]

1295: \displaystyle c(t) = \int_t^T \frac{(\cp{B}{w(s)}^+)^2}{4\beta}\,ds,

1296: & t \in [0,T].

1297: \end{array}\right.

1298: \end{equation}

1299: Moreover, the optimal strategy is

1300: \begin{equation}

1301: \label{eq:22}

1302: z^*(t) = \frac{\cp{B}{v_x(t)}^+}{2\beta} =

1303: \frac{\cp{B}{w(t)}^+}{2\beta}, \qquad t \in [0,T]

1304: \end{equation}

1305: (see \cite{levico} for more details).

1306: 

1307: We extend now the analysis of this specific situation. Let us begin

1308: with a rather explicit characterization of the optimal trajectory,

1309: which could be numerically approximated simply by solving a linear ODE

1310: with delay.

1311: %

1312: In particular, let $w=(w_0,w_1)$ be the solution of (\ref{eq:HJBf3}).

1313: Then, setting $z^*(t)={1\over 2\beta}\cp{B}{w(t)}^+$, the optimal

1314: trajectory is the $\erre$-valued component $Y_0$ of the (mild)

1315: solution of the abstract SDE

1316: $$ dY(t)=[AY(t) + Bz^*(t)]\,dt + G\,dW(t), $$

1317: which is given by

1318: $$

1319: Y(t) = e^{tA}Y(0) + \int_0^t e^{(t-s)A}Bz^*(s)\,ds

1320: + \int_0^t e^{(t-s)A}G\,dW(s).

1321: $$

1322: In particular $Y$ is a $X$-valued Gaussian process with mean

1323: and covariance operator

1324: $$

1325: \mu_t = e^{tA}Y(0) + \int_0^t e^{(t-s)A}Bz^*(s)\,ds ,

1326: \quad

1327: Q_t = \int_0^t e^{(t-s)A}GG^*e^{(t-s)A^*}\,ds,

1328: $$

1329: respectively.

1330: It follows that $Y_0$ is a Gaussian process itself with mean

1331: \begin{eqnarray}

1332: \E Y_0(t) &=& \cp{\mu_t}{e_1}_X =

1333: \left\langle e^{tA}Y(0),e_1\right\rangle_X

1334: + \left\langle \int_0^t e^{(t-s)A}Bz^*(s)\,ds,e_1\right\rangle_X \nonumber\\

1335: \label{eq:opttraj}

1336: &=& \left\langle Y(0),e^{tA^*}e_1\right\rangle_X

1337: + \int_0^t\left\langle Bz^*(s),e^{(t-s)A^*}e_1\right\rangle_X ds,

1338: \end{eqnarray}

1339: where $e_1=(1,0)\in X$.\\

1340: %

1341: Since $Y(0)$ is given as in Proposition \ref{prop:equiv} and

1342: $Bz^*(\cdot)$ is also easy to compute ($z^*$ is one dimensional and

1343: $B$ is just multiplication by a fixed vector in $X$), we are left with

1344: the problem of computing $e^{tA^*}e_1$. However, as one can prove by a

1345: direct calculation, the semigroup $e^{tA^*}$ is given by

1346: $$

1347: e^{tA^*}(x_0,x_1(\cdot)) = \Big(\phi(t),\phi(t+\xi)|_{\xi \in [-r,0]}\Big),

1348: $$

1349: where $\phi(\cdot)$ solves the linear ODE with delay

1350: \begin{equation}

1351: \label{eq:sgdelay}

1352: \left\{\begin{array}{l}

1353: \ds {d\phi(t)\over dt} =

1354: a_0 \phi(t) + \int_{-r}^0a_1(\xi)\phi(t+\xi)\,d\xi,

1355: \quad 0\leq t\leq T \\[10pt]

1356: \phi(0)=x_0; \quad \phi(\xi)=x_1(\xi) \;\; \forall\xi\in[-r,0].

1357: \end{array}\right.

1358: \end{equation}

1359: %

1360: Therefore $e^{tA^*}e_1$ is given by $(\phi(t),\phi(t+\xi)|_{\xi \in

1361:   [-r,0]})$, where $\phi$ solves (\ref{eq:sgdelay}) with initial

1362: condition $x_0=1$, $x_1(\cdot)=0$. Such $\phi$ can be computed

1363: numerically by discretizing (\ref{eq:sgdelay}), and then $\E Y_0(t)$

1364: can be obtained by approximating the integrals in (\ref{eq:opttraj})

1365: with finite sums.

1366: 

1367: Analogously one can write the variance of the optimal trajectory in

1368: such a way that it can be easily approximated by numerical methods. In

1369: particular, one has

1370: \begin{eqnarray}

1371: \mathrm{Var}\,Y_0(t) &=& \cp{Q_te_1}{e_1}

1372: = \left\langle

1373: \int_0^t e^{(t-s)A}GG^*e^{(t-s)A^*}\,ds\, e_1,e_1

1374: \right\rangle \nonumber\\

1375: &=& \int_0^t

1376: \left\langle e^{(t-s)A}GG^*e^{(t-s)A^*} e_1,e_1 \right\rangle ds \nonumber\\

1377: &=& \int_0^t

1378: \left\langle Ge^{(t-s)A^*}e_1,Ge^{(t-s)A^*}e_1 \right\rangle ds \nonumber\\

1379: &=& \int_0^t \left|Ge^{(t-s)A^*}e_1\right|^2\,ds.\label{eq:optvar}

1380: \end{eqnarray}

1381: Setting $\psi(s)=e^{(t-s)A^*}e_1$, which can be approximated as

1382: indicated before, one finally has

1383: $$

1384: \mathrm{Var}\,Y_0(t) = \sigma^2 \int_0^t \psi(s)^2\,ds.

1385: $$

1386: 

1387: \medskip

1388: 

1389: One can also perform simple comparative statics on the

1390: value function. For instance we can compute explicitly its sensitivity

1391: with respect to the (maximal) delay $r$:

1392: \begin{eqnarray*}

1393: \frac{\partial V}{\partial r}(t,x;r) &=&

1394: {\partial \over \partial r}\cp{w_1(t)}{x_1}_{L^2([-r,0])} +

1395: \frac{\partial c}{\partial r}(t;r) \\

1396: &=& w_1(t,-r)x_1(-r)

1397: + {1 \over 2\beta}

1398: \int_t^T \cp{B}{w(s)}

1399: {\partial\over\partial r}\left(\int_{-r}^0 b_1(\xi)w_1(s,\xi)\,d\xi\right)\,ds \\

1400: &=& w_1(t,-r)x_1(-r) + {b_1(-r) \over 2\beta}

1401: \int_t^T \cp{B}{w(s)} w_1(s,-r)\,ds,

1402: \end{eqnarray*}

1403: where we have used the fact that $\cp{B}{w(t)}^+=\cp{B}{w(t)}$.

1404: \par\noindent

1405: Note that in the above expression everything can be computed

1406: explicitly, as soon as we fix the delay kernel $b_1$.

1407: %

1408: Let us consider, as an example, the special case of

1409: $b_1(\xi)=b_1\chi_{[-r,0]}(\xi)$, where on the right-hand side, with a

1410: slight abuse of notation, $b_1$ is a positive constant. One has

1411: \begin{eqnarray*}

1412: \frac{\partial V}{\partial r}(t,x;r) &=& w_1(t,-r)x_1(-r) +

1413: {b_1 \over 2\beta} \int_t^T w_1(s,-r)

1414: \Big(b_0w_0(s) + b_1\int_{-r}^0w_1(s,\xi)\,d\xi\Big)\,ds.

1415: \end{eqnarray*}

1416: %

1417: Furthermore, if we consider the special case of delay in the control

1418: only, that is $a_1(\cdot)=0$, we obtain, after some calculations,

1419: \begin{eqnarray*}

1420: \frac{\partial V}{\partial r}(t,x;r) &=&

1421: \gamma e^{a_0(T-t+r)}x_1(-r)

1422: - \frac{b_1}{4\beta a_0}\gamma^2e^{a_0r}%

1423: \Big(b_0-{b_1\over a_0}(1-e^{a_0r})\Big)%

1424: (1-e^{2a_0(T-t)}),

1425: \end{eqnarray*}

1426: for $t \in [r,T]$.

1427: 

1428: 

1429: %\subsubsection{Delay in the control only}

1430: 

1431: \medskip

1432: 

1433: In the special case of $a_1(\cdot)\equiv 0$ an explicit solution of

1434: (\ref{eq:HJBf3}) is easily obtained. This solvability in closed form

1435: then ``propagates'' to other quantities of interest. In fact, note

1436: that (\ref{eq:HJBf3}) reduces to

1437: \begin{equation}%\label{eq:HJBf5}

1438: \left\{

1439: \begin{array}{l}

1440: \ds w_0'(t) + a_0w_0(t) = 0 \\[8pt]

1441: w_0(T)=\gamma,

1442: \end{array}\right.

1443: \end{equation}

1444: yielding

1445: $$ w_0(t) = \gamma e^{(T-t)a_0}, $$

1446: and therefore

1447: $$

1448: w_1(t,\xi) = \gamma e^{(T-(t+\xi))a_0}\,\chi_{[0,T]}(t+\xi)

1449: \quad

1450: c(t) = \int_t^T {\cp{B}{w(s)}^2 \over 4\beta}\,ds.

1451: $$

1452: That is, the last three formulae explicitly give a solution of the

1453: HJB equation (\ref{eq:HJB0}) in our specific case.

1454: 

1455: As a consequence we can also determine the unique optimal feedback control

1456: $z^*$ as follows:

1457: \begin{equation}

1458: \label{eq:27}

1459: z^*(t) = {\cp{B}{v_x}^+ \over 2\beta} = {\cp{B}{w(t)}^+ \over 2\beta}

1460: = {\gamma e^{(T-t)a_0} \over 2\beta} \left[%

1461: b_0 + \int_{-r}^0 b_1(\xi)e^{-a_0\xi}\chi_{[0,T]}(t+\xi)\,d\xi \right].

1462: \end{equation}

1463: 

1464: The optimal trajectory can be characterized in a completely similar

1465: way as above, with the difference that now we can explicitly write:

1466: $$

1467: e^{tA^*}e_1 = (e^{a_0t},e^{a_0t+\xi}|_{\xi\in[-r,0]}),

1468: $$

1469: hence simplifying (\ref{eq:opttraj}) in the present case. Even

1470: simpler is the expression for the variance of the optimal trajectory,

1471: which can be obtained by (\ref{eq:optvar}):

1472: $$

1473: \mathrm{Var}\,Y_0(t) = \sigma^2 \int_0^t e^{2a_0(t-s)}\,ds

1474: =\frac{\sigma^2}{2a_0} (e^{2a_0t}-1).

1475: $$

1476: 

1477: \begin{figure}[t]

1478: \includegraphics[width=\textwidth]{Fig1.eps}

1479: \caption{Optimal advertising policy in four different ``churn'' settings}

1480: \label{fig:1}

1481: \end{figure}

1482: 

1483: Sharp characterizations of the optimal advertising trajectory as

1484: well as the resulting expected profit functions in the case of

1485: linear reward and quadratic cost function allow for interesting

1486: observations regarding the importance of the proper accounting for

1487: the memory effects in planning the advertising campaign. Figure

1488: \ref{fig:1} displays the optimal advertising spending rates

1489: $z^*(t)$, as expressed by (\ref{eq:22}), with

1490: $a_1(\xi)=\hat{a}_1e^{-|\xi|/\delta_a}$ and

1491: $b_1(\xi)=\hat{b}_1e^{-|\xi|/\delta_b}$ in four different settings:

1492: $a_1 = b_1 = 0$ (``no churn''), $a_1=-5$, $b_1=0$ (``goodwill

1493: churn''), $a_1=0$, $b_1=5$ (``advertising churn''), $a_1=-5$,

1494: $b_1=5$ (``goodwill-advertising churn''). Note that in the absence

1495: of churn, the optimal advertising trajectory, as implied by

1496: (\ref{eq:27}), is a monotone function of time with $z^*(t)=\gamma

1497: b_0/(2\beta)$. While the advertising rates are similar in all

1498: settings in the beginning of the pre-launch period as well as right

1499: before the product launch time $T$, the details of advertising

1500: policies differ dramatically in the middle of the pre-launch period.

1501: For example, in the presence of a strong ``goodwill churn'' the

1502: optimal advertising trajectory takes a characteristic ``impulse''

1503: shape, while in the strong ``advertising churn'' setting the optimal

1504: advertising spending quickly builds up a strong goodwill level in

1505: the middle of the pre-launch region, slowing down significantly

1506: right before the product launch. When the presence of both types of

1507: ``churn'' is pronounced, the optimal advertising policy is

1508: represented by a set of advertising sprees with rapidly growing

1509: intensity. Figure \ref{fig:2} illustrates how the strong influence

1510: of memory effects on the shape of optimal advertising policies

1511: translates into performance differences between the optimal

1512: advertising policies and the policies which neglect the presence of

1513: advertising delays. In this figure we plot the relative difference

1514: $$

1515: \frac{V(0,x)-V^0(0,x)}{V(0,x)}

1516: $$

1517: between the optimal expected profit function $V(0,x)$ and the expected

1518: profit value $V^0(0,x)$ obtained by applying, for $t\in [0,T]$, the

1519: advertising policy $z^0(t)=\gamma b_0e^{(T-t)a_0}/(2\beta)$ optimal in

1520: the absence of memory effects (i.e. in the setting $a_1=b_1=0$).  This

1521: relative difference is plotted as a function of the amplitude of the

1522: ``goodwill churn'' term $a_1$ (Figure \ref{fig:2}a), and as a function

1523: of the amplitude of the ``advertising churn'' term $b_1$ (Figure

1524: \ref{fig:2}b). The initial goodwill conditions were selected as

1525: $x_0=10$ and $x_1(\xi)= x_0 e^{-|\xi|}$ for $\xi \in [-r,0]$ and the

1526: advertising history $z_0(\xi)$ was set equal to $0$ for

1527: $\xi\in[-r,0]$. We observe that the relative loss of efficiency

1528: associated with the use of the ``memoryless'' policy $z^0(t)$ can be

1529: quite significant -- in the examples we use it exceeds 5\% and can be

1530: as high as 20\% in settings with strong ``churn'' effects.

1531: 

1532: \begin{figure}[t]

1533: \includegraphics[width=\textwidth]{Fig2.eps}

1534: \caption{Relative performance gap of the ``memoryless'' advertising policy}

1535: \label{fig:2}

1536: \end{figure}

1537: 

1538: 

1539: 

1540: \section{The case of delay in the state term only}

1541: \label{sec:ex}

1542: 

1543: In this section we consider a model for the dynamics of goodwill with

1544: forgetting, but without lags in the effect of advertising expenditure

1545: (carryover), i.e. with $b_1(\cdot)=0$ in (\ref{eq:SDDE}). An analysis

1546: of this model was sketched in \cite{levico}, where only an abstract

1547: existence result was given. Here we present a more refined result (see

1548: theorem \ref{thm:L2} below) and obtain some qualitative properties of

1549: the value function, together with a characterization of the optimal

1550: strategies in terms of the value function in two specific cases. In

1551: particular, theorem \ref{thm:L2} formulates sufficient conditions

1552: ensuring that the value function is the unique solution (in a suitable

1553: sense) of the associated Bellman equation and that the optimal

1554: advertising policy is of the feedback type. Let us recall once again

1555: that such a situation is not possible in the more general case

1556: discussed in the previous section. Moreover, in the case of linear

1557: cost function, the optimal control takes a particularly simple

1558: ``bang-bang'' form (Corollary \ref{cor:bang}).

1559: 

1560: Let us also mention that for stochastic control problem with delay

1561: terms in the state variable one can apply both the approach of

1562: Hamilton-Jacobi equations in $L^2$ spaces developed by Goldys and

1563: Gozzi \cite{GolGoz}, and the forward-backward SDE approach of

1564: Fuhrman and Tessitore \cite{FT1}.  We follow here the first approach,

1565: showing that both the value function and the optimal advertising

1566: policy can be characterized in terms of the solution of a Bellman

1567: equation in infinite dimensions.

1568: 

1569: We assume, for the sake of simplicity, that the goodwill evolves

1570: according to the following equation, where the distribution of the

1571: forgetting factor is concentrated on a point:

1572: \begin{equation}

1573: \label{eq:ex1}

1574: \left\{\begin{array}{l}

1575: dy(t) = \ds [a_0 y(t) + a_1y(t-r)

1576:     + b_0 z(t)]\,dt + \sigma\,dW_0(t), \quad 0\leq s\leq t\leq T \\[10pt]

1577: y(s)=x_0; \quad y(s+\xi)=x_1(\xi), \;\; \xi\in[-r,0].

1578: \end{array}\right.

1579: \end{equation}

1580: %

1581: The following standard infinite dimensional Markovian reformulation of

1582: this dynamics will turn out to be useful.

1583: 

1584: Let us define the operator $\A:D(\A)\subset X \to X$ as

1585: $$

1586: \A:(x_0,x_1(\cdot)) \mapsto (a_0x_0+a_1x_1(-r),x_1'(\cdot)),

1587: \qquad

1588: D(\A) = \erre\times W^{1,2}([-r,0];\erre).

1589: $$

1590: It is well-known (see e.g. \cite{DZ96}) that $\A$ is the generator of

1591: a strongly continuous semigroup $S(t)$ on $X$. More precisely, one has

1592: $$

1593: S(t)(x_0,x_1) = (u(t),u(t+\xi)|_{\xi\in[-r,0]}),

1594: $$

1595: where $u(\cdot)$ is the solution of the deterministic delay equation

1596: \begin{equation}

1597: \label{eq:sgA}

1598: \left\{\begin{array}{l}

1599: \ds {du(t)\over dt} = a_0 u(t) + a_1 u(t-r), \quad 0\leq t\leq T \\[10pt]

1600: u(0)=x_0; \quad u(\xi)=x_1(\xi), \;\; \xi\in[-r,0].

1601: \end{array}\right.

1602: \end{equation}

1603: %

1604: Furthermore, set $\bar{z}=(\sigma^{-1}b_0 z,z_1(\cdot))$, with

1605: $z_1(\cdot)$ a fictitious control taking values in

1606: $L^2([-r,0],\erre)$, and

1607: define $G:X \to X$ as

1608: $$

1609: G: (x_0,x_1(\cdot)) \mapsto (\sigma x_0,0).

1610: $$

1611: Let $W_1$ be a cylindrical Wiener process taking

1612: values in $L^2([-r,0],\erre)$, so that $W=(W_0,W_1)$ is an $X$-valued

1613: cylindrical Wiener process.

1614: 

1615: Chojnowska-Michalik \cite{choj78} proved the following equivalence result.

1616: \begin{lemma}

1617: \label{lem:equiv}

1618:   Let $Y=(Y_0,Y_1)$ be the unique mild solution of the following stochastic

1619:   evolution equation on $X$:

1620: \begin{equation}

1621: \label{eq:ex1a}

1622: \left\{\begin{array}{l}

1623: dY(t) = (\A Y(t) + G\bar{z}(t))\,dt + G\,dW(t),

1624:               \quad 0\leq s \leq t \leq T,\\[10pt]

1625: Y(s) = x.

1626: \end{array}\right.

1627: \end{equation}

1628: Then $Y_0(t)$ solves the stochastic delay equation (\ref{eq:ex1}).

1629: \end{lemma}

1630: %

1631: Define $h:X\to\erre$ and $\varphi:X\to\erre$ as

1632: \begin{eqnarray*}

1633: h(x_0,x_1) &=& -h_0(\sigma b_0^{-1}x_0) \\

1634: \varphi(x_0,x_1) &=& \varphi_0(x_0).

1635: \end{eqnarray*}

1636: Then we have, thanks to lemma \ref{lem:equiv},

1637: $$

1638: J(s,x;z) =

1639: \E_{s,x}\left[\varphi(Y(T))+\int_t^T h(\bar{z}(s))\,ds \right],

1640: $$

1641: and

1642: \begin{equation}

1643: \label{ex1j}

1644: V(s,x) = \sup_{\bar{z}\in\mathcal{Z}} J(s,x;\bar{z}),

1645: \end{equation}

1646: where $\mathcal{Z}$ denotes the set of all strategies $z:[0,T]\to

1647: \tilde{U} \times L^2([-r,0],\erre)$ adapted to the filtration

1648: generated by $Y$, and $\tilde{U}$ is the image of $U$ under the action

1649: of the map $x\mapsto \sigma^{-1}b_0x$.

1650: 

1651: \smallskip

1652: 

1653: We can now prove some qualitative properties of the value function.

1654: \begin{prop}\label{prop:V-conv}

1655:   If $\varphi_0$ is concave and $h_0$ is convex, then the value

1656:   function $V(s,x)$ is concave with respect to $x$.

1657: \end{prop}

1658: %

1659: \begin{proof}

1660: Identical to the proof of proposition \ref{prop:Vconv}, thus omitted.

1661: \end{proof}

1662: %

1663: In the following proposition we use the ordering in $X$ defined right before

1664: Proposition \ref{prop:Vincr}.

1665: %

1666: \begin{prop}\label{prop:V-incr}

1667:   Let $a_1 \geq 0$ and $\varphi_0$ be increasing. Then the value

1668:   function $V(s,x)$ is increasing with respect to $x$. Moreover, if

1669:   $\varphi_0$ is strictly increasing, then the value function $V(s,x)$

1670:   is strictly increasing with respect to $x$, and $V(s,x^1)=V(s,x^2)$

1671:   if and only if $x^1=x^2$.

1672: \end{prop}

1673: %

1674: \begin{proof}

1675: Let $x^1\geq x^2$ in the sense just defined. One has

1676: \begin{eqnarray*}

1677: J(s,x^1;z)-J(s,x^2;z) &=&

1678: \E\Big[ \varphi(y^{s,x^1,z}(T)) - \varphi(y^{s,x^2,z}(T)) \Big] \\

1679: &=& \E\Big[ \varphi(e^{\A(T-s)}x^1 + \zeta) -

1680:             \varphi(e^{\A(T-s)}x^2 + \zeta)

1681:       \Big],

1682: \end{eqnarray*}

1683: where $\zeta:=\int_s^T e^{\A(T-t)}G\bar{z}(t)\,dt+%

1684: \int_s^T e^{\A(T-t)}G\,dW(t)$. The assumption $a_1\geq 0$

1685: together with (\ref{eq:sgA}) implies that the semigroup generated by

1686: $\A$ is positivity preserving, i.e.  $x^1\geq x^2$ implies

1687: $e^{\A(T-s)}x^1\geq e^{\A(T-s)}x^2$.  Therefore, by the monotonicity of

1688: $\varphi_0$, one also has $\varphi(e^{\A(T-s)}x^1 + \zeta) \geq

1689: \varphi(e^{\A(T-s)}x^2 + \zeta)$ a.s., hence $J(s,x^1;z) \geq

1690: J(s,x^2;z)$, and finally $V(s,x^1)=\sup_{z\in\mathcal{Z}} J(s,x^1;z)

1691: \geq \sup_{z\in\mathcal{Z}} J(s,x^2;z)=V(s,x^2)$. The other assertions

1692: follow analogously, using (\ref{eq:sgA}).

1693: \end{proof}

1694: %

1695: \begin{rmk}

1696:   In the above proof the positivity preserving property of the

1697:   semigroup $e^{t\A}$ is crucial, and the assumption $a_1\geq 0$ is

1698:   ``sharp'' in the following sense: if $a_1<0$, one can find $x>0$

1699:   such that $e^{t\A}$ \emph{inverts} the sign, i.e. $e^{t\A}x<0$.

1700: 

1701:   Moreover, under the assumptions of the theorem, the value function

1702:   is increasing with respect to the real valued component of the

1703:   initial datum. By this we mean that given $x^1 \geq x^2$ with

1704:   $x^1_0>x^2_0$ and $x^1_1=x^2_1$ a.e., then $V(s,x^1)>V(s,x^2)$.

1705:   Therefore one also has $D^-_0V\geq 0$. The subdifferential can be

1706:   replaced by the derivative if we can guarantee that $V$ is

1707:   continuously differentiable with respect to $x_0$. Conditions for

1708:   the continuous differentiability of $V$ with respect to $x$ are

1709:   given in proposition \ref{prop:Vdiff} below.

1710: \end{rmk}

1711: 

1712: In contrast with the general case considered in the previous section,

1713: if delay enters only the state term, then it is possible to uniquely

1714: solve the associated Bellman equation, and thus to characterize the

1715: value function and construct optimal strategies.  The following

1716: result, which relies on \cite{GolGoz}, gives precise conditions for

1717: the above assertions to hold.

1718: %

1719: \begin{thm}\label{thm:L2}

1720:   Assume that $h_0$ is convex, let $H_0:\erre\ni p \mapsto

1721:   \sup_{u\in U}(pb_0u-h_0(u))$, and suppose that $a_0 < -a_1 <

1722:   \sqrt{\gamma^2+a_0^2}$, where $\gamma\,\mathrm{coth}\,\gamma = a_0$,

1723:   $\gamma\in]0,\pi[$. Then the value function $V(s,x)$ coincides

1724:   $\mu$-a.e. with the mild solution in $L^2(X,\mu)$ (in the sense of

1725:   \cite{GolGoz}) of the equation

1726: \begin{equation}

1727: \label{eq:ex1-hjb-expl}

1728: \left\{\begin{array}{l}

1729: \ds \partial_t v

1730: + {1\over 2}\sigma^2 \partial_0^2 v

1731: + (a_0x_0 + a_1x_1(-r)) \partial_0 v

1732: + \int_{-r}^0 x'_1(\xi) \partial_1 v(\xi)\,d\xi

1733: + H_0(\partial_0 v) = 0 \\[10pt]

1734: v(T,x_0,x_1) = \varphi_0(x_0),

1735: \end{array}\right.

1736: \end{equation}

1737: where $\mu$ is a measure of full support on $X$.

1738: %

1739: Moreover, the optimal strategy admits the feedback representation

1740: \begin{equation}

1741: \label{ex1-zopt}

1742: z^*(t) \in \sigma b_0^{-1}D^-_0H( \sigma\partial_0 v(t,Y^*_0(t),Y_1^*(t) ),

1743: \end{equation}

1744: with $H$ defined in (\ref{eq:h}), provided there exists a solution

1745: $Y_0^*(t)$, $Y_1^*(t)$ of the closed-loop differential inclusion

1746: \begin{equation}

1747: \label{eq:di}

1748: \left\{\begin{array}{l}

1749: \ds dY_0(t) \in \left[a_0Y_0(t) + a_1Y_0(t-r) +

1750: \sigma D^-_0H(\sigma \partial_0 v(t,Y^*_0(t),Y_1^*(t))

1751: \right]dt + \sigma\,dW_0(t) \\[10pt]

1752: \ds dY_1(t)(\xi) = \frac{d}{d\xi}Y_1(t)(\xi).

1753: \end{array}\right.

1754: \end{equation}

1755: \end{thm}

1756: %

1757: \begin{proof}

1758: By the usual heuristic application of the dynamic programming principle

1759: one can associate to the control problem (\ref{ex1j}) the following

1760: Hamilton-Jacobi-Bellman equation on $X$:

1761: \begin{equation}

1762: \label{ex1hjb}

1763: \left\{\begin{array}{ll}

1764: \ds v_t + {1\over 2}\tr(GG^*v_{xx}) + \cp{\A x}{v_x} + H_0(v_x) = 0,

1765: \quad 0 \leq t \leq T, \\[10pt]

1766: v(T,x) = \varphi(x),

1767: \end{array}\right.

1768: \end{equation}

1769: which coincides, after some calculations, with (\ref{eq:ex1-hjb-expl}).

1770: %

1771: Note that the Hamiltonian $H_0$ can be regarded as a function of $X$

1772: in $\erre$ and can be equivalently written as

1773: \begin{eqnarray}

1774: H_0(p) = H_0(p_0) &=&

1775: \sup_{z\in \tilde{U}\times L^2([-r,0],\erre)}

1776: \Big(\cp{Gz}{p} + h(z)\Big) \nonumber \\

1777: &=& \sup_{z_0\in \tilde{U}}

1778: \Big(\sigma z_0p_0 - h_0(\sigma b_0^{-1}z_0)\Big).

1779: \label{eq:h0}

1780: \end{eqnarray}

1781: In order to apply the results of \cite{GolGoz}, we also need to define

1782: \begin{equation}

1783: \label{eq:h}

1784: H(q)=H(q_0)=H_0(\sigma^{-1}q_0) =

1785: \sup_{z_0\in\tilde{U}} \Big(z_0q_0 - h_0(\sigma b_0^{-1}z_0)\Big).

1786: \end{equation}

1787: %

1788: Since $h_0$ is bounded from below, (\ref{eq:h}) implies that $H$ is

1789: Lipschitz continuous (in $\erre$ and in $X$). Moreover, the assumption

1790: on $a_0$, $a_1$ and assumption (i) of section \ref{sec:formul} imply

1791: that the uncontrolled version of (\ref{eq:ex1a}), i.e.

1792: \begin{equation}

1793: \label{eq:ex1auc}

1794: dY(t) = AY(t)\,dt + G\,dW(t),

1795: \end{equation}

1796: admits a unique non-degenerate invariant measure $\mu$ on $X$ (see

1797: \cite{DZ96}), which is Gaussian with mean zero and covariance

1798: operator $Q_\infty=\int_0^\infty e^{sA}GG^*e^{s^*A}\,ds$. In

1799: particular, the restriction of $\mu$ on the $\erre$-valued component

1800: of $X$ has a density

1801: $\rho(x)=\frac{1}{\nu\sqrt{2\pi}}e^{-|x|^2/2\nu^2}$ for some $\nu>0$.

1802: This implies that $\varphi\in L^2(X,\mu)$: in fact,

1803: $$

1804: \int_X |\varphi(x)|^2\,\mu(dx) =

1805: \int_\erre |\varphi_0(x)|^2\rho(x)\,dx \leq

1806: K \int_\erre (1+|x|)^m e^{-|x|^2/2\nu^2}\,dx < \infty.

1807: $$

1808: Therefore, theorems 3.7 and 5.7 of

1809: \cite{GolGoz} yield the existence and uniqueness of a solution in

1810: $L^2(X,\mu)$ of (\ref{ex1hjb}), or equivalently of

1811: (\ref{eq:ex1-hjb-expl}), which coincides $\mu$-a.e. with the value

1812: function $V$.

1813: %

1814: Finally, observing that the maximum in (\ref{eq:h}) is reached by

1815: $D^-_0H(q_0)$ (setting, if needed, $h_0(x)=+\infty$ for $x\not\in U$),

1816: a slight modification of the proof of theorem 5.7 in \cite{GolGoz}

1817: shows that the optimal strategy is given by $\bar{z}_0^*(t) \in

1818: D^-_0H(\sigma\partial_0 v(t,Y^*_0(t),Y_1^*(t))$, where

1819: $Y^*=(Y_0^*,Y_1^*)$ is a solution (if any) of the stochastic

1820: differential inclusion (\ref{eq:di}). The relation $z^*(t)=\sigma

1821: b_0^{-1}\bar{z}_0^*(t)$ thus completes the proof.

1822: \end{proof}

1823: %

1824: Let us briefly comment on the previous result: the HJB equation

1825: (\ref{eq:ex1-hjb-expl}) is ``genuinely'' infinite dimensional, i.e. it

1826: reduces to a finite dimensional one only in very special cases. For

1827: example, by the results in \cite{lari}, (\ref{eq:ex1-hjb-expl})

1828: reduces to a finite dimensional PDE if and only if $a_0=-a_1$.

1829: However, under this assumption, we cannot guarantee the existence of a

1830: non-degenerate invariant measure for the Ornstein-Uhlenbeck semigroup

1831: associated to (\ref{eq:ex1auc}). Even more extreme would be the

1832: situation of distributed forgetting time: in this case the HJB

1833: equation is finite dimensional only if the term accounting for

1834: distributed forgetting vanishes altogether.

1835: %

1836: Moreover, note that if $a_1$ is negative, i.e. it can be interpreted

1837: as a deterioration factor, the assumption of the theorem says that

1838: $a_1$ cannot be ``much more negative'' than $a_0$. On the other hand,

1839: if $a_1$ is positive, then the improvement effect as measured by $a_1$

1840: cannot exceed the deterioration effect as measured by $|a_0|$.  In

1841: essence, the condition on $a_0$, $a_1$, which is needed to ensure

1842: existence of an invariant measure for equation (\ref{eq:ex1auc}), does

1843: not impose severe restrictions on the dynamics of goodwill.

1844: 

1845: \smallskip

1846: 

1847: If the data of the problem are smoother, a different approach allows

1848: one to obtain regularity of the value function.

1849: \begin{prop}\label{prop:Vdiff}

1850:   Assume that $\varphi_0\in C^1(\erre)$,

1851:   $|\varphi_0'(x)|\leq K(1+|x|)^m$, and $H_0\in C^1(\erre)$.

1852:   Then $V\in C^{0,1}([0,T]\times X)$.

1853: \end{prop}

1854: \begin{proof}

1855:   Follows by the regularity results for solutions of semilinear

1856:   partial differential equations in Hilbert spaces obtained through

1857:   the FBSDE approach. In particular, denoting by $C$ a positive

1858:   constant, boundedness of $U$ implies that $|b_0 z|<C$, $|h_0(z)|

1859:   \leq K(1+|z|)^m < C$, and finally $H_0$ is Lipschitz as follows by

1860:   \begin{eqnarray*}

1861:     |H_0(p)-H_0(q)| &=& |\sup_{z\in U}(\cp{Bp}{z}-h_0(z))

1862:                        - \sup_{z\in U}(\cp{Bp}{z}-h_0(z))| \\

1863:     &\leq& \sup_{z\in U} |\cp{(p-q)}{B^*z}| \leq C|p-q|.

1864:   \end{eqnarray*}

1865:   Then all hypotheses of \cite{FT-corso}, theorem 4.3.1, are

1866:   satisfied, which yields the claim.

1867: \end{proof}

1868: %

1869: %In the following corollary we do not need $h_0$ to satisfy a

1870: %polynomial growth condition.

1871: \begin{coroll}\label{cor:Vdiff}

1872:   Let $\varphi_0$ be as in proposition \ref{prop:Vdiff} and $h_0$

1873:   strictly convex. Then $V\in C^{0,1}([0,T]\times X)$.

1874: \end{coroll}

1875: \begin{proof}

1876:   Since convexity implies continuity in the interior of the domain,

1877:   then $h_0(U)$ is bounded. Extending $h_0$ as $h_0(x)=+\infty$ for

1878:   $x\not\in U$, $h_0$ is clearly $1$-coercive, hence $H$ is convex and

1879:   finite on the whole $\erre$ (\cite{lema}, prop. E.1.3.8). The strict

1880:   convexity of $h_0$ implies that $H_0$ is continuously differentiable

1881:   in the interior of its domain, i.e. on $\erre$ (\cite{lema}, thm.

1882:   E.4.1.1). Then the smoothness of $V$ follows again by

1883:   \cite{FT-corso}, theorem 4.3.1.

1884: \end{proof}

1885: %

1886: %

1887: \begin{rmk}

1888:   We should also mention that in the framework of the FBSDE approach

1889:   to HJB equations (\cite{FT-corso}), if the Hamiltonian $H$ and the

1890:   terminal condition $\varphi$ satisfy some smoothness and boundedness

1891:   conditions, then we do not need the assumption about the existence

1892:   of an invariant measure for the uncontrolled state equation. The

1893:   approach used above (\cite{GolGoz}), while requiring the

1894:   existence of the above mentioned invariant measure, allows for more

1895:   singular data (for instance one could choose $\varphi_0(x)=-M$,

1896:   $M>\!\!>0$, for $x\in\erre_-$, and $\varphi_0(x)\geq 0$ for

1897:   $x\in\erre_+$).

1898: \end{rmk}

1899: 

1900: \medskip

1901: 

1902: In general, obtaining explicit expressions of the value function $V$

1903: trying to solve (\ref{eq:ex1-hjb-expl}) is impossible. However, under

1904: specific assumptions on the model we can obtain stronger

1905: characterizations, at least from a qualitative point of view, of the

1906: value function and/or of the optimal strategy.

1907: %

1908: \begin{coroll}

1909: Assume that $h_0(x)=\beta x^2$ and $U=[0,R]$, $R<\infty$. Then

1910: the optimal strategy is given by

1911: \begin{equation}

1912: \label{eq:L2-ex1}

1913: z^*(t) =

1914: \left\{\begin{array}{ll}

1915: \ds 0, & D_0V^* < 0 \\[8pt]

1916: \ds \frac{b_0D_0V^*}{2\beta}, & 0 \leq D_0V^* \leq

1917:       2b_0^{-1}\beta R \\[8pt]

1918: \ds R, & D_0V^* > 2b_0^{-1}\beta R,

1919: \end{array}\right.

1920: \end{equation}

1921: where $V^*:=V(t,Y_0^*(t),Y_1^*(t))$.

1922: \end{coroll}

1923: %

1924: \begin{proof}

1925: One has

1926: \begin{eqnarray*}

1927: H_0(p) &=& \sup_{0\leq z_0\leq \tilde{R}} \Big(\cp{Gp}{z}+h(z)\Big)

1928: = \sup_{0\leq z_0 \leq \tilde{R}} (\sigma p_0 z_0 - \tilde\beta z_0^2) \\

1929: &=& \ds \frac{(\sigma p_0)^2}{4\tilde\beta}

1930:         I_{\{0 \leq p_0 \leq \frac{2\tilde\beta\tilde{R}}{\sigma} \}}

1931:      + (\sigma p_0 \tilde{R} - \tilde\beta\tilde{R}^2)

1932:         I_{\{ p_0 > {2\tilde\beta\tilde{R} \over \sigma} \}}

1933: \end{eqnarray*}

1934: where $\tilde{R}=\sigma^{-1}b_0R$ and $\tilde\beta:=\sigma^2 b_0^{-2}\beta$.

1935: Therefore

1936: $$

1937: H(q) = q_0^2/4\tilde\beta \, I_{\{0 \leq q_0 \leq 2\tilde\beta\tilde{R} \}}

1938:        + (q_0 \tilde{R} - \tilde\beta\tilde{R}^2) \, I_{\{q_0 > 2\tilde\beta\tilde{R}\}}

1939: $$

1940: and

1941: $$

1942: DH(q) = q_0/2\tilde\beta I_{\{0 \leq q_0 \leq 2\tilde\beta\tilde{R}\}}

1943: + \tilde{R} I_{\{ q_0 > 2\tilde\beta\tilde{R}\}}.

1944: $$

1945: %

1946: Theorem \ref{thm:L2} now yields (\ref{eq:L2-ex1}).

1947: \end{proof}

1948: %

1949: Note that whenever $\varphi_0$ is increasing, we get $D^-_0V^*\geq 0$, hence

1950: the optimal control is either linear in $D_0V^*$ or constant for

1951: $D_0V^*$ over a threshold.

1952: 

1953: \begin{coroll}     \label{cor:bang}

1954: Assume that $h_0(x)=\beta x$. Then the optimal strategy is of

1955: the bang-bang type and is given by

1956: $$

1957: z^*(t) =

1958: \left\{\begin{array}{ll}

1959: \ds 0, & D_0V^* < \sigma b_0^{-2}\beta \\[6pt]

1960: \ds \rho, & D_0V^* = \sigma b_0^{-2}\beta \\[6pt]

1961: \ds R, & D_0V^* > \sigma b_0^{-2}\beta,

1962: \end{array}\right.

1963: $$

1964: where $\rho$ is an arbitrary real number.

1965: \end{coroll}

1966: %

1967: \begin{proof}

1968: Setting $\tilde{R}=\sigma^{-1}b_0R$ and

1969: $\tilde\beta:=\sigma^2 b_0^{-2}\beta$, one has

1970: $$

1971: H_0(p) = \sup_{0\leq z_0 \leq \tilde{R}} (\sigma p_0 z_0 - \tilde\beta z_0)

1972: = (\sigma p_0-\tilde\beta)\tilde{R} \, I_{\{p_0 > \tilde\beta/\sigma\}}

1973: $$

1974: and $H(q) = (q_0-\tilde\beta)\tilde{R} \, I_{\{q_0 > \tilde\beta\}}$,

1975: thus

1976: $$

1977: D^-H(q) = \erre \, I_{\{q_0 = \tilde\beta\}}

1978:           + \tilde{R}\, I_{\{q_0 > \tilde\beta\}}.

1979: $$

1980: Theorem \ref{thm:L2} yields the conclusion.

1981: \end{proof}

1982: 

1983: In general, even specifying a functional form of $h_0$, an explicit

1984: solution of the HJB equation for arbitrary $\varphi_0$ is not

1985: available, hence the above expressions of the optimal control strategy

1986: in terms of the value function and their corresponding qualitative

1987: properties are the ``best'' that one can expect, at least in the cases

1988: we have considered.

1989: 

1990: 

1991: 

1992: \section{Concluding remarks}

1993: 

1994: A number of deterministic advertising models allowing for delay effects have

1995: been proposed in the literature. However, the corresponding problems in the

1996: stochastic setting have not been investigated. One of the reasons is

1997: certainly that a theory of continuous-time stochastic control with delays

1998: has only been developed recently, following two approaches. The first

1999: approach is based on the solution of an associated infinite-dimensional

2000: Hamilton-Jacobi-Bellman equation in spaces of integrable functions (see

2001: \cite{GolGoz}). The other one relies on the analysis of an appropriate

2002: infinite-dimensional forward-backward stochastic differential equation (see

2003: \cite{FT1}). Both approaches, however, cannot be applied to problems with

2004: distributed lag in the effect of advertising.

2005: 

2006: Problems with memory effects in both the state and the control have

2007: been studied first by Vinter and Kwong \cite{VK} (in a deterministic

2008: LQ setting), and by Gozzi and Marinelli \cite{levico} (in the case of

2009: linear stochastic dynamics and general objective function). A general

2010: theory of solvability of corresponding HJB equations is currently not

2011: available, while an infinite-dimensional Markovian reformulation and a

2012: ``smooth'' verification theorem have been proved in \cite{levico}.

2013: 

2014: In this paper we have concentrated on deriving qualitative properties

2015: of the value function (such as convexity, monotonicity with respect to

2016: initial conditions, smoothness). For specific choices of the reward

2017: and cost functions, we obtain more explicit characterizations of value

2018: function and optimal state-control pair.

2019: 

2020: While our work makes a substantial initial step in the analysis of the

2021: stochastic advertising problems with delays, more remains to be done.

2022: Potential extensions of the present work include the analysis of problems

2023: with budget constraints as well as problems of advertising through multiple

2024: media outlets with different delay characteristics.

2025: 

2026: 

2027: \bibliographystyle{amsplain}

2028: \bibliography{ref}

2029: 

2030: 

2031: \end{document}

2032: