1: 

2: \documentclass[11pt]{article}

3:  

4: \input psfig.sty

5: \psfull

6: 

7: \topmargin      0.25truein

8: \oddsidemargin  0.0truein

9: \evensidemargin 0.0truein

10: \textheight     8.5truein

11: \textwidth      6.5truein

12: \footskip       0.6truein

13: \headheight     0.0truein

14: \headsep        0.0truein

15: \parskip 0.3cm

16: 

17: \begin{document}

18:  

19: \newtheorem{lemma}{Lemma}

20: \newtheorem{theorem}{Theorem}

21: \newtheorem{corollary}{Corollary}

22: \newtheorem{definition}{Definition}

23: \newtheorem{example}{Example}

24: \newtheorem{proposition}{Proposition}

25: \newtheorem{condition}{Condition}

26: 

27: \newcommand{\be}{\begin{equation}}

28: \newcommand{\ee}{\end{equation}}

29: \newcommand{\bea}{\begin{eqnarray}}

30: \newcommand{\eea}{\end{eqnarray}}

31: \newcommand{\beaa}{\begin{eqnarray*}}

32: \newcommand{\eeaa}{\end{eqnarray*}}

33: \newcommand{\ben}{\begin{enumerate}}

34: \newcommand{\een}{\end{enumerate}}

35: \newcommand{\bi}{\begin{itemize}}

36: \newcommand{\ei}{\end{itemize}}

37: 

38: \newcommand{\lip}{\langle}

39: \newcommand{\lan}{\langle}

40: \newcommand{\rip}{\rangle}

41: \newcommand{\ran}{\rangle}

42: \newcommand{\uu}{\underline}

43: \newcommand{\oo}{\overline}

44: \newcommand{\til}{\tilde}

45: 

46: \newcommand{\La}{\Lambda}

47: \newcommand{\la}{\lambda}

48: \newcommand{\eps}{\epsilon}

49: \newcommand{\vph}{\varphi}

50: \newcommand{\al}{\alpha}

51: \newcommand{\bet}{\beta}

52: \newcommand{\gam}{\gamma}

53: \newcommand{\Gam}{\Gamma}

54: \newcommand{\kap}{\kappa}

55: \newcommand{\Del}{\Delta}

56: \newcommand{\Th}{\Theta}

57: \newcommand{\s}{\sigma}

58: \newcommand{\sig}{\sigma}

59: \newcommand{\Sig}{\Sigma}

60: \newcommand{\del}{\delta}

61: \newcommand{\om}{\omega}

62: \newcommand{\Om}{\Omega}

63: \newcommand{\X}{{\bf \Xi}}

64: 

65: \newcommand{\integers}{Z\!\!\!Z}

66: \newcommand{\Z}{Z\!\!\!Z}

67: \newcommand{\N}{{\rm I\!N}}

68: \newcommand{\rationals}{{\rm I\!Q}}

69: \newcommand{\reals}{{\rm I\!R}}

70: \newcommand{\R}{{\rm I\!R}}

71: \newcommand{\RJ}{{{\rm I\!R}^J}}

72: \newcommand{\RJP}{{{\rm I\!R}^J_+}}

73: \newcommand{\naturals}{{\rm I\!N}}

74: 

75: \newcommand{\calA}{{\cal A}}

76: \newcommand{\calB}{{\cal B}}

77: \newcommand{\calC}{{\cal C}}

78: \newcommand{\calD}{{\cal D}}

79: \newcommand{\calF}{{\cal F}}

80: \newcommand{\calG}{{\cal G}}

81: \newcommand{\calH}{{\cal H}}

82: \newcommand{\calJ}{{\cal J}}

83: \newcommand{\calL}{{\cal L}}

84: \newcommand{\calM}{{\cal M}}

85: \newcommand{\calP}{{\cal P}}

86: \newcommand{\calS}{{\cal S}}

87: \newcommand{\calT}{{\cal T}}

88: \newcommand{\calU}{{\cal U}}

89: \newcommand{\calV}{{\cal V}}

90: \newcommand{\calX}{{\cal X}}

91: \newcommand{\calY}{{\cal Y}}

92: 

93: \newcommand{\proof}{\noindent {\bf Proof:\ }}

94: \newcommand{\proofOf}[1]{\noindent {\bf Proof of #1:\ }}

95: \newcommand{\remark}{\noindent {\bf Remark:\ }}

96: \newcommand{\remarks}{\noindent {\bf Remarks:\ }}

97: \newcommand{\note}{\noindent {\bf Note:\ }}

98: \newcommand{\esssup}{{\rm ess}\sup}

99: \newcommand{\essinf}{{\rm ess}\inf}

100: \newcommand{\pl}{\partial}

101: \newcommand{\noi}{\noindent}

102: \newcommand{\goto}{\to}

103: \newcommand{\ink}{\rule{.5\baselineskip}{.55\baselineskip}}

104: \newcommand{\qed}{\rule{.5\baselineskip}{.55\baselineskip}}

105: 

106: \def\ve{\varepsilon}

107: \def\vr{\varrho}

108: 

109: \newcommand{\diag}{{\rm diag}}

110: \newcommand{\trace}{{\rm trace}}

111: \newcommand{\tr}{{\rm tr}}

112: \newcommand{\w}{\wedge}

113: \newcommand{\dint}{\int\!\!\!\int}

114: \newcommand{\lt}{\left}

115: \newcommand{\rt}{\right}

116: \newcommand{\dist}{{\rm dist}}

117: 

118: \newcommand{\policy}{{u}}

119: \def\OBdry{\partial_o}

120: \def\CBdry{\partial_c}

121: \newcommand{\Sfrac}[2]{{{#1}\slash {#2}}}

122: 

123: \title{

124: An escape time criterion for queueing networks:

125: Asymptotic risk-sensitive control via differential games%

126: \thanks{This

127: research  was supported in part by the United States---Israel Binational

128: Science Foundation (BSF 1999179)}}

129: \author{Rami Atar\footnote{Electrical Engineering,

130: Technion---Israel Institute of Technology,

131: Haifa 32000, Israel. Research of this author also supported

132: in part by the fund for promotion of research at the Technion.},

133: Paul Dupuis\footnote{Lefschetz Center for Dynamical Systems,

134: Brown University,

135: Division of Applied Mathematics,

136: Providence,  R.I.\  02912. 

137: Research of this author also supported in part by

138: the National

139: Science Foundation (NSF-DMS-0072004, NSF-ECS-9979250) and the Army Research Office 

140: (DAAD19-99-1-0223).}

141: \ and Adam Shwartz\footnote{Electrical Engineering,

142: Technion---Israel Institute of Technology,

143: Haifa 32000, Israel. Research of this author also supported

144: in part by INTAS grant 265, and in part by the fund for promotion

145: of research at the Technion.}\\[.2in]

146:  }

147: 

148: %

149: \date{February 2, 2003}

150: 

151: \maketitle

152:  

153: \begin{abstract}

154: We consider the problem of risk-sensitive control of a stochastic network.

155: In controlling such a network,

156: an escape time criterion can be useful if one wishes

157: to regulate the occurrence of large buffers and buffer overflow. 

158: In this paper a risk-sensitive escape time criterion is formulated,

159: which in comparison to the ordinary escape time criteria penalizes exits which occur on short time intervals more heavily.

160: The properties of the risk-sensitive problem are studied in the large buffer limit,

161: and related to the value of a deterministic differential game with constrained dynamics.

162: We prove that the game has value, and that the value

163: is the (viscosity) solution of a PDE.

164: For a simple network, the value is computed,

165: demonstrating the applicability of the approach.

166: \end{abstract}

167: 

168: \section{Introduction}

169: 

170: In this paper we consider a problem of risk-sensitive control

171: (or rare event control) for queueing networks.  

172: The network includes

173: servers that can offer service to two or more

174: classes of customers, and a choice must be made

175: regarding which classes to offer service at each time.

176: We study a stochastic control

177: problem in which this choice is regarded as the control, and where

178: the cost is a risk-sensitive version of the time to

179: escape a bounded set. Hence, fixing $c>0$,

180: and denoting by $\s$ the time when the queue-lengths process

181: first exits a given domain, we consider $E_xe^{-c\s}$ as a criterion

182: to be minimized.

183: Such a criterion penalizes short exit times more heavily

184: than ordinary escape time criteria

185: (such as $E_x\s$, a criterion to be maximized).

186: There are at least two motivations for the use of such criteria when designing policies for the control of a network.

187: The first is that in many communication networks system performance is measured in terms of rare event probabilities (e.g., probabilities of data loss or excessive delay).

188: The second motivation follows from the connection between risk-sensitive controls and robust controls.

189: Indeed, as discussed in \cite{dupjampet},

190: the optimization of a single fixed stochastic network with respect to a risk-sensitive cost criteria automatically produces controls with specific and predictable robust properties.

191: In particular,

192: these controls give good performance for a family of perturbed network models (where the perturbation is around the design model and the size of the perturbation is measured by relative entropy), and with respect to a corresponding ordinary (i.e., not risk-sensitive) cost.

193: 

194: In many problems,

195: one considers the limit of the risk-sensitive problem as a scaling parameter of the system converges,

196: in the hope that the limit model is more tractable.

197: We follow the same approach here,

198: and show that the normalized costs in the risk-sensitive problems converge to the value function of a differential game with constraints.

199: As is well known, 

200: the convergence analysis is closely related to the large deviation properties of the sequence of controlled processes.

201: An interesting feature in the setting of stochastic networks is that the asymptotic analysis of a sequence of {\it controlled} networks is in many ways simpler than the analogous asymptotic analysis of a sequence of {\it uncontrolled} networks.

202: For example,

203: if one were to fix a particular state feedback service policy at each station,

204: then the calculation of the large deviation asymptotics is very difficult.

205: In contrast,

206: it turns out that calculation of the large deviation asymptotics of the optimally controlled network is quite feasible.

207: This is largely due to the fact that a {\it fixed} service policy invariably includes some {\it state discontinuities}.

208: For example,

209: a priority policy switches drastically when the highest priority queue empties.

210: When the policy is left as a parameter that is to be optimized these sharp discontinuities are not dealt with directly,

211: since the control and the large deviation behavior are identified simultaneously.

212: The situation is analogous to one found in the control of unconstrained processes such as diffusions.

213: If a fixed nonsmooth feedback control is considered then large deviation asymptotics are generally intractable,

214: but when the combined large deviation and optimal control problem is considered, much is possible \cite{dupmce}.

215: 

216: For simplicity, we restrict in this paper to a class of Markovian networks,

217: and consider just one simple cost structure.

218: Much more general statistical models can be treated with similar arguments,

219: as can a more general cost.  

220: A more fundamental restriction is on the routing in the network.

221: We assume a re-entrant line structure, so 

222: that the input streams follow a fixed route through the network--we do not allow either randomized or controlled routing.

223: Relaxing the last conditions

224: leads to a problem that is significantly more difficult to analyze,

225: and would require a considerable extension of the results we prove.

226: 

227: The deterministic game that is associated with the limit stochastic control

228: problem involves two players. One player

229: allocates service in a way analogous to the control in

230: the stochastic control problem, and the other player perturbs

231: the service and arrival rates. The cost is expressed in terms

232: of the large deviation rate function for the underlying arrival and service 

233: processes, cumulated up to the time the dynamics exit the domain.

234: Heuristically, the first player

235: identifies those classes it is most worthwhile to allocate

236: service to, so as to delay the escape as much as possible and thereby maximize the cost.

237: The player who selects the perturbed rates attempts to

238: minimize the cost by driving it out of the domain,

239: while paying a cost for perturbing the rates.

240: 

241: Our main result states that as the scaling parameter of the system

242: converges, the value for the stochastic control problem

243: converges to the value of the game. By way of proving the

244: result, we also show that the Hamilton-Jacobi-Bellman

245: equation associated with the game has a unique (Lipschitz continuous)

246: viscosity solution.

247: 

248: Several works have considered

249: problems of optimal exit probabilities in the context of controlled

250: diffusion processes, in the asymptotically small white noise

251: intensity regime.

252: Fleming and Souganidis \cite{flesou} use viscosity

253: solutions techniques to study a controlled diffusion where the

254: control enters in the drift coefficient. Dupuis and Kushner \cite{dupkus}

255: extend their results to

256: the case where the diffusion coefficient is possibly degenerate.

257: Their technique relates the stochastic control problem

258: to the game in a more direct way, using time discretization,

259: without involving PDE analysis.

260: The stochastic control problem studied in the current paper

261: has the property that the jump rates in certain directions

262: (those that correspond to services, not to arrivals)

263: can be controlled to assume arbitrarily low values, including zero.

264: It appears to be a more subtle problem than the ones in the above

265: cited papers, in that it is analogous to a

266: controlled diffusion problem where the control

267: enters also in the {\em diffusion} coefficient, and where no uniform

268: non-degeneracy condition is assumed. This kind of degeneracy makes

269: it difficult to apply the time discretization idea of \cite{dupkus}.

270: The main idea of \cite{dupkus}, in which one directly relates

271: the control problem to the game, is still fruitful in the

272: current setting. Following this approach,

273: we relate the limit inferior [resp., superior] of the 

274: asymptotic value for the control problem to the upper [resp., lower]

275: value of the game. However, showing that the game has value

276: and thereby obtaining the full convergence

277: result for the control problem requires a PDE analysis.

278: 

279: The PDE analysis uses viscosity solutions methods.

280: There are three types of boundary conditions associated with the PDE:

281: Neumann, Dirichlet, and ``state space constraint.''

282: The first two types of boundary conditions correspond in the game

283: to the nonnegativity constraint on queue lengths and to stopping upon exit from the domain, respectively.

284: The third type of boundary condition arises when there are portions

285: of the boundary where exit can be blocked unilaterally by

286: one of the players, and it is optimal for it to do so.

287: It is well known since Soner \cite{son} that such a scenario 

288: leads to the last boundary condition mentioned above.

289: Combining techniques of \cite{atadup2} and \cite{son},

290: we prove uniqueness of viscosity solutions for the PDE and show that

291: the game's upper and lower values are viscosity solutions,

292: thus establishing existence of value.

293: The trivial but crucial fact used in the uniqueness proof is

294: that the Isaacs condition holds (equation (\ref{eq:isaacs})).

295: 

296: As an example, we analyze a case

297: where the domain is a hyper-rectangle,

298: and where the network consists of one server and many

299: queues, each customer requiring service only once.

300: We find an explicit solution to the corresponding PDE,

301: assuming the parameter $c$ is large enough.

302: This is only an initial result in this direction,

303: but it shows that explicit solutions can be found.

304: The solution turns out to be of particularly simple form

305: (see equation (\ref{eq:form})). The optimal service discipline

306: stemming from the solution corresponds to giving priority

307: to class $i$ whenever the state of the system is within

308: a subset $G_i$ of the domain. The partitioning of the domain

309: into subsets has a simple structure too (see Figure \ref{fig:example}

310: in Section \ref{sec:example} for an example in two dimensions).

311: See \cite{atadupshw} for explicit solutions in the case of tandem queues,

312: as well as identities relating the perturbed rates with

313: the unperturbed ones in a more general network.

314: 

315: There is relatively little work on risk-sensitive and robust control of networks.

316: Ball et.\ al. have considered a robust formulation for network problems arising in vehicular traffic \cite{balday1,balday2},

317: and have explicitly identified the value function in certain instances.

318: Although their model is similar to ours in that the network dynamics are modeled via a Skorokhod Problem,

319: many other features, most notably the cost structure, are qualitatively different.

320: In addition,

321: the model they consider is not naturally related to a risk-sensitive control problem for a jump Markov model of a network.

322: 

323: 

324: 

325: The organization of the paper is as follows.

326: Section \ref{sec:setting} introduces the network and the stochastic

327: control problem, describes a key tool in our analysis, namely

328: the Skorokhod Problem (SP), introduces the

329: differential game, and states the main result.

330: Section~\ref{sec:limit} establishes

331: the relation between the control problem asymptotics

332: and the game's upper and lower values.

333: Characterization of the upper and lower values of the game

334: as viscosity solutions of a PDE, as well as uniqueness for this PDE

335: are established in Section \ref{sec:pde}.

336: Section~\ref{sec:example} presents an example, and the paper concludes with Section~\ref{sec:lemmas}, which gives the proofs of several lemmas.

337: Throughout the paper,

338: numbering such as Lemma $a.b$ refers to the $b$th item of Lemma $a$.

339: 

340: 

341: 

342: 

343: 

344: \section{Problem setting and the main result}\label{sec:setting}

345: 

346: \noi\uu{\bf The queueing network control problem.}

347: We consider a system with $J$ customer classes, and without loss 

348: assume that each class is identified with a queue at one of $K$ servers.

349: Each server provides service to at least one class.

350: Thus if $C(k)$ denotes the set of classes that are served by $k$,

351: then the control determines who receives service effort at server $k$

352: from among $i \in C(k)$. In particular, the sets $C(k)$, $k=1,\ldots, K$ are disjoint, 

353: with $\cup_k C(k)=\{1,\ldots,J\}$.

354: The state of the network is the vector of queue lengths, denoted

355: by $X$.

356: After a customer of class $i$ is served, it turns into a customer of

357: class $r(i)$, where $i=0$ is used to denote the ``outside.''

358: We let $ e_j $ denote the unit vector in direction $j$ and set $ e_0 = 0 $

359: so that following service to class $j$ the state changes by

360: $e_{r(j)} - e_j$.

361: The control will be described by the vector $u=(u_1,...,u_J)$,

362: where $u_i=1$ if class $i$ customers  are given service and $u_i=0$

363: otherwise. Since service can be given at any moment

364: to only one class at each station, the control vector must satisfy $\sum_{i\in C(k)}u_i\le1$ for

365: each $k$.

366: We next consider the scaled process $X^n$ under the scaling

367: which accelerates time by a factor of $n$ and shrinks space by

368: the same factor.

369: We are interested in a

370: risk-sensitive cost functional that is associated with

371: exit from a bounded set.

372: Let $G$ be a bounded subset of $\RJP $ that contains the origin

373: (additional assumptions on $G$ are given in Condition \ref{cond:G}).

374: Define

375: \[

376: \sigma^n \doteq \inf \{t:X^n(t) \not \in G \}.

377: \]

378: Then the control problem is to minimize the cost

379: $E_xe^{-nc\sigma^n}$, 

380: where $E_x$ denotes expectation starting

381: from $x$, and $c>0$ is a constant.

382: With this cost structure ``risk-sensitivity'' means that atypically short exit times are weighted heavily by the cost.

383: A ``good'' control will avoid such an event with high

384: probability. 

385: The significance from the point of view of stabilization of the

386: system is clear.

387: (See also \cite{dupmce} for the robust interpretation).

388: 

389: A precise description of the stochastic control problem is as follows.

390: Let $G^n \doteq n^{-1}\Z_+^J \cap G$. 

391: Define

392: $$

393:  U \doteq \left\{(u_i), i=1,\ldots,J:\sum_{i\in C(k)} u_i\le1, k=1,\ldots,K,

394: u_i\ge0, i=1,\ldots,J\right\}.

395: $$

396: For $u\in U$ and $f:\Z^J\to\R$ let

397: \be\label{eq:gen_constr}

398: \til{\cal L}^{u}f(x)

399:  \doteq \sum_{j=1}^{J}\lambda _{j}\left[ f(x+e_{j})-f(x)\right]

400: +\sum_{j=1}^J u_j \mu_j 1_{\{x+\tilde v_j\in\Z_+^J\}}

401:  \lt[ f(x+\tilde v_j)-f(x)\right],

402: \ee

403: where $\til v_j=e_{r(j)}-e_j$.

404: It is assumed that for each $i$, $\la_i\ge0$,

405: while $\mu_i>0$.

406: For each $n\in\N$ consider the scaling defined by

407: \be\label{eq:gens}

408: \til\calL^{n,u}f(x) \doteq n\til\calL^ug(nx),

409: \ee

410: where $f:n^{-1}\Z^J\to\R$ and $g(\cdot)=f(n^{-1}\cdot)$.

411: A {\em controlled Markov process} starting from $x$ will consist of a 

412: complete filtered

413: probability space $(\Om,\calF,(\calF_t),P_x^{n,u})$,

414: a state process $X^n$ taking values in  $G^n$ that is continuous from the right

415: and with limits from the left,

416: a control process $u$ taking values in  $U$,

417: such that $X^n$ is adapted to $\calF_t$,

418: $u$ is measurable and adapted to $\calF_t$,

419: $P_x^{u,n}(X^n(0)=x)=1$,

420: and for every function $f:n^{-1}\Z^J\to\R$

421: \[

422: f(X^n(t))-\int_0^t\til\calL^{n,u(s)}f(X^n(s))ds

423: \]

424: is an $\calF_t-$martingale.

425: $E_x^{n,u}$ denotes expectation with respect to $P_x^{n,u}$.

426: For a parameter $c>0$, the value function for the stochastic control problem is defined by

427: \be\label{eq:control}

428:   V^n(x) \doteq -\inf n^{-1}\log E_x^{u,n}e^{-nc\sig_n},

429:   \quad x\in G^n.

430: \ee

431: In this definition the infimum is over all controlled Markov processes.

432: 

433: A measurable function $u(x,t)$, $u:G^n\times[0,\infty)\to U$

434: is said to be a {\em feedback control}.

435: We will make use of two well known facts:

436: to each feedback control there corresponds a controlled Markov process with $u(t)=u(X^n(t),t)$,

437: and in the definition of the value function the infimum can be restricted to feedback controls.

438: 

439:  

440: In the formulation just given we allow the maximizing player to choose a control from the convex set $U$.

441: This is a relaxed formulation,

442: which allows the server to simultaneously split the effort between 2 or more customer classes.

443: An alternative control space that is more natural in implementation consists of only the vertices of $U$,

444: in which case the server can only server one class at a time.

445: In a general game setting,

446: the distinction between such ``relaxed'' and ``pure'' control spaces can be significant.

447: However,

448: in the present setting it will turn out that the value is the same for both cases.

449: This is essentially due to the fact that the game arises from a risk-sensitive control problem,

450: which imposes additional structure on the game,

451: and will be further commented on below.

452: 

453: \noi\uu{\bf Dynamics via the Skorokhod Problem.}

454: Our main goal will be to study the asymptotics of $V^n$,

455: and in particular, to show that they are

456: governed by the value of a deterministic differential

457: game. In order to define the dynamics of this game we first need a

458: formulation of the {\em Skorokhod Problem} (SP). We 

459: give here the simplest formulation which covers our needs.

460: The reader is referred to \cite{dupram23} for a more general framework.

461: Let 

462: \[

463:  D_{+}([0,\infty ):\RJ ) \doteq \left \{\psi\in D([0,\infty ):\RJ ):

464:    \psi(0)\in \RJP \right \},

465: \]

466: where $D([0,\infty ):\RJ )$ is the space of left continuous functions with right

467: hand limits, endowed with the uniform on compacts topology.

468: When restricting to continuous functions

469: we replace ``$D$'' with ``$C $''.

470: Let a set of vectors $\{ \gamma_i , \ i=1, \ldots , J \}$

471: be given and set $I(x) \doteq \{i:x_i=0\}$. 

472: For each point $x$ on $ \partial \RJP $ -- the boundary of $\RJP $ -- let

473: \[

474: d(x) \doteq \left\{ \sum_{i \in I(x)}a_i \gamma_i: a_i \geq 0, \left\|

475: \sum_{i \in I(x)}a_i \gamma_i \right\| = 1 \right\}.

476: \]

477: The Skorokhod Map (SM) assigns to every path $\psi\in

478: D_{+ }([0,\infty ) :\RJ )$ a path $\phi$ that starts at $\phi(0)=\psi(0)$,

479: but is constrained to $\RJP $ as follows.

480: If $\phi$ is in the interior of $ \RJP $ then the evolution of $\phi$

481: mimics that of $\psi$,

482: in that the increments of the two functions are the same until $\phi$ hits

483: $ \partial \RJP $.

484: When $\phi$ is on the boundary a constraining ``force'' is applied to keep

485: $\phi$ in the domain,

486: and this force can only be applied in one of the directions $d(\phi(t))$,

487: and only for $t$ such that $\phi(t)$ is on the boundary.

488: The precise definition is as follows.  

489: For $\eta\in

490: D([0,\infty ) : \RJ )$ and $t\in  [0, \infty)$ we let $|\eta|(t)$ denote the total

491: variation of

492: $\eta$ on $[0,t]$ with respect to the Euclidean norm on $ \RJ $.

493: 

494: \begin{definition}\label{def:sp}

495: Let $\psi\in D_{+}([0,\infty ) : \RJ )$ be given. Then $(\phi,\eta)$ solves

496: the SP for $\psi$ (with respect to $ \RJP $ and $\gamma_i, i=1,...,J $) if

497: $\phi(0)=\psi(0)$, and

498: if for all $t\in[0,\infty ) $

499: \begin{enumerate}

500: \item $\phi(t)=\psi(t)+\eta(t)$,

501: \item $\phi(t)\in \RJP $,

502: \item $|\eta|(t)<\infty$,

503: \item $|\eta|(t)=\int_{[0,t]} 1_{\{\phi(s)\in\partial  \RJP \}}

504: d|\eta|(s)$,

505: \item There exists a Borel measurable function

506:       $\gamma : [0,\infty ) \to \RJP $ such that $d|\eta|$-almost everywhere

507:       $\gamma(t)\in d(\phi(t))$, and such that

508: $$ \eta(t) = \int_{[0,t]}\gamma(s)d|\eta|(s).

509: $$

510: \end{enumerate}

511: \end{definition}

512: Under a certain condition on $\{ \gamma_i \}$ (known in the literature as the

513: {\it completely}-${\cal S}$ condition \cite{reiwil}), it is known

514: that solutions to the SP exist in all of $D_{ +}([0,\infty): \RJP )$.

515: Under further conditions 

516: (namely, {\em existence of the set $B$} -- see~\cite{harrei,dupish1,dupram23} and also Lemma \ref{lem:mu} below),

517: it is known that the Skorokhod Map is Lipschitz continuous, and consequently

518: the solution is unique.

519: Denoting the map $\psi\mapsto\phi$ in Definition \ref{def:sp} by

520: $\Gamma$, the Lipschitz property states that there is a constant

521: $K_1 $ such that

522: \be\label{sp:lip}

523:  \sup_{t\in[0,\infty)}\|\Gamma(\psi_1)(t)-\Gamma(\psi_2)(t)\|

524:  \le K_1 \sup_{t\in[0,\infty)}\|\psi_1(t)-\psi_2(t)\|,\quad

525:  \psi_1,\psi_2\in D_{ + }([0,\infty): \RJ ).

526: \ee

527: The SP that will be considered here is the one for which

528: $\gamma_i=e_i-e_{r(i)} = - {\tilde v}_i $. For this problem, the following is well known.

529: \begin{theorem}[\cite{harrei,dupram23}]\label{th:SP}

530: The SP associated with the domain $ \RJP $ and the constraint vectors

531: $\gamma_i, i=1,...,J $

532: possesses a unique solution, and the Skorokhod Map is

533: Lipschitz continuous on the space $D_{ +}([0,\infty): \RJP )$.

534: Moreover, the Skorokhod Map takes 

535: $ C_{+}([0,\infty): \RJ ) $ into $ C_{+}([0,\infty): \RJ ) $,

536: and therefore $ \Gamma (\phi ) $ is continuous if $ \phi $ is.

537: \end{theorem}

538: We next define a constrained ordinary differential equation.

539: As is proved (in greater generality) in \cite{dupish1},

540: one can define a projection $\pi: \RJ  \goto  \RJP $ that is consistent with

541: the constraint directions $\{\gamma_i, i=1,...,J \}$,

542: in that $\pi(x) = x$ if $x \in  \RJP $,

543: and if $x \not \in  \RJP $ then $\pi(x)-x = \alpha r$,

544: where $\alpha \geq 0$ and $r \in d(\pi(x))$.

545: With this projection given,

546: we can now define 

547: for each point $x \in \partial  \RJP $ and each $v \in  \RJ $

548: the {\it projected velocity}

549: \[

550: \pi(x,v) \doteq \lim_{\Delta \downarrow 0} \frac{\pi(x+\Delta

551: v)-\pi(x)}{\Delta}.

552: \]

553: For details on why this limit is always well defined and further properties

554: of the projected velocity we refer to \cite[Section 3 and Lemma

555: 3.8]{buddup}.

556: Let $v:[0,\infty) \goto  \RJ $ have the property  that each component of

557: $v$ is integrable over each interval $[0,T]$, $T<\infty$.

558: Then the ODE of interest takes the form

559: \be

560: \label{eqn:ODE}

561: \dot \phi(t) = \pi(\phi(t),v(t)), \quad \phi (0) = \phi_0 \in \RJP .

562: \ee

563: An absolutely continuous function $\phi:[0,\infty) \goto  \RJP $ is a solution

564: to (\ref{eqn:ODE}) if  the equation is satisfied a.e.\ in $t$.

565: By using the regularity properties (\ref{sp:lip}) of the associated Skorokhod Map

566:  and because of the

567: particularly simple nature of the right hand side,

568: one can show that $\phi$ solves (\ref{eqn:ODE}) if and only if $\phi$ is

569: the image of $\psi(t) \doteq \int_0^t v(s)ds +x$ under the Skorokhod

570: Map, and thus all the standard qualitative properties (existence and

571: uniqueness of solutions, stability with respect to perturbations, etc.)

572: hold \cite{dupish1,dupnag}.

573: 

574: As mentioned above, the SP formulation will be our means of defining

575: the dynamics of a deterministic game. Before discussing this game,

576: let us show how the same formulation is also useful for

577: the stochastic control problem defined earlier in this section.

578: First, since $\til v_j = e_{r(j)}-e_j = - \gamma_j$, it is easy to verify that for the particular SP considered here

579: $\pi(x,v)=v1_{x+v\in\Z_+^J}$ for all $x\in\Z_+^J$ and

580: $v\in\{\til v_j:j=1,\ldots,J\}$.

581: Therefore the generator $\til\calL^{u}$ of (\ref{eq:gen_constr})

582: can also be written as

583: $$

584:  \til\calL^uf(x)=\sum_{j=1}^J\la_j[f(x+e_j)-f(x)]+

585:  \sum_{j=1}^Ju_j\mu_j[f(x+\pi(x,\tilde v_j))-f(x)].

586: $$

587: A measurable function $u(t)$, $u:[0,\infty)\to U$

588: is said to be an {\em open loop control}. 

589: Note that this control has no state feedback.

590: When $u$ is an open loop control,

591: it is possible to write the corresponding controlled process $X$ as $\Gamma(Y)$.

592: The process $Y$, which will be called

593: {\em the unconstrained controlled process},

594: is a controlled Markov process with a simpler structure.

595: To be precise, let

596: $$

597:  \til{\cal L}_0^{u}f(x)

598:  =\sum_{j=1}^{J}\lambda _{j}\left[ f(x+e_{j})-f(x)\right]

599:  +\sum_{j=1}^Ju_j\mu_j\left[ f(x+ \tilde v_j)-f(x)\right],

600: $$

601: and let $\til\calL_0^{n,u}$ be defined analogously to (\ref{eq:gens}).

602: A controlled Markov process

603: $X^n$ on $G^n$ [respectively, $Y^n$ on $n^{-1}\Z^J$] is defined as before, 

604: but now using the

605: generator $(\calL^{n,u}f)(t)=(\til\calL^{n,u(t)}f)(x)$

606: [resp., $(\calL_0^{n,u}f)(t)=(\til\calL_0^{n,u(t)}f)(x)$].

607: The simplification that the SP introduces

608: is that if $u$ is an open loop control, and if $Y^n$ is a

609: controlled Markov process corresponding to $\calL_0^{n,u}$

610: on $(\Om,\calF,(\calF_t),P_x^{n,u})$, then $X^n=\Gamma(Y^n)$

611: is a controlled Markov process corresponding to $\calL^{n,u}$

612: on the same filtered probability space.

613: The role played by the SP in relating constrained and

614: unconstrained processes is exhibited here in a simple fashion,

615: for introductory purposes. We will, in fact, use it in

616: a slightly more complicated setting later on in Lemma \ref{lem:xbar} and Lemma 

617: \ref{lem:xbar2}.

618: 

619: \noi\uu{\bf A differential game.}

620: In this paper,

621: we prove that the value function $V^n(x)$ for a stochastic control problem associated with our queueing network model is approximately equal (for large $n$) to the value function of a related differential

622: game.

623: In addition,

624: the dynamics of this game are defined in terms of an associated SP. 

625: Before introducing the game

626: formally we explain why this is to be expected.

627: In a problem with no control, the exponential decay rate

628: of quantities such as $Ee^{-cn\sig_n}$ is

629: given in terms of the sample path

630: large deviation rate function associated with the process, which in

631: turn can be expressed in terms of the rate function for the Poisson

632: primitives that drive the model.

633: This is supported by the

634: well known Laplace's principle \cite{dupell}. Heuristically,

635: one thinks of the rate function as a cost paid for changing the measure

636: so as to make the rare event of exiting

637: on short time interval a probable event.

638: Laplace's principle asserts that the decay rate can be expressed

639: as the solution to a deterministic optimization problem

640: involving the cost $-c\sig$ combined with the cost of changing the measure:

641: cf.\ \cite[Eq.\ (5.20)--(5.23)]{shwe}.

642: When the stochastic model involves optimal control, there is one

643: more variable to optimize over in the limit, and this results in a game.

644: The game's deterministic dynamics are the natural law of large numbers limit under the changed measure.

645: Boundary constraints and constraining meachanisms which are present in the prelimit model are represented in the limit model by the SP.

646: The cost for the game involves

647: the large deviation rate function for the Poisson primitives,

648: the time till the dynamics exits the domain, and the parameter $c$.

649: 

650: We thus consider a zero sum game involving two players.

651: One (which we call the maximizing player)

652: selects the service allocation and attempts to maximize.

653: The other (called the minimizing player)

654: chooses the perturbed arrival and service rates and attempts to minimize.

655: Throughout, the perturbed rates will be denoted by an overbar,

656: as in $\bar\la_i,\bar\mu_i$.

657: 

658: Recall that for $u\in U$, $u_i$

659: stands for the fraction of service effort given to class $i$.

660: The control space for the maximizing player is

661: $$ \bar U\doteq \{u:[0,\infty)\to U\ ;\ u \mbox{ is measurable}\}.

662: $$

663: Let $l:\R\to\R_+\cup\{+\infty\}$

664: be defined as

665: $$

666:  l(x)\doteq \lt\{\begin{array}{ll}x\log x-x+1 & x\ge0,  \\ +\infty & x<0,

667:      \end{array}\rt.

668: $$

669: where $0\log0\doteq0$.

670: Denoting $M =[0,\infty)^{2J}$, the control space for the minimizing

671: player will be

672: \begin{equation}

673: \label{embar}

674:  \bar M =\{m=(\bar\la_1,\ldots,\bar\la_J,\bar\mu_1,\ldots,\bar\mu_J):

675: [0,\infty)\to M;

676: \ \mbox{$m$ is measurable, $l\circ m$ is  locally integrable}\}.

677: \end{equation}

678: For $u\in U$ and $m\in M$ define

679: \[

680: v(u,m) \doteq \sum_{j=1}^J \bar \lambda_jv_j + \sum_{i=1}^J

681: u_i \bar\mu_i \tilde v_i,

682: \]

683: where $v_j=e_j$, and as before $\tilde v_i=e_{r(i)}-e_i$.

684: Then the dynamics are given by

685: \[

686: \lt\{\begin{array}{ll} \dot \phi(t) = \pi (\phi(t),v(u(t),m(t))), \\ 

687:  \phi(0)=x.

688:  \end{array}\rt.

689: \]

690: To define the cost for the game, let

691: $\rho:U\times M\to\R_+\cup\{+\infty\}$ be

692: $$ \rho(u,m)\doteq \sum_{i=1}^{J}\lambda _{i}l\left( \frac{\bar{\lambda}_{i}}{

693: \lambda _{i}}\right) +\sum_{i=1}^{J}u_i\mu _il\left( \frac{\bar{\mu}_i}

694: {\mu _i}\right).

695: $$

696: By convention, if $\la_i=0$ and $\bar\la_i>0$ for some $i$, we let

697: $\rho=\infty$ (recall that by assumption, $\mu_i>0$).

698: Let the exit time be defined by

699: $$

700:  \sig\doteq \inf\{t:\phi(t)\not\in G\}.

701: $$

702: With $c > 0$ as in~(\ref{eq:control}),

703: the cost is given by

704: $$ C(x,u,m)=\int_0^\sig [c+\rho(u(t),m(t))]dt.

705: $$

706: 

707: As in \cite{ellkal} we need the notion

708: of strategies. We endow both $\bar U$ and $\bar M$ with the metric

709: $\til\rho(\om_1,\om_2)=\sum_n2^{-n}(\int_0^n

710: |\om_1(t)-\om_2(t)|dt \w1)$,

711: and with the corresponding Borel $\s$-fields.

712: A mapping $\al:\bar M \to\bar U$ is called a

713: {\em strategy for the maximizing player}

714: if it is measurable

715: and if for every $m,\tilde m\in\bar M$ and $t>0$ such that

716: $$ m(s)=\tilde m(s) \mbox{ for a.e.\ } s\in[0,t],

717: $$

718: one has

719: $$ \al[m](s) = \al[\tilde m](s) \mbox{ for a.e.\ } s\in[0,t].

720: $$

721: In an analogous way, one defines a mapping $\beta:\bar U\to \bar M$

722: to be a {\em strategy for the minimizing player}.

723: The set of all strategies for the maximizing [resp., minimizing]

724: player will be denoted by $A$ [resp., $B$].

725: The lower value for the game is defined as

726: \[

727: V^-(x) = \inf_{\beta\in B} \sup_{u\in\bar U}C(x,u,\beta[u]),

728: \]

729: and the upper value as

730: $$

731: V^+(x)=\sup_{\al\in A}\inf_{m\in\bar M}C(x,\al[m],m).

732: $$

733: To avoid confusion, we remark that despite the terms ``upper'' and

734: ``lower'' value,  it is not in general obvious that $V^-\le V^+$.

735: 

736: 

737: \noi\uu{\bf Main result.}

738: We make the following assumption on the domain $G$. Let

739: $$

740:  \calJ_+\doteq\{i\in\{1,\ldots,J\}:\la_i>0\}.

741: $$

742: 

743: \begin{condition}\label{cond:G}

744: We assume that the domain $G$ satisfies one of the following.

745: \begin{enumerate}

746: \item

747: $G$ is a rectangle given by

748: $$ G=\{(x_1,\ldots,x_J): 0\le x_i<z_i, i\in\calJ_+;\ 0\le x_j\le z_j,

749: j\not\in\calJ_+\},

750: $$

751: for some $z_i>0$, $i=1,\ldots,J$.

752: \item

753: $G$ is simply connected and bounded, and given by

754: $$

755:  G=\bigcap_{i\in\calJ_+}G_i,

756: $$

757: where for $i\in\calJ_+$, we are given positive Lipschitz functions

758: $\phi_i:\R^{J-1}\to\R$, and

759: $$

760:  G_i=\{(x_1,\ldots,x_J)\in \RJP : 0\le x_i<\phi_i(x_1,\ldots,x_{i-1},

761: x_{i+1},\ldots,x_J)\}.

762: $$

763: \end{enumerate}

764: \end{condition}

765: This condition covers many typical constraints one would consider on buffer size, 

766: including separate constraints on individual queues (Condition \ref{cond:G}.1) and one constraint on the sum of the queues (Condition \ref{cond:G}.2).

767: 

768: The shape of the domain is simpler in   Condition \ref{cond:G}.1,

769: in that it is restricted to a hyper-rectangle.

770: On the other hand, it is also possible under this condition for the

771: maximizing player to unilaterally prevent an exit 

772: through a certain portion of $\pl G\setminus\pl\R_+^J$.

773: Although it is in principle possible that the dynamics

774: could exit through this portion of the boundary, it will always be optimal for

775: the maximizing player to not allow it.

776: Consider the simple network illustrated in Figure \ref{fig1}.

777: The maximizing player can prevent exit through the dashed portion of the boundary simply by stopping service at the first queue. 

778: As a consequence,

779: there are in general three different types of boundary--the constraining boundary due to non-negativity constraints on queue length,

780: the part of the boundary where exit can be blocked,

781: and the remainder.

782: These three types of boundary behavior will result,

783: in the PDE analysis, in three types of boundary conditions.

784: We now define the three portions of the boundary.

785: Under Condition \ref{cond:G}.1, let

786: $$

787: \pl_cG=\{(x_1,\ldots,x_J)\in G: x_j=z_j,\,\mbox{some}\,j\not\in\calJ_+\}.

788: $$

789: For notational convenience, we let $\pl_cG=\emptyset$ under

790:   Condition \ref{cond:G}.2. In both cases we then set

791: $$

792: \pl_oG=\pl G\setminus G, \qquad

793: \pl_+G=(G\cap\pl\R_+^J)\setminus\pl_cG.

794: $$

795: Note that in both cases, $\pl_cG$, $\pl_oG$ and $\pl_+G$

796: partition $\pl G$. Also, $\pl_cG\subset G$ while $\pl_oG\subset

797: G^c$. As usual, we will denote $G^o=G\setminus\pl G$

798: and $\bar G=G\cup\pl G$.

799: $ \pl_cG $ is the part of the boundary were the

800: maximizing player can prevent the dynamics from exiting,

801: and $ \pl_oG $ is the part where it can not.

802: Finally, it will be convenient to denote

803: $$

804: \pl_{co}G=\pl_oG\cup\pl_cG.

805: $$

806: 

807: 

808: \begin{figure} % export: 30 percent.

809: \centerline{

810: \begin{tabular}{cc}

811: \psfig{file=fig_net_1.eps}\qquad\qquad\qquad

812: \psfig{file=fig_ex_1.eps}

813: \end{tabular}

814: }

815: \caption{

816: A simple queueing network, a rectangular domain and three

817: types of boundary.

818: Full line: $~$ \hfill $\pl_+G$, dashed line: $\pl_cG$,

819: and dotted line: $\pl_oG$

820: \hfill $~$ }

821: \label{fig1}

822: \end{figure}

823: 

824: 

825: Our main result is the following.

826: \begin{theorem}\label{th:main}

827: Let Condition \ref{cond:G} hold.

828: Then $V^+=V^-\doteq V$ on $G$. Moreover, if $x_n\in G^n$,

829: $n\in\N$ are such that $x_n\to x\in G$, then

830: $\lim_{n\to\infty}V^n(x_n) = V(x)$.

831: \end{theorem}

832: 

833: \remark 

834: A stronger form of the convergence statement in fact holds.

835: Namely,

836: $$

837:  \limsup_{\eps\downarrow0}

838:  \limsup_{n\to\infty}\sup\{|V^n(x)-V(y)|: x\in G^n,y\in G, |x-y|\le\eps\}

839:  =0.

840: $$

841: This is an immediate consequence of Theorem \ref{th:main}

842: and the fact that for each $n$, $V^n$ is Lipschitz on $G^n$,

843: with a constant that does not depend on $n$ (Lemma \ref{lem:unif}).

844: \qed

845: 

846: The proof is established in two major steps.

847: Step 1 will be an immediate consequence of the main results of

848: Section \ref{sec:limit},

849: and Step 2 will follow from Section \ref{sec:pde}.

850: 

851: \noi{\em Step 1.} We define a version of the game, technically

852: easier to work with, in which

853: all perturbed rates ($\bar\la_i,\bar\mu_i$) are bounded

854: by $b<\infty$. The corresponding upper and lower values,

855: defined analogously, are denoted by $V^{b,+}$ and $V^{b,-}$.

856: Then we show that for all $b$ large enough

857: (cf.\ Theorem \ref{th:limit})

858: $$

859:  V^{b,+} (x)\le\liminf_{n\to\infty}V^n(x_n)\le\limsup_{n\to\infty}V^n(x_n)

860:  \le V^{b,-} (x) .

861: $$

862: 

863: \noi{\em Step 2.}

864: We show that for $b$ large,

865: $V^{b,+} =V^{b,-} $ on $G$. To this end, we formulate a PDE

866: for which we show that uniqueness of (Lipschitz) viscosity solutions

867: holds (Theorem \ref{th:unique}),

868: and also show that both $V^{b,+}$ and $V^{b,-}$ are viscosity solutions

869: (Theorem \ref{th:solve}).

870: Since $ V^n (x) $ does not depend on $b$, neither do $ V^{b,\pm} (x) $.

871: Theorem \ref{th:main} follows.

872: 

873: 

874: \section{The control problem and the game}\label{sec:limit}

875: 

876: We begin by stating some basic properties of the stochastic

877: control problem and of the deterministic game.

878: The proofs of these properties are 

879: deferred to Section \ref{sec:lemmas}.

880: 

881: Consider the following generators, defined for any $u\in U$ and

882: $m\in M$, for constrained and unconstrained controlled Markov

883: processes:

884: \beaa

885: \calL^{n,u,m}f(x) &=& \sum_{j=1}^J n \bar \lambda_j \left [f \left (x+\frac{1}{n} v_j \right )-f(x) \right ]

886: + \sum_{i=1}^J n \bar \mu_i u_i\left [ f \left ( x+\frac{1}{n}\pi(x, \tilde v_i) \right )-f(x) \right ], \\

887: \calL_0^{n,u,m}f(x) 

888: &=& \sum_{j=1}^J n \bar \lambda_j \left [ f \left (x+\frac{1}{n} v_j \right )-f(x) \right ]

889: + \sum_{i=1}^J n \bar \mu_i u_i \left [f \left ( x+\frac{1}{n} \tilde v_i \right )-f(x) \right ].

890: \eeaa

891: The definition of the corresponding controlled processes will be made precise in Lemmas

892: \ref{lem:xbar} and \ref{lem:xbar2}.

893: 

894: Owing to the logarithmic transform in (\ref{eq:control}), one expects $V^n$ to satisfy an Isaacs equation \cite{flesou}.

895: In fact, $V^n$ satisfies 

896: \be\label{eq:dpe}

897: \lt\{\begin{array}{cc}

898: 0=\sup_{u\in U} \inf_{m\in M} [\calL^{n,u,m}V^n(x) +c+\rho(u,m)],

899:   & x \in G^n \\ 

900: V^n(x)=0, & x \not \in G^n.

901: \end{array}\rt.

902: \ee

903: We comment that this is also the dynamic programming equation (DPE)

904: for an associated stochastic game

905: that is related to the deterministic game

906: via a law of large numbers scaling and limit,

907: and will not be further considered in this paper.

908: 

909: 

910: \begin{lemma}\label{lem:dpe}

911: The value function $V^n$ of (\ref{eq:control}) uniquely solves

912: the DPE (\ref{eq:dpe}).

913: \end{lemma}

914: 

915: The following lemma gives a key estimate on the value function.

916: 

917: \begin{lemma}\label{lem:unif}

918: Under Condition~\ref{cond:G},

919: $ V^n (x) $ satisfies the Lipschitz property

920: on $(n^{-1}\Z_+^J) \cap \bar G$

921: with a constant that does not depend on $n\in\N$.

922: Consequently, $ \sup_{n, x \in G^n} V^n (x) < \infty $.

923: \end{lemma}

924: We comment that the above result is, in general, not valid for $V^n$

925: on $n^{-1}\Z_+^J$, since $V^n$ changes abruptly near

926: the portion $\pl_cG$ of the boundary.

927: 

928: For each fixed $u\in U$, the mapping $m\rightarrow \rho(u,m)$,

929: when restricted to $\bar \mu_i$ such that $u_i>0$, is strictly convex with compact level sets.

930: We conclude that the infimum over $m$ in the DPE

931: is achieved, 

932: and denote such a point by $m^n(x,u)$. 

933: Although part 1 of the following lemma is not used elsewhere,

934: it indicates why the Isaacs condition should hold in (\ref{eq:dpe}).

935: 

936: \begin{lemma}\label{lem:bound:m} Let Condition \ref{cond:G} hold.

937: Then

938: \ben

939: \item

940: $m^n(x,u)$ can be chosen independently of $u$, and 

941: \item

942: there is $b_0<\infty$ such that for all $x,n$ and $u$, $ m^n(x,u) \leq b_0$. 

943: \een

944: \end{lemma}

945: 

946: 

947: We introduce two parametric variations of the game defined in

948: Section \ref{sec:setting}. The first will be associated with

949: domain perturbation (parameterized by the symbol $a$), and the second

950: with a bound on the perturbed rates (parameterized by the symbol $b$).

951: 

952: For some fixed $a_0>0$, consider perturbations $G_a$,

953: $a\in(-a_0,a_0)$ of the domain $G$

954: defined as follows. If $G$ satisfies Condition~\ref{cond:G}.1,

955: then $G_a$ is defined as $G$, but with $z_i$ replaced by $z_i+a$,

956: $i=1,\ldots,J$.

957: If $G$ is as in Condition~\ref{cond:G}.2, then $G_a$ is defined

958: as $G$, but where $\phi_i$ is replaced by $\phi_i+a$, $i\in\calJ_+$.

959: 

960: For any $b\in(0,\infty)$, let $M^b=[0,b]^{2J}$. Analogously to

961: the definition (\ref{embar}) of $\bar M$, let

962: \begin{equation}

963: \label{embarb}

964:  \bar M^b =\{m=(\bar\la_1,\ldots,\bar\la_J,\bar\mu_1,\ldots,\bar\mu_J):

965: [0,\infty)\to M^b \ ;\ \mbox{$m$ is measurable}\}.

966: \end{equation}

967: Strategies and values for the game are then defined

968: analogously to the way strategies and values are defined for

969: the original game, using $\bar M^b$ in place of $\bar M$.

970: It will be convenient to set $M^\infty\doteq M$ and

971: $\bar M^\infty\doteq\bar M$, and to refer to the original game

972: of Section \ref{sec:setting} as the case $b=\infty$.

973: 

974: The cost, sets of strategies, lower and upper values of the

975: games resulting by the introduction of the parameters

976: $a$ and $b$ will be denoted as $C_a(x,u,m)$, $A^b$, $B^b$,

977: $V^{b,-}_a$ and $V^{b,+}_a$. When $a=0$ [resp., $b=\infty$],

978: the dependence on $a$ [$b$] will be eliminated from the notation,

979: as in $V^-_a$, $V^{b,-}$.

980: 

981: Let $b_0$ be as in Lemma \ref{lem:bound:m}.

982: Denote

983: \begin{equation}

984: \label{eq:lal}

985: b^* \doteq \max \{ b_0, \lambda_i , \mu_i , i = 1 , \ldots , J \} + 1.

986: \end{equation}

987: 

988: \begin{lemma}\label{lem:cont}

989: Assume Condition~\ref{cond:G}.

990: Then

991: \ben

992: \item

993: $\dist(\pl_{co} G_a,\pl_{co} G)\doteq 

994: \inf\{|x-y|:x \in \pl_{co} G_a, y \in \pl_{co} G\} >0\mbox{ if } 0<|a|<a_0$;

995: \item

996: the values

997: $V^{b,\pm}$ are bounded on $G$, uniformly for $b\in[b^*,\infty]$,

998: and there is a constant $c_0$ such that for any

999: $x\in G$, $|a|<\eps$ (where $\eps$ depends on $x$), and $b\in[b^*,\infty]$,

1000: one has

1001: $|V^{b,-}_a(x)-V^{b,-}(x)|\le c_0|a|$ and

1002: $|V^{b,+}_a(x)-V^{b,+}(x)|\le c_0|a|$.

1003: \een

1004: \end{lemma}

1005: 

1006: 

1007: The following lemma shows that any nearly optimal strategy for the minimizing player will satisfy a uniform upper bound on the integrated running cost.

1008: Moreover, there is a finite time $T_0$ such that 

1009: for each such minimizing strategy, any open loop control used by the maximizing player leads to  exit by $T_0$.

1010: Similarly,

1011: given any strategy for the maximizing player the minimizing player can restrict to open loop controls that force exit by $T_0$.

1012: \begin{lemma}\label{lem:apriori}

1013: Fix $b\in[b^*,\infty]$.

1014: Given $\beta \in B^b$,

1015: write

1016: $(\bar\la_i(\cdot ),\bar\mu_i(\cdot ))=\beta[u](\cdot)$.  For $z,T>0$ let

1017: $B^{z,T}$ denote the set of $\beta\in B^b$ which satisfy 

1018: $$

1019:   \int_0^T\sum_i[\la_il(\bar\la_i(t)/\la_i)+u_i(t)

1020:   \mu_il(\bar\mu_i(t)/\mu_i)]dt

1021:   \le z,

1022: $$

1023: for all $u\in\bar U$.

1024: For $\al\in A^b$, let $\bar M(\al,T)$ denote the set of $m\in\bar M$

1025: for which $\s(x,\al[m],m)\le T$.

1026: Then there are constants $z_0,T_0>0$ such that

1027: $$

1028: V^- (x)=\inf_{\beta\in B^{z_0,T_0}}\sup_{u\in\bar U}

1029: \int_0^{\sig\w T_0}[c+\rho(u(t),\beta[u](t))]dt,

1030: $$

1031: and

1032: $$

1033: V^+(x)=\sup_{\al\in A}\inf_{m\in\bar M(\al,T_0)}

1034: \int_0^{\sig\w T_0}[c+\rho(\al[m](t),m(t))]dt.

1035: $$

1036: \end{lemma}

1037: 

1038: In the rest of the section  the strategies $\beta$

1039: will be assumed (without loss)

1040: to be in $B^{z_0,T_0}$, where $z_0,T_0$ are as in

1041: Lemma \ref{lem:apriori}, and are fixed throughout. Also, $m\in\bar M$

1042: will be assumed to be in $\bar M(\al,T_0)$ whenever it is

1043: clear which $\al$ is considered.

1044: With an abuse of notation, we denote

1045: $B^{z_0,T_0}$ by $B$.

1046: 

1047: \begin{lemma}\label{lem:vallip} Under Condition \ref{cond:G},

1048: $V^{b,-}$ and $V^{b,+}$ are Lipschitz on $G$, uniformly for

1049: $ b\in[b^*,\infty]$.

1050: \end{lemma}

1051: 

1052: We are now ready to prove the following result.

1053: \begin{theorem}\label{th:limit}

1054: Let Condition \ref{cond:G} hold, and let $b\in[b^*,\infty)$.

1055: Then for any $x\in G$ and any $\{ x_n \} $ converging to $x$ (with

1056: $x_n\in G^n$),

1057: $$

1058:  V^{b,+}(x)\le\liminf_{n\to\infty}V^n(x_n)\le\limsup_{n\to\infty}

1059:  V^n(x_n)\le V^{b,-}(x).

1060: $$

1061: \end{theorem}

1062: 

1063: \proofOf{Theorem~\ref{th:limit}}

1064: The result is established by considering a sequence of stochastic

1065: processes, defined using the constrained ODEs, but for which

1066: the controls $u$ and $m$ are governed by, on one hand, a nearly optimal

1067: strategy for the game, and on the other hand, a nearly optimal

1068: control for the stochastic control problem.

1069: The technique uses standard martingale estimates, and is based

1070: on the construction (deferred to Section \ref{sec:lemmas})

1071: of an auxiliary controlled Markov process

1072: that is controlled by

1073: the selected strategy and stochastic control.

1074: 

1075: \noi{\em \uu{Upper bound}}

1076: 

1077: \noi

1078: Fix $b\in[b^*,\infty)$.

1079: The dependence on $b$ will be suppressed in the notation for $V^-$, $V^-_a$, etc.

1080: We first show that

1081: \be\label{eq:ub}

1082: \limsup_{n\to\infty} V^n(x_n) \leq V^-(x).

1083: \ee

1084: According to  Lemma~\ref{lem:cont}.2, it is enough to show that

1085: for all $a>0$

1086: \[

1087: \limsup_{n\to\infty} V^n(x_n) \leq V_a^-(x).

1088: \]

1089: Let $\beta\in B^b $ and $a>0$ be given,

1090: and set $C_a(x,\beta)=\sup_{u\in\bar U} C_a(x,u,\beta[u])$.  

1091: It is enough to show that

1092: \be\label{eq:ca}

1093: \limsup_{n\to\infty} V^n(x_n) \leq C_a(x,\beta), \quad a>0.

1094: \ee

1095: We therefore fix $\beta$ throughout, and turn to prove (\ref{eq:ca}).

1096: We can assume without loss that

1097: \begin{equation}\label{eq:31}

1098: C_a(x,\beta)<\infty.

1099: \end{equation}

1100: Note that in the DPE (\ref{eq:dpe}) the supremum is with respect to $u$ in

1101: a compact set $U$, and that the function being maximized is continuous in $u$ (for each $y$).

1102: Let $u^n(y)$ denote a point where it is achieved.

1103: Then for any $m\in M$ and $y\in G^n$,

1104: \be\label{ineq1}

1105: 0 \leq  \calL^{n,u^n(y),m}V^n(y) +c+\rho(u^n(y),m).

1106: \ee

1107: \begin{lemma}\label{lem:xbar}

1108: Let $n$ be fixed, and let $b\in[b^*,\infty)$,

1109: $\beta$ and $x_n$ be as above.

1110: Then there is a filtered probability space

1111: $(\bar\Om,\bar F,(\bar F_t),\bar P)$, and $\bar F_t$-adapted

1112: RCLL processes

1113: $\bar X^n$, $\bar Y^n$ and $m^n$ such that with $\bar P$-probability one

1114: $m^n(t)=\beta[\bar u^n](t)$ a.e.\ $t$, $\bar u^n(t)=u^n(\bar X^n(t))$,

1115: $\bar X^n=\Gamma(\bar Y^n)$,

1116: $\bar X^n(0)=\bar Y^n(0)=x_n$,

1117: and for any $f$

1118: \[

1119: f(\bar X^n(t)) - \int_0^t \calL^{n,\bar u^n, m^n(s)}f(\bar X^n(s))ds

1120: \]

1121: \[

1122: f(\bar Y^n(t)) - \int_0^t \calL_0^{n,\bar u^n, m^n(s)}f(\bar Y^n(s))ds

1123: \]

1124: are $(F_t)$-martingales.

1125: Moreover, with $T_0$ as in Lemma~\ref{lem:apriori} let $N_n $ denote

1126: the total number of jumps of $ \bar Y^n $ on $ [0,T_0]$. Then

1127: \be\label{ineq:Nn}

1128: EN_n\le 2JT_0bn.

1129: \ee

1130: \end{lemma}

1131: 

1132: \proof See Section \ref{sec:lemmas}.

1133: 

1134: Returning to the proof of~(\ref{eq:ca}), let $\bar u^n(t) \doteq u^n(\bar X^n(t))$

1135: and let $\bar\sigma^n$ be the first

1136: exit time of $\bar X^n$ from $G$.

1137: Combining

1138: (\ref{ineq1}) and Lemma \ref{lem:xbar}, for any bounded stopping

1139: time $S\le\bar\sig^n$,

1140: \be\label{ineq2}

1141: V^n(x_n) \leq \bar E_{x_n} \left[ V^n(\bar X^n(S))

1142: +\int_0^{S}[c+\rho(\bar u^n(s),\beta[\bar u^n](s))]ds \right].

1143: \ee

1144: Denoting $\beta[\bar u^n](t)= \{ (\bar\la_i^n(t),\bar\mu_i^n(t)) \} $,

1145: define $\phi^n$ as $\phi^n=\Gamma(\psi^n)$, where

1146: \[

1147:    \psi^n (t)= x+\int_0^t v(\bar u^n,\beta[\bar u^n])ds,

1148: \]

1149: and let

1150: \[

1151: \hat\sig^n_a \doteq \inf\{t:\phi^n (t) \not \in G_a\}.

1152: \]

1153: Then the definition of $C_a(x,\beta)$ implies

1154: \[

1155: \int_0^{\hat\sig^n_a}[c+\rho(\bar u^n(s),\beta[\bar u^n](s))]ds

1156:  \leq C_a(x,\beta).

1157: \]

1158: Apply (\ref{ineq2}) with $S=\hat\sig_a^n\w\bar\sig^n\w T$.

1159: If $T$ is sufficiently large, then (\ref{eq:31}) and the

1160: fact that $c>0$ imply $\hat\sig_a^n \leq T$. Thus, using

1161: $\bar E_{x_n}V^n(\bar X^n(\bar\sig^n)))=0$,

1162: \[

1163: V^n(x_n) \leq \bar E_{x_n} \left[ V^n(\bar X^n(\hat\sig^n_a))

1164: 1_{\{\hat\sig^n_a < \bar\sigma^n\}}

1165:  +\int_0^{\hat\sig^n_a}[c+\rho(\bar u^n(s),\beta[\bar u^n](s))]ds \right].

1166: \]

1167: Again using the uniform boundedness of $V^n(x)$ (Lemma \ref{lem:unif}), there is a constant $b_2 <\infty$

1168: such that for all $n$

1169: \be\label{ineq3}

1170: V^n(x_n) \leq b_2  \bar P_{x_n}(\hat\sig^n_a \le \bar\sigma^n)

1171:  + C_a(x,\beta).

1172: \ee

1173: In what follows, we show that $\bar P_{x_n}(\hat\sig^n_a \le \bar\sigma^n)$

1174: tends to zero. To this end,

1175: note that $\calL_0^{n,u,m}\,{\rm id}(y)=\sum_i\bar\la_iv_i+\sum_iu_i\bar\mu_i

1176: \tilde v_i$, where ${\rm id}$ is the identity map.

1177: Therefore, using again Lemma \ref{lem:xbar},

1178: \[

1179: \bar Y^n(t) -x_n = \int_0^t \lt[ \sum_{i=1}^J \bar\la_i^n(s)v_i

1180:  +\sum_{i=1}^J\bar u_i^n(s)\bar\mu_i^n(s)\tilde v_i\rt]

1181:  ds + \eta^n(t),

1182: \]

1183: where $\eta^n$ is a zero mean martingale.

1184: To prove that

1185: \be\label{conv:mgale}

1186:  \sup_{t\in[0,T]}|\eta^n(t)| \to 0 \quad \mbox{in distribution},

1187: \ee

1188: it is enough, by Doob's maximal inequality, to show that

1189: $$

1190: \bar E|\eta^n(T)|^2\to0.

1191: $$

1192: Let $[x](t)=\sum_{s\in[0,t]}|\Del x_s|^2$.

1193: By the Burkholder-Davies-Gundy inequality (see \cite{delmey}, VII.92),

1194:  $$

1195:  \bar E|\eta^n(T)|^2 \le c_1\bar E[\eta^n](T),

1196: $$

1197: where $c_1$ is a constant.

1198: Since each jump is bounded by $c_2n^{-1}$

1199: ($c_2$ a constant) and the total number of jumps

1200: $N_n(T)$ satisfies (\ref{ineq:Nn}),

1201: $$

1202: \bar E|\eta^n(T)|^2 \le c_3n^{-2}\bar EN_n(T) \le c_4n^{-1},

1203: $$

1204: which proves (\ref{conv:mgale}).

1205: This implies that $\sup_{[0,T]}|\bar Y^n(t)-\psi^n(t)|\to0$ in

1206: distribution, and therefore the continuity of $\Gamma$ implies $\sup_{[0,T]}|\bar X^n(t)-\phi^n(t)|\to0$

1207: in distribution.

1208: By Lemma \ref{lem:cont}.1,

1209: \beaa

1210: \bar P_{x_n} \left( \hat\sig^n_a \le \bar\sigma^n \right)

1211: &\leq &

1212: \bar P_{x_n}(\bar X^n(\hat\sig_a^n)\in G,

1213: \phi^n(\hat\sig_a^n)\in\pl_{co}G_a)\\

1214: &\le&

1215: \bar P_{x_n} \left (\sup_{t\in[0,T]}|\bar X^n(t)-\phi^n(t)|\ge b_1 \right ),

1216: \eeaa

1217: where $b_1>0$ depends only on $a$.

1218: Hence by (\ref{conv:mgale}),

1219: $\bar P_{x_n}[ \hat\sig^n_a \le \bar\sigma^n]\to0$

1220: as $n\to\infty$. Therefore (\ref{ineq3}) implies

1221: \[

1222: \limsup_{n \goto \infty} V^n(x_n) \leq C_a(x,\beta).

1223: \]

1224: This gives (\ref{eq:ca}) and completes the proof of (\ref{eq:ub}).

1225: 

1226: 

1227: \noi{\em \uu{Lower bound}}

1228: 

1229: \noi

1230: Next we prove

1231: \be\label{eq:lb}

1232: \liminf_{n\to\infty} V^n(x_n) \geq V^+(x).

1233: \ee

1234: By Lemma \ref{lem:cont}.2, it is enough to show that

1235: for all $a<0$

1236: \[

1237: \liminf_{n\to\infty} V^n(x_n) \geq V_a^+(x).

1238: \]

1239: Let $\al\in A$ be given,

1240: and set $C_a(x,\al)=\inf_{m\in\bar M^b} C_a(x,\al[m],m)$.  

1241: Then it suffices to show

1242: \be\label{eq:ca2}

1243: \liminf_{n\to\infty} V^n(x_n) \geq C_a(x,\al), \quad a<0.

1244: \ee

1245: Fixing $\al$, we now prove (\ref{eq:ca2}).

1246: 

1247: Interchanging the order of infimum and supremum in equation (\ref{eq:dpe})

1248: (see \cite{roc}, Corollary 37.3.2), and noting that

1249: the infimum over $m$ is of a continuous function with

1250: compact level sets, we denote by $m^n(y)$ a point where

1251: the infimum is achieved.

1252: By Lemma~\ref{lem:bound:m}, the components

1253: $ \bar\lambda_i^n (y) $ and $ \bar\mu_i^n (y) $ of $ m^n (y) $

1254: are all bounded by $b_0 $.

1255: For $u\in U$ and $y\in G^n$,

1256: \be\label{ineq12}

1257: 0 \geq \calL^{n,u,m^n(y)}V^n(y) +c+\rho(u,m^n(y)).

1258: \ee

1259: \begin{lemma}\label{lem:xbar2}

1260: Let $n$ be fixed, and let $\al$ and $x_n$ be as above.

1261: Then there is a filtered probability space

1262: $(\bar\Om,\bar F,(\bar F_t),\bar P)$, and $\bar F_t$-adapted

1263: RCLL processes

1264: $\bar X^n$, $\bar Y^n$ and $u^n$ such that with $\bar P$-probability one

1265: $u^n(t)=\al[\bar m^n](t)$ a.e.\ $t$, $\bar m^n(t)=m^n(\bar X^n(t))$,

1266: $\bar X^n=\Gamma(\bar Y^n)$,

1267: $\bar X^n(0)=\bar Y^n(0)=x_n$,

1268: and for any $f$,

1269: \[

1270: f(\bar X^n(t)) - \int_0^t \calL^{n, u^n(s),m^n}f(\bar X^n(s))ds

1271: \]

1272: \[

1273: f(\bar Y^n(t)) - \int_0^t \calL_0^{n, u^n(s), m^n}f(\bar Y^n(s))ds

1274: \]

1275: are $(\bar F_t)$-martingales.

1276: \end{lemma}

1277: 

1278: \proof See Section \ref{sec:lemmas}.

1279: 

1280: 

1281: Let $\bar m^n(t)= m^n(\bar X^n(t))$

1282: and let $\bar\sigma^n$ be the first

1283: exit time of $\bar X^n$ from $G$.

1284: By (\ref{ineq12}) and Lemma \ref{lem:xbar2}, for any bounded stopping

1285: time $S\le\bar\sig^n$,

1286: \be\label{ineq22}

1287: V^n(x_n) \geq \bar E_{x_n} \left[ V^n(\bar X^n(S))

1288: +\int_0^{S}[c+\rho(\al[\bar m^n](s),\bar m^n(s))]ds \right].

1289: \ee

1290: Denoting $\bar m^n(t)=((\bar\la_i^n(t),\bar\mu_i^n(t))$,

1291: define $\phi^n$ as $\phi^n=\Gamma(\psi^n)$, where

1292: \[

1293:    \psi^n = x+\int_0^\cdot v(\bar u^n,\bar m^n)ds,

1294: \]

1295: and let

1296: \[

1297: \hat\sig^n_a \doteq \inf\{t:\phi^n (t)\not \in G_a\}.

1298: \]

1299: Then the definition of $C_a(x,\alpha)$ implies 

1300: \[

1301: \int_0^{\hat\sig^n_a}[c+\rho(\al[\bar m^n](s),\bar m^n(s))]ds

1302:  \geq C_a(x,\al).

1303: \]

1304: Apply (\ref{ineq22}) with $S=\hat\sig_a^n\w\bar\sig^n\w T$,

1305: with large enough $T$,

1306: using the fact that $V^n\ge0$ to get

1307: \beaa

1308: V^n(x_n) &\geq& \bar E_{x_n} \left[

1309:  \int_0^{\hat\sig^n_a\w\bar\s^n\w T}

1310:  [c+\rho(\al[\bar m^n](s), \bar m^n(s))]ds \right]\\

1311:  &\ge&

1312: \bar E_{x_n}\left[1_{\hat\sig^n_a\le\bar\sig^n}

1313:  \int_0^{\hat\sig^n_a\w T}

1314:  [c+\rho(\al[\bar m^n](s), \bar m^n(s))]ds \right]\\

1315:  &\ge&

1316: \bar P_{x_n}(\hat\s^n_a\le\bar\s^n)C_a(x,\al).

1317: \eeaa

1318: The proof that

1319: $\bar P_{x_n}(\hat\sig^n_a \le \bar\sigma^n)$ tends to one

1320: is analogous to the proof of the that

1321: $\bar P_{x_n}(\hat\sig^n_a \le \bar\sigma^n)\to0$ in the

1322: upper bound. It is therefore omitted. Hence

1323: \[

1324: \liminf_{n \goto \infty} V^n(x_n) \geq C_a(x,\al).

1325: \]

1326: This gives (\ref{eq:ca2}),

1327: and the proof of (\ref{eq:lb}) is established.

1328: \qed

1329: 

1330: In fact, the value of the game is independent of $b$ for large $b$,

1331: so that the game has a value with the unbounded action space $M$.

1332: As the result depends on Theorems \ref{th:limit}, \ref{th:unique},

1333: \ref{th:solve} we postpone the proof to Section~\ref{sec:lemmas}.

1334: \begin{theorem}\label{th:b}

1335: For all $b\in[b^*,\infty]$, $V^{b,+}=V^+=V^{b,-}=V^-$.

1336: \end{theorem}

1337: \proof See Section \ref{sec:lemmas}.

1338: 

1339: 

1340: \section{The PDE}\label{sec:pde}

1341: 

1342: In this section we show that the upper and lower values of the game

1343: are the unique Lipschitz viscosity solutions of the PDE (\ref{eq:pde}).

1344: Throughout, the parameter $b\in[b^*,\infty)$ is fixed.

1345: Let

1346: \begin{equation}\label{def:H}

1347:  H(q)=\inf_m\sup_u[\lan q,v(u,m)\ran+\rho(u,m)+c].

1348: \end{equation}

1349: It will be useful to

1350: note that the infimum is over the compact set $M^b$,

1351: and the map $(q,u,m)\mapsto [\lan q,v(u,m)\ran+\rho(u,m)+c]$

1352: is continuous.

1353: The PDE of interest is

1354: \be\label{eq:pde}

1355: \lt\{

1356: \begin{array}{ll}

1357:  H(DV(x))=0, & x\in G^o,\\ 

1358:  \lan DV(x),\gamma_i\ran=0, & i\in I(x),\ x\in\pl_+G ,\\ 

1359:  V(x)=0, & x\in\OBdry G.

1360: \end{array}\rt.

1361: \ee

1362: Here, $\gamma_i$ are the directions of constraint that were introduced in Section 2.

1363: \begin{definition}

1364: Let a Lipschitz continuous function $u:X \to\R$ be given

1365: (where $X\subset G$).

1366: We say that $u$ is a subsolution [respectively, supersolution]

1367: to (\ref{eq:pde}) on $X$ if the following conditions hold. Let

1368: $\theta: X \to\R$ be continuously differentiable on $\bar X$.

1369: Let $y\in X$ be a local maximum [minimum] of the map

1370: $x\mapsto u(x)-\theta(x)$. Then

1371: \be\label{eq:subs}

1372:  H(D\theta(y))\vee\max_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\ge0,

1373: \ee

1374: \be\label{eq:supers}

1375:  [\quad

1376:  H(D\theta(y))\w\min_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\le0,

1377:  \quad]

1378: \ee

1379: and

1380: \be\label{eq:subsb}

1381:  V(x)\le0,\quad x\in\bar X\cap\OBdry G,

1382: \ee

1383: \be\label{eq:supersb}

1384:  [\quad

1385:  V(x)\ge0,\quad x\in\bar X\cap\OBdry G.

1386:  \quad]

1387: \ee

1388: We say that $V$ is a viscosity solution to (\ref{eq:pde}),

1389: if it is both a subsolution on $G$ and a supersolution

1390: on $G\setminus\pl_cG$.

1391: \end{definition}

1392: 

1393: \remark

1394: In case that $\pl_cG\ne\emptyset$, a viscosity solution is often

1395: called a constrained viscosity solution (cf.\ Soner \cite{son},

1396: Capuzzo-Dolcetta and Lions~\cite{capLion}).

1397: The requirement that $V$ is a subsolution up to the boundary

1398: $\pl_cG$---the part of the boundary where exit can be unilaterally

1399: blocked---serves as a boundary condition on this

1400: part of the boundary. Note that in the current paper,

1401: the term `constrained' refers to the part $\pl_+G$ of the

1402: boundary, where it is the mechanism associated with the Skorokhod

1403: Problem that constrains the dynamics to $G$.

1404: \qed

1405: 

1406: First, we address uniqueness of solutions to (\ref{eq:pde}).

1407: \begin{theorem}\label{th:unique}

1408: Let $u$ be a subsolution and $v$ a supersolution to (\ref{eq:pde}).

1409: Then $u\le v$ on $G$.

1410: \end{theorem}

1411: 

1412: The proof combines ideas from two sources, namely \cite{atadup2} (which is based on \cite{dupish2}, and discusses how to deal  with the constrained

1413: dynamics on $\pl_+G$), and \cite{son} (to accommodate the fact that under Condition \ref{cond:G}.1

1414: part of the boundary ($\pl_cG$) can be thought

1415: of as imposing a state-space constraint on the maximizing player).

1416: 

1417: The following lemma will be used in proving Theorem \ref{th:unique}.

1418: In the interest of consistency with previous

1419: publications, we use $B$ in Lemma~\ref{lem:mu} below to denote a certain

1420: subset of $\R^J$ (although everywhere except in this section, $B$ denotes

1421: a set of strategies).

1422: Part 1 states that the ``Set B'' Condition holds, namely a condition

1423: under which it was proved in \cite{dupish1} that the SM enjoys

1424: the regularity property (\ref{sp:lip}).

1425: The proof that this condition holds in

1426: the current setting can be found in \cite{dupram23}.

1427: The existence of a smooth version of the set $B$ is proved in

1428: \cite{atadup2}

1429: (before Lemma 2.1). For Parts 2 and 3, see Lemmas 2.1 and 2.2 of

1430: \cite{atadup2} (note that the condition that $\gamma_i$ are independent

1431: holds).

1432: \begin{lemma}\label{lem:mu}

1433: \ben

1434: \item

1435: There exists a compact, convex, and symmetric set $B\subset \RJ $ with

1436: $0\in B^o$, such that if $z\in\pl B$ and if $n$ is an outward

1437: normal to $B$ at $z$, then for all $i\in\{1,\ldots,J\}$

1438: $$

1439:  \lan z,e_i\ran\ge-1 {\rm\ implies\ } \lan\gamma_i,n\ran\ge0

1440:  {\rm\ and\ } \lan z,e_i\ran\le1 {\rm\ implies\ } \lan\gamma_i,n\ran\le0.

1441: $$

1442: In addition, the unit outward normal $n(x)$ to $B$ at $x\in\pl B$ is

1443: unique and continuous (as a function on $\pl B$).

1444: \item

1445: Let $\bar n$ be the extension of $n$ to $ \RJ $

1446: satisfying $\bar n(x)=n(y)$ whenever $ax=y\in\pl B$, some $a\in(0,\infty)$

1447: (and define $\bar n(0)$ arbitrarily).

1448: Let $\Xi: \RJ \to\R_+$ be defined via

1449: $$

1450:  \Xi(x)=a\ \Leftrightarrow\ x\in\pl(aB)

1451: $$

1452: for all $a\in[0,\infty)$, and let $\xi(x)=(\Xi(x))^2$.

1453: Then there exist constants $m,M\in(0,\infty)$ and a function

1454: $\vr: \RJ \to[m,M]$ such that the $C^1( \RJ )$

1455: function $\xi$ satisfies $m\|x\|^2\le\xi(x)\le M\|x\|^2$, and

1456: $D\xi(x)=\vr(x)\Xi(x)\bar n(x)$.

1457: \item

1458: There exists a constant $m_1\in (0,\infty)$ and a continuously differentiable

1459: function $\mu: \RJP \to[0,m_1]$ such that $\|D\mu\|\le m_1$ on

1460: $ \RJP $, and

1461: $$

1462:  \lan D\mu(x),\gamma_i\ran<0,\quad x\in \RJP ,\quad i\in I(x).

1463: $$

1464: \een

1465: \end{lemma}

1466: 

1467: 

1468: In what follows, we keep the notation of Lemma \ref{lem:mu}

1469: for $B, \bar n, \Xi, \xi, \vr$ and $\mu$.

1470: 

1471: 

1472: \noindent{\bf Proof of Theorem \ref{th:unique}:}

1473: For $a>0$, let

1474: $$

1475:  U(x)=u(x)-a\mu(x),

1476: $$

1477: $$

1478:  V(x)=v(x)+a\mu(x).

1479: $$

1480: Let $\del>0$. Then it suffices to show that for all small $a>0$,

1481: $\del>0$, one has $U\le (1+\del)V$ on $G$. Arguing by contradiction,

1482: we assume that this is not true. Then there are $a$ and $\del$

1483: arbitrarily small such that 

1484: $$

1485:  \rho=\sup_{x\in G}[U(x)-(1+\del)V(x)]>0.

1486: $$

1487: Below we let $c_i, i=1,2,...$ denote positive constants.

1488: Consider  Condition \ref{cond:G}.1 first.

1489: Let

1490: \begin{equation}\label{eq:phixy}

1491:  \Phi(x,y)= U(x)-(1+\del)V(y)-\frac 1 \eps\xi(x -y-\eps^{1/2}y).

1492: \end{equation}

1493: Let $(\bar x,\bar y)\in \bar G^2$

1494: achieve the maximum of $\Phi$ in $ \bar G \times

1495: \bar G $.

1496: By continuity of $U$ and $V$, there exists $ \bar z\in\bar G $ so that 

1497: $ \rho = U( \bar z)-(1+\del)V(\bar z) $. Note that

1498: \begin{equation}\label{eq:10}

1499: (1+\eps^{1/2})^{-1}\bar z\in\bar G.

1500: \end{equation}

1501: Hence by the Lipschitz continuity

1502: of $V$,

1503: \bea\label{e:phi1}\nonumber

1504: \Phi ( \bar x , \bar y )

1505: & \geq & \Phi 

1506:          \left ( \bar z, \frac {\bar z} {1 + \eps^{\Sfrac 1 2 }} \right )

1507: \\

1508: \nonumber

1509: & =  & U(\bar z)-(1+\del)

1510: V \left (\frac {\bar z} {1 + \eps^{\Sfrac 1 2 }} \right )

1511: \\

1512: & \geq & \rho - c_1 \eps^{\Sfrac 1 2 } .

1513: \eea

1514: By Lipschitz continuity of $U$ and the lower bound on $\xi$ given in Lemma \ref{lem:mu}, 

1515: \bea\label{e:phi2}\nonumber

1516: \Phi ( \bar x , \bar y ) 

1517: & =    & U(\bar x)-(1+\del)V(\bar y)

1518:          -\frac 1 \eps \xi (\bar x  - \bar y-\eps^{1/2}\bar y)   \\

1519: \nonumber

1520: & \leq & U (\bar y ) + c_2 | \bar x - \bar y | - (1+\del)V(\bar y)

1521:          - \frac m \eps | \bar x - \bar y-\eps^{1/2}\bar y |^2\\

1522: &\le&

1523: \rho + c_2 | \bar x - \bar y | 

1524:      - \frac{c_3}{\eps} | \bar x- \bar y-\eps^{1/2}\bar y |^2 .

1525: \eea

1526: By (\ref{e:phi1}) and (\ref{e:phi2}),

1527: \bea\label{e:phi3}

1528: c_2 | \bar x - \bar y | + c_1 \eps^{\Sfrac 1 2}

1529: &\geq&

1530: \frac{c_3}{\eps} | \bar x- \bar y-\eps^{1/2}\bar y |^2

1531: \\ \nonumber

1532: &\geq&

1533: \frac{c_4}{\eps} | \bar x - \bar y |^2 - c_4 | \bar y |^2 .

1534: \eea

1535: Since $ \bar x $ and $ \bar y $ are bounded,

1536: (\ref{e:phi3}) implies

1537: $| \bar x - \bar y |^2 \leq c_5 \eps$

1538: and so

1539: \begin{equation}\label{e:phi5}

1540: | \bar x - \bar y | \leq c_6 \eps^{\Sfrac 1 2 }.

1541: \end{equation}

1542: Using this in~(\ref{e:phi3}) we have

1543: \begin{equation}\label{e:phi4}

1544: | \bar x- \bar y-\eps^{1/2}\bar y | \leq c_7 \eps^{\Sfrac 3 4} .

1545: \end{equation}

1546: By (\ref{e:phi5}),

1547: $\bar x-\bar y\to0$ as $\eps\downarrow0$. Also,

1548: we claim that

1549: for all $\eps>0$ small, $\bar x$ and $\bar y$ are 

1550: bounded away from $\pl_o G$.

1551: To see this, assume the contrary. Then along a subsequence, both

1552: $\bar x$ and $\bar y$ must converge to the same point on $\pl_o G$.

1553: Using the continuity of $u$ and $v$,

1554: (\ref{eq:subsb})--(\ref{eq:supersb}),

1555: and the non-negativity of $\xi$,

1556: $\limsup\Phi(\bar x,\bar y)

1557: \le \limsup \left [ u(\bar x)-(1+\del)v(\bar y) \right ] \le0$,

1558: where the limit superior is taken along this subsequence.

1559: However, by (\ref{e:phi1}),

1560: for all small $\eps$, $\Phi(\bar x,\bar y)\ge\rho/2>0$, which

1561: gives a contradiction.

1562: 

1563: Let

1564: $$

1565:  \theta(x)=\frac1\eps\xi(x-\bar y-\eps^{1/2}\bar y)+a\mu(x),

1566: $$

1567: and note that the map $x\mapsto u(x)-\theta(x)$ has a maximum

1568: at $\bar x\in G$.

1569: Since $u$ is a subsolution, (\ref{eq:subs}) must

1570: be satisfied at $\bar x$. Denoting

1571: \begin{equation}\label{eq:que}

1572: q^\eps=

1573: \vr(\bar x-\bar y-\eps^{1/2}\bar y)\Xi(\bar x-\bar y-\eps^{1/2}\bar y)

1574: \bar n(\bar x-\bar y-\eps^{1/2}\bar y),

1575: \end{equation}

1576: we have from Lemma \ref{lem:mu}.2 that

1577: $$

1578:  D\theta(\bar x)=\frac1\eps q^\eps+aD\mu(\bar x).

1579: $$

1580: Suppose $i$ is such that $\bar x_i=0$. Then 

1581: $ \lan\bar x-\bar y-\eps^{1/2}\bar y,e_i\ran\le0 $

1582: and so by Lemma \ref{lem:mu}.1,

1583: $$

1584: \lan\gamma_i, \bar n(\bar x-\bar y-\eps^{1/2}\bar y)\ran\le0.

1585: $$

1586: Since by Lemma \ref{lem:mu}.3, $\lan\gamma_i, D\mu(\bar x)\ran<0$,

1587: it follows that $\lan\gamma_i,D\theta(\bar x)\ran<0$. It follows from

1588: (\ref{eq:subs}) that

1589: $$

1590:  H(D\theta(\bar x))\ge0,

1591: $$

1592: namely,

1593: \be\label{eq:H1}

1594:  H\lt(\frac1\eps q^\eps+aD\mu(\bar x)\rt)\ge0.

1595: \ee

1596: On the other hand, let

1597: $$

1598:  \al(y)=-a\mu(y)-\frac{1}{\eps(1+\del)}\xi(\bar x-y-\eps^{1/2}y).

1599: $$

1600: Note that

1601: $$

1602:  D\al(\bar y)=-aD\mu(\bar y)+\frac{1+\eps^{1/2}}{\eps(1+\del)} q^\eps

1603: $$

1604: and that

1605: the map $y\mapsto v(y)-\al(y)$ has a minimum at $\bar y$.

1606: Since $\bar x\in\bar G$, (\ref{e:phi4}) implies that

1607: $\bar y\in G\setminus\pl_cG$ for all small $\eps$.

1608: Since $v$ is a supersolution,

1609: (\ref{eq:supers}) is satisfied at $y$. An argument as above shows

1610: that

1611: $$

1612:  H(D\al(\bar y))\le0,

1613: $$

1614: and therefore

1615: $$

1616:  H\lt(\frac{1}{1+\del}\lt(\frac{1+\eps^{1/2}}\eps q^\eps

1617:  -a(1+\del)D\mu(\bar y)\rt)\rt)\le0.

1618: $$

1619: It follows from the definition of $H$, using $\rho(u,m)\ge0$, that

1620: $$

1621:  H\lt(\frac 1{1+\del}\, p\rt)\ge\frac 1{1+\del}H(p)+\frac\del{1+\del}c,

1622: $$

1623: and therefore

1624: \be\label{eq:H2}

1625:  H\lt(\frac1\eps q^\eps+\frac1{\eps^{1/2}}q^\eps

1626: -a(1+\del)D\mu(\bar y)\rt)+\del c\le0.

1627: \ee

1628: Now $D\mu$ is bounded, and by (\ref{e:phi4}), boundedness of

1629: $n$ and $\vr$, and

1630: the Lipschitz continuity of $\Xi$, it follows that

1631: $\eps^{-1/2}q^\eps$ converges to zero as $\eps\to0$.

1632: Note that by (\ref{def:H}) and the following comment,

1633: $H$ is uniformly continuous on $\R^J$.

1634: Therefore,

1635: (\ref{eq:H1}) and (\ref{eq:H2}) give a contradiction when $a>0$

1636: and $\eps>0$ are small and $\del>0$ fixed.

1637: 

1638: Under Condition \ref{cond:G}.2 the above argument

1639: is not valid, since (\ref{eq:10}) may not hold.

1640: However,

1641: in this case the minimizing player can force exit from any point on $\partial G\backslash \partial_+ G$,

1642: and the additional complications due to the ``state-space constraint'' used under part 1 are no longer needed.

1643: In other words,

1644: instead of (\ref{eq:phixy})  we can consider 

1645: $$

1646: \Phi(x,y)=U(x)-(1+\del)V(y)-\frac1\eps\xi(x-y),

1647: $$

1648: and a review of the above proof shows that (\ref{eq:H1})

1649: and (\ref{eq:H2}) still hold if the expression

1650: $\eps^{1/2}$ is replaced by zero everywhere in (\ref{eq:que}) and

1651: (\ref{eq:H2}). A contradiction is then obtained analogously.

1652: \qed

1653: 

1654: We next consider the upper and lower values of the game,

1655: and remind the reader that in this section the rates $m$ are assumed bounded.

1656: \begin{theorem}\label{th:solve}

1657: $V^-$ and $V^+$ are solutions to (\ref{eq:pde}).

1658: \end{theorem}

1659: Recall that from Lemma \ref{lem:vallip}, $V^\pm$ are Lipschitz.

1660: 

1661: 

1662: 

1663: \noindent

1664: {\bf Proof of Theorem \ref{th:solve}:}

1665: We use the specific form of $v(u,m)$ and $\rho(u,m)$. These can be

1666: written as

1667: $$

1668:  v(u,m)=b_0(m)+\sum_{i=1}^Ju_ib_i(m),

1669: $$

1670: $$

1671:  \rho(u,m)=c_0(m)+\sum_{i=1}^Ju_ic_i(m).

1672: $$

1673: We have $\sum_{i\in C(k)}u_i\le1$.

1674: The $b_i$ are linear, and the $c_i$ are convex.

1675: Hence, as a direct consequence of \cite[Corollary 37.3.2]{roc}, 

1676: the {\em Isaacs condition} holds, namely

1677: \be\label{eq:isaacs}

1678:  H(q)=\inf_m\sup_u[\lan q,v(u,m)\ran+c+\rho(u,m)]

1679:  =\sup_u\inf_m[\lan q,v(u,m)\ran+c+\rho(u,m)].

1680: \ee

1681: Another fact that we will use is that for any

1682: $y\in G\setminus\pl_cG$ there

1683: is $\del_0=\del_0(y)>0$ which serves as a lower bound on the exit time.

1684: Namely, if $\phi$ solves $\dot\phi=\pi(\phi,v(u,m))$, $\phi(0)=y$,

1685: then

1686: \be\label{eq:boundsig}

1687:  \sig \doteq \inf\{ t\geq 0: \phi(t) \not \in G\} \ge\del_0,\quad u\in \bar U, m\in \bar M.

1688: \ee

1689: The bound is an immediate consequence of the $u$ and $m$ being

1690: uniformly bounded.

1691: 

1692: By definition, $ V ^\pm (x) = 0 $ for $ x \in \pl_o G $.

1693: Thus we only need to

1694: establish (\ref{eq:subs})--(\ref{eq:supers}).

1695: The proof consists of four parts.

1696: 

1697: \noindent{\em \underline{Proof that $V^-$ is a supersolution on $G \backslash \partial_cG$.}}

1698: 

1699: Standard dynamic programming arguments show that for $\del>0$,

1700: \be\label{eq:dp1}

1701:  V^-(x)=\inf_\beta\sup_u\lt[\int_0^{\s\w\del}(c+\rho(u,\beta[u]))dt

1702:  +V^-(\phi(\s\w\del))\rt],

1703: \ee

1704: where $\phi$ is the solution to $\dot\phi=\pi(\phi,v(u,\beta[u]))$,

1705: with $\phi(0)=x$.

1706: Let $\theta$ be smooth, and let $y\in G\setminus\pl_cG$

1707: be a local minimum of

1708: $V^--\theta$. We can assume without loss that $V^-(y)=\theta(y)$.

1709: We need to show

1710: \be\label{alpha}

1711:  H(D\theta(y))\w\min_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\le0.

1712: \ee

1713: We shall assume the contrary and reach a contradiction.

1714: Thus, there exists $a>0$ such that

1715: $H(D\theta(y))\ge a$, and

1716: \be\label{eq:D1}

1717:  \lan D\theta(y),\gamma_i\ran\ge a,\quad i\in I(y).

1718: \ee

1719:  From the definition of $H$ and (\ref{eq:isaacs}),

1720: $$

1721:  \sup_u\inf_m[\lan D\theta(y),v(u,m)\ran+c+\rho(u,m)]\ge a,

1722: $$

1723: and therefore there exists a $u_0$ such that for all $m$,

1724: $$

1725:  \lan D\theta(y),v(u_0,m)\ran+c+\rho(u_0,m)\ge a/2.

1726: $$

1727: For any strategy $\beta$, if $\bar u(t)\equiv u_0$,

1728: \be\label{eq:D2}

1729:  \lan D\theta(y),v(\bar u(t),\beta[\bar u](t))\ran+c

1730:  +\rho(\bar u(t),\beta[\bar u](t))\ge a/2

1731: \ee

1732: for all $t$. Let $\phi$ denote the dynamics corresponding to $\bar u$

1733: and a generic $\beta$, starting from $y$.

1734: Note that the mapping $z\mapsto I(z)$ is upper semi-continuous, in

1735: the sense that for any $z$ there is a neighborhood of $z$ on which

1736: $I(\cdot)\subset I(z)$. Using the boundedness of $m$

1737: this implies that for any $\beta\in B$, one has

1738: $I(\phi(r))\subset I(y)$ for $r\in[0,\del]$, if $\del>0$ is

1739: chosen small enough.

1740: We now use that $\phi$ is a solution to the SP.

1741: Choosing such a $\del >0$, for any $r\in[0,\del]$ there

1742: exist $a_i\ge0$ (that may depend on $r$) such that

1743: $$

1744:  \dot\phi(r)=v(\bar u(r),\beta[\bar u](r))+\sum_{i\in I(y)}a_i\gamma_i.

1745: $$

1746: Using the continuity of $D\theta$ and taking $\del>0$ smaller

1747: if necessary, (\ref{eq:D1}) and (\ref{eq:D2}) imply,

1748: for $t\in[0,\del]$,

1749: \beaa

1750:  \frac{d}{dt}\theta(\phi(t)) &=&

1751: \lan D\theta(\phi(t)),\dot\phi(t)\ran\\

1752:  &\ge&

1753: -c-\rho(\bar u(t),\beta[\bar u](t))+a/4.

1754: \eeaa

1755: Taking $\del$ even smaller if necessary (so that it is at most

1756: $\del_0$), we have from (\ref{eq:boundsig}) that

1757: $$

1758:  \theta(\phi(\del))-\theta(y)\ge

1759:  -\int_0^{\del}(c+\rho(\bar u(t),\beta[\bar u](t)))dt

1760:  +a\del/4.

1761: $$

1762:  From (\ref{eq:dp1}), one can find a $\beta$ such that

1763: $$

1764:  V^-(y)\ge\sup_u\lt[\int_0^{\del}(c+\rho(u,\beta[u]))dt

1765:  +V^-(\phi(\del))-a\del/8\rt].

1766: $$

1767: Letting $u=\bar u$, the last two displays give (using $\theta(y)

1768: =V^-(y)$)

1769: $$

1770:  \theta(\phi(\del))\ge V^-(\phi(\del))+a\del/8,

1771: $$

1772: so that $V^-(\phi(\del))-\theta(\phi(\del))<0$ for all

1773: $\del>0$ small, contradicting the assumption that $y$ is

1774: a local minimum of $V^--\theta$. This proves that $V^-$ is a

1775: supersolution on $G \backslash \partial_cG$.

1776: 

1777: \noindent

1778: {\em \underline{Proof that $V^-$ is a subsolution on $G$.}}

1779: 

1780: Let $\theta$ be smooth and $y\in G$ a local maximum of

1781: $V^--\theta$.

1782: In case that $y\in\pl_cG$, let $\bar U_{y,\beta,\del}$

1783: be the set

1784: of controls $u\in\bar U$ for which the trajectory $\phi$ determined

1785: by $u$ and $\beta[u]$ and starting from $y$ does not exit $G$

1786: on $[0,\del]$. Given $y\in\pl_cG$, it is clear that

1787: $\bar U_{y,\beta,\del}$ is not empty for all $\del$

1788: small and all $\beta$, by considering the control $u=0$.

1789: Moreover, for all $\del$ small enough,

1790: (\ref{eq:dp1}) is valid where the supremum

1791: extends only over $u\in\bar U_{y,\beta,\del}$. Indeed, given

1792: $u\not\in\bar U_{y,\beta,\del}$, consider

1793: $u'$ that agrees with $u$ on $[0,\sig]$ and $u'=0$ on

1794: $(\sig,\del]$. Then the expression in brackets in (\ref{eq:dp1})

1795: is identical under $u$ and under $u'$, but

1796: $u'\in\bar U_{y,\beta,\del}$.

1797: 

1798: Assume without loss that $V^-(y)=\theta(y)$.

1799: We would like to show that

1800: \be\label{gamma}

1801:  H(D\theta(y))\vee\max_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\ge0.

1802: \ee

1803: Assuming the contrary, there exists $a>0$ such that

1804: $H(D\theta(y))\le-a$, and

1805: \be\label{eq:D3}

1806:  \lan D\theta(y),\gamma_i\ran\le-a,\quad i\in I(y).

1807: \ee

1808: Using the definition of $H$ and (\ref{eq:isaacs}), for all $u$ there exists $m_u$ such that

1809: \be\label{eq:D4}

1810: \lan D\theta(y),v(u,m_u)\ran

1811:  +c+\rho(u,m_u)\le-a/2.

1812: \ee

1813: Note that it is possible to choose $m_u$ so that it depends continuously

1814: on $u$.

1815: Define $\bar\beta$ as $\bar\beta[u](t)=m_{u(t)}$ for all $t$. Since

1816: $\bar\beta[u]$ is measurable if $u$ is, $\bar\beta$ maps $\bar U$

1817: into $\bar M$.

1818: Let $\phi$ be the trajectory corresponding to $\bar\beta$ and a generic

1819: $u\in \bar U$, (or a generic $u\in\bar U_{y,\beta,\del}$ if

1820: $y\in\pl_cG$) starting from $y$.

1821: Arguing as before by upper semi-continuity

1822: of $I(\cdot)$, if $\del$ is small enough, then

1823: $$

1824:  \dot\phi(r)=v(u(r),\bar\beta[u](r))+\sum_{i\in I(y)}a_i\gamma_i,

1825:  \quad r\in[0,\del],

1826: $$

1827: where $a_i\ge0$ may depend on $r$.

1828: By possibly taking $\del$ smaller, and smaller

1829: than $\del_0$, we have, using the continuity of $D\theta$ and

1830: (\ref{eq:D3}), (\ref{eq:D4}) that

1831: \beaa

1832:  \frac{d}{dt}\theta(\phi(t)) &=&

1833: \lan D\theta(\phi(t)),\dot\phi(t)\ran\\

1834:  &\le&

1835: -c-\rho(u(t),\bar\beta[u](t))-a/4,

1836: \eeaa

1837: and

1838: $$

1839:  \theta(\phi(\del))-\theta(y)\le-\int_0^\del(c+

1840:  \rho(u(t),\bar\beta[u](t)))dt-a\del/4.

1841: $$

1842: Now, (\ref{eq:dp1}) implies that for any $\beta$ there

1843: is $u$ such that

1844: $$

1845:  V^-(y)\le\int_0^\del(c+\rho(u,\beta[u]))dt+V^-(\phi(\del))+a\del/8.

1846: $$

1847: Specializing to $\beta=\bar\beta$, the last two displays show

1848: that $V^-(\phi(\del))-\theta(\phi(\del))>0$ for all $\del>0$

1849: small. This contradicts the assumption that $y$ is a local

1850: maximum of $V^--\theta$, and as a result, $V^-$ is a subsolution.

1851: 

1852: \noindent

1853: {\em \underline{Proof that $V^+$ is a supersolution on $G \backslash \partial_cG$.}}

1854: 

1855: The proof is analogous to the proof that $V^-$ is a subsolution.

1856: Most details are therefore skipped.

1857: The dynamic programming principle states that

1858: for $\del>0$,

1859: \be\label{eq:dp2}

1860:  V^+(x)=\sup_\al\inf_m\lt[\int_0^{\s\w\del}(c+\rho(\al[m],m))dt

1861:  +V^+(\phi(\s\w\del))\rt],

1862: \ee

1863: where $\phi$ is the dynamics corresponding to $\al$ and $m$,

1864: starting from $x$. Taking a smooth $\theta$, and leting

1865: $y\in G\setminus\pl_cG$

1866: be a local minimum of $V^+-\theta$, showing

1867: $$

1868:  H(D\theta(y))\w\min_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\le0

1869: $$

1870: can be obtained by an argument analogous to that used to prove (\ref{gamma}),

1871: using (\ref{eq:dp2}) in place of (\ref{eq:dp1}).

1872: 

1873: \noindent

1874: {\em \underline{Proof that $V^+$ is a subsolution on $G$.}}

1875: 

1876: We need to show that

1877: \be\label{beta}

1878:  H(D\theta(y))\vee\max_{i\in I(y)}\lan D\theta(y),\gamma_i\ran\ge0,

1879: \ee

1880: where $\theta$ is smooth, and $y\in G$ is a local maximum of

1881: $V^+-\theta$.

1882: In the special case where $y\in\pl_cG$, we can assume

1883: without loss that the supremum in (\ref{eq:dp2}) extends

1884: only over $\al\in A_{y,\del}$, the set of strategies

1885: under which, for any $m\in\bar M^b$, the dynamics associated

1886: with $\al$ and $m$, and starting from $y$,

1887: does not leave $G$ before $\del$.

1888: The proof of (\ref{beta})

1889: is analogous to the proof of (\ref{alpha}), and is skipped.

1890: 

1891: 

1892: This completes the proof that $V^-$ and $V^+$ are solutions

1893: to (\ref{eq:pde}).

1894: \qed

1895: 

1896: 

1897: 

1898: \section{A competing queues example}\label{sec:example}

1899: 

1900: Consider a queueing network with only one server, providing

1901: service to $J$ classes. Each customer requires service once.

1902: In this example all arrival rates are positive:

1903: $\la_i>0$ for all $i$, hence $\calJ_+=\{1,\ldots,J\}$.

1904: This network, ``the $k$ competing queues,'' has been studied extensively,

1905: in discrete and continuous

1906: time (see~\cite{BDM,W} and references therein). When the criterion 

1907: (to be minimized) is

1908: either the average cost or the discounted cost, and the one-step cost

1909: is a positive linear combination

1910: $\sum_i c_ix_i$ of the queue sizes $x_i$, the optimal policy is

1911: the $ \mu $-$c$ rule, which is a priority discipline,

1912: giving absolute priority to the non-empty queue for which

1913: $\mu_ic_i$ is maximal.

1914: Under the cost studied here, the optimal policy is quite different.

1915: 

1916: \begin{proposition}

1917: Consider the case where $G$ is a hyper-rectangle, given

1918: as $G=\{x:0\le x<z_i\}$, where $z_i>0$ are constants.

1919: Assume that $\la_i>0$ for all $i=1,\ldots,J$.

1920: If $c$ is large enough,

1921: then the viscosity solution to the PDE (\ref{eq:pde}) is given as

1922: \begin{equation}\label{eq:form}

1923: V(x)=\min_i\al_i(z_i-x_i),

1924: \end{equation}

1925: where $\al_i>0$ are constants depending on $c$.

1926: \end{proposition}

1927: We remark that the constants $\al_i$ are uniquely defined by

1928: (\ref{eq:form1}) below.

1929: In the totally symmetric case, where $\mu_i=\mu$, $\la_i=\la$,

1930: $z_i=z$ for all $i$, the solution takes the form

1931: $V(x)=\al\min_i(z-x_i)$. In this case, the optimal

1932: service discipline can be interpreted as ``serve the longest

1933: queue.'' An asymmetric two dimensional example

1934: is given in Figure \ref{fig:example}, where the domain $G$

1935: is divided into two subdomains $G_1$ and $G_2$ in accordance

1936: with the structure (\ref{eq:form}), and the optimal service

1937: discipline corresponds to giving priority to

1938: class $i$ when the state is within $G_i$, $i=1,2$.

1939: Thus the optimal control under our escape-time criterion is very different

1940: from the optimal controls for the average or discounted cost criteria.

1941: 

1942: 

1943: \begin{figure} % export: 30 percent.

1944: \centerline{

1945: \begin{tabular}{cc}

1946: \psfig{file=fig_net_2.eps}\qquad\qquad\qquad

1947: \psfig{file=fig_ex_2.eps}

1948: \end{tabular}

1949: }

1950: \caption{Priority to class $i$ when the state is in $G_i$, $i=1,2$.}

1951: \label{fig:example}

1952: \end{figure}

1953: 

1954: \proof

1955: The constraint directions are given by $\gamma_i=e_i$.

1956: The Hamiltonian is given by

1957: $$

1958: H(p)=\sup_u\inf_m H(p,u,m),

1959: $$

1960: where

1961: $$

1962: H(p,u,m)=c+\sum_i\lt[p_i(\bar\la_i-u_i\bar\mu_i)

1963: +\la_i l\left (\frac{\bar\la_i}{\la_i}\right)

1964: +u_i\mu_il\left(\frac{\bar\mu_i}{\mu_i}\right)\rt].

1965: $$

1966: Using strict convexity and smoothness of the map

1967: $m\mapsto H(p,u,m)$, the minimum over $m$ is attained

1968: at $\bar\la_i=\la_i e^{-p_i}$, $\bar\mu_i=\mu_ie^{p_i}$.

1969: Thus

1970: $$

1971: H(p,u)\doteq \inf_m H(p,u,m)=

1972: c+\sum_i[\la_i(1-e^{-p_i})+u_i\mu_i(1-e^{p_i})].

1973: $$

1974: For the proposed solution, $DV(x)\in-\R_+^J$

1975: wherever the gradient is defined. For $p\in-\R_+^J$, maximizing

1976: $H(p,u)$ over $u$ clearly gives

1977: \begin{equation}\label{eq:ham}

1978: H(p)=\sup_u H(p,u)=c+\sum_i\la_i(1-e^{-p_i})+\max_i\mu_i(1-e^{p_i}).

1979: \end{equation}

1980: We use the well known fact that the definition

1981: of viscosity solutions can be equivalently stated in terms

1982: of sub- and superdifferentials (see \cite{barcap}, Lemma II.1.7).

1983: Note that (\ref{eq:subsb}) and (\ref{eq:supersb})

1984: hold, since $V=0$ on $\pl_oG$.

1985: Hence it suffices to verify

1986: that (\ref{eq:subs}) [resp., (\ref{eq:supers})] holds where

1987: $D\theta(y)$ is replaced by any superdifferential [subdifferential]

1988: of $V$ at $y$. 

1989: 

1990: We show first that the equation $H(DV(x))=0$ holds wherever

1991: $DV$ is defined. 

1992: The proposed form (\ref{eq:form}) satisfies $DV(x)=-\al_ie_i$,

1993: wherever the gradient is defined, with $i=i_x$ depending on $x$.

1994: By the special form of the gradient,

1995: the equation $H(DV(x))=0$ takes the form

1996: \begin{equation}\label{eq:form1}

1997: H(DV(x))=c+\la_i(1-e^{\al_i})+\mu_i(1-e^{-\al_i})=0,

1998: \end{equation}

1999: where $i=i_x$. Denote $c_i=c/(\la_i+\mu_i)$.

2000: Then equivalently,

2001: $

2002: 1+c_i-F_i(\al_i)=0,

2003: $

2004: where

2005: $$

2006: F_i(\al_i)=\frac{\la_ie^{\al_i}+\mu_ie^{-\al_i}}{\la_i+\mu_i}.

2007: $$

2008: The function $F_i$ is strictly convex, $F_i(0)=1$,

2009: and $F_i(\al_i)\to\infty$ as $\al_i\to\infty$. Since $c_i>0$,

2010: it follows that there are unique positive constants

2011: $\al_i$ where $F_i(\al_i)=1+c_i$, $i=1,\ldots,J$.

2012: These are the constants 

2013: in (\ref{eq:form}). In particular, (\ref{eq:form1}) holds

2014: for $i=i_x$, and $H(DV(x))=0$.

2015: 

2016: Next consider any interior point $x$ at which the

2017: gradient is not defined. Clearly

2018: there are no subdifferentials at that point, and any

2019: superdifferential is given as a convex combination of

2020: $-\al_ie_i$, $i=1,\ldots,J$.

2021: Let $B(e_i,\eps)$ be the open ball of radius $\eps$ about $e_i$.

2022: Denote

2023: $\til S=\{\nu\in\R^J:\nu_i\ge0,\sum\nu_i=1\}$,

2024: $S=\{\nu\in\R^J:\nu_i\ge0,\sum\nu_i\le1\}$,

2025: $S_\eps=S\cap\cup_iB(e_i,\eps)$, and $S_\eps^c=S-S_\eps$.

2026: Let $q=-\sum_i\nu_i\al_ie_i$. It suffices to show

2027: that $H(q)\ge0$ for $\nu\in\til S$, but since we later need a stronger

2028: statement than that, we show that in fact $H(q)\ge0$

2029: holds for $\nu\in S$. By (\ref{eq:ham}),

2030: $$

2031: H(q)=c+\sum_i\la_i(1-e^{\nu_i\al_i})+\max_i\mu_i(1-e^{-\nu_i\al_i}).

2032: $$

2033: Define

2034: $$

2035: H^1(q)=c+\sum_i\la_i(1-e^{\nu_i\al_i})+\mu_1(1-e^{-\nu_1\al_1}),

2036: $$

2037: $$

2038: \bar H(q)=c+\sum_i[\la_i(1-e^{\nu_i\al_i})+\mu_i(1-e^{-\nu_i\al_i})].

2039: $$

2040: By (\ref{eq:form1}), $c+\la_i(1-e^{\al_i})+\mu_i\ge0$ and

2041: $c+\la_i(1-e^{\al_i})\le0$, and it follows that there are

2042: constants $A_1,A_2,A_3$ and $A_4$ such that for all $c$ and $i=1,\ldots,J$,

2043: \begin{equation}\label{eq:A}

2044: A_1+\log(c+A_2)\le\al_i\le A_3+\log(c+A_4).

2045: \end{equation}

2046: 

2047: We first consider small perturbations $\nu$ of $e_1$.

2048: To show that $H(q)\ge0$, it suffices to show that $H^1(q)\ge0$.

2049: Note that (\ref{eq:form1}) implies $H^1(q)|_{\nu=e_1}=0$. Also,

2050: $$

2051: \nabla_\nu H^1(q)|_{\nu=e_1}

2052: =(-\la_1\al_1e^{\al_1}+\mu_1\al_1e^{-\al_1})e_1

2053: -\sum_{i\ne1}\la_i\al_ie_i.

2054: $$

2055: Hence, for $\gamma=e_i-e_1$

2056: (where $i\ne1$), using $c+\la_1(1-e^{\al_1})\le0$,

2057: (\ref{eq:form1}) and (\ref{eq:A}),

2058: \beaa

2059: \nabla_\nu H^1(q)|_{\nu=e_1}\cdot\gamma

2060: &=&

2061: \la_1\al_1e^{\al_1}-\mu_1\al_1e^{-\al_1}-\al_i\la_i\\

2062: &=&

2063: \al_1(2\la_1e^{\al_1}-\mu_1-c-\la_1)-\al_i\la_i\\

2064: &\ge&

2065: \al_1(c-\mu_1)-\al_i\la_i\\

2066: &\ge&

2067: [A_1+\log(c+A_2)](c-\mu_1)-[A_3+\log(c+A_4)]\la_i\\

2068: &\ge&

2069: 1,

2070: \eeaa

2071: for all $c$ large. 

2072: Analogous calculations give $\nabla_\nu H^1(q)|_{\nu=e_1}

2073: \cdot\gamma\ge1$ for $\gamma=-e_1$ as well.

2074: As a result, the directional derivatives

2075: $(\pl/\pl\til\gamma)H^1(q)|_{\nu=e_1}$ in the direction $\til\gamma$,

2076: where $\til\gamma$ are of the form

2077: $\til\gamma=(y-e_1)/\|y-e_1\|$, $y\in S$, are bounded below

2078: by $1/2$. Hence $H^1(q)\ge0$ for $\nu\in S$ within

2079: a neighborhood of $e_1$

2080: and $c$ large. Consequently, a similar statement holds for

2081: $H(q)$. Since the same argument holds for neighborhoods of

2082: $e_i$, $i=2,\ldots,J$, we conclude that there is $\eps>0$

2083: and $c_0$ such that $H(q)\ge0$ for $\nu\in S_\eps$ and $c\ge c_0$.

2084: 

2085: Next consider $\nu\in S_\eps^c$.

2086: We first provide a lower bound on $(\pl/\pl c)\bar H(q)$.

2087: Differentiating (\ref{eq:form1}) with respect to $c$,

2088: $\dot\al_i\doteq

2089: \pl\al_i/\pl c=(\la_i e^{\al_i}-\mu_ie^{-\al_i})^{-1}$.

2090: Using (\ref{eq:A}), for all $c$ large,

2091: $0\le\dot\al_i\le(\la_ie^{\al_i}-1)^{-1}$. Using this,

2092: the fact that $\nu$ is bounded away from $\cup_i\{e_i\}$,

2093: and by taking $c$ large, one has

2094: \beaa

2095: \frac{\pl}{\pl c}\bar H(q)

2096: &\ge&

2097: 1-\sum_i\nu_i\la_i\dot\al_ie^{\nu_i\al_i}\\

2098: &\ge&

2099: 1-\sum_i\nu_i[e^{(1-\nu_i)\al_i}-1]^{-1}\\

2100: &\ge&

2101: \frac12.

2102: \eeaa

2103: Note that the above bound holds for all $c\ge c_1$ and

2104: all $\nu\in S_\eps^c$, where $c_1$ is a constant.

2105: It follows that there is $c_2$ such that for all $c\ge c_2$

2106: and all $\nu\in S_\eps^c$, one has $\bar H(q)\ge\sum_i\mu_i$.

2107: Since $H\ge \bar H-\sum_i\mu_i$, $H(q)\ge0$.

2108: We conclude that $H(q)\ge0$ for all $\nu\in S$.

2109: In particular, $H(q)\ge0$ where $q$ is any

2110: superdifferential of $V$ at any interior point.

2111: 

2112: Finally, consider a point $x\in G\cap\pl\R_+^J$. Any superdifferential

2113: of $V$ at $x$ is given as

2114: $q=\sum_{i\in I(x)}\eta_ie_i-\sum_{j=1}^J\nu_j\al_je_j$,

2115: where $\eta_i\ge0$.

2116: If $\max_{i\in I(x)}\lan q,\gamma_i\ran\ge0$, then

2117: (\ref{eq:subs}) holds. Otherwise, $\lan q,\gamma_i\ran<0$

2118: for all $i\in I(x)$. Consequently, any $q$ of the form above

2119: is given as $-\sum_{j=1}^J\nu_j'\al_je_j$, with $\nu'\in S$.

2120: As we have shown, in this case, $H(q)\ge0$. Therefore

2121: (\ref{eq:subs}) holds.

2122: 

2123: Similarly, any subdifferential of $V$ at $x\in G\cap\pl\R_+^J$ is

2124: of the form $-\sum_{i\in I(x)}\eta_ie_i-\sum_{j=1}^J\nu_j\al_je_j$.

2125: In particular, $\lan q,e_i\ran\le0$ for all $i$, and (\ref{eq:supers})

2126: holds.

2127: \qed

2128: 

2129: \section{Proofs of lemmas}\label{sec:lemmas}

2130: 

2131: \noi{\bf Proof of Lemma \ref{lem:dpe}:}

2132: Let 

2133: \[

2134: W^{n}(x)\doteq \inf E_{x}^{u,n}e^{-nc\sigma _{n}}. 

2135: \]

2136: Since $c>0$, $W^{n}$ is well defined.  

2137: Standard iterative methods can be used to construct a solution to the DPE 

2138: \begin{equation}\label{eq:dpe3}

2139: 0=\inf_{u\in U}\left[ \til {\cal L}^{n,u}\bar W^{n}(x)-nc\bar W^{n}(x)\right] ,x\in G^{n}

2140: \end{equation}

2141: and the boundary condition $\bar W^{n}(x)=1$ if $x\notin G^{n}$.

2142: We claim that this solution coincides with the risk-sensitive cost.

2143:  To see this,

2144: consider a controlled Markov process $(X^n,u)$ that starts at $x$.

2145: Then

2146: \[

2147: Y(t) \doteq \bar W^n(X^n(t))-\bar W^n(x)

2148:  -\int_0^t\tilde{\cal L}^{n,u(s)}\bar W^{n}(X^n(s))ds

2149: \]

2150: is a martingale.

2151: Equation (\ref{eq:dpe3}) implies 

2152: ${\cal L}^{n,u(s)}\bar W^{n}(X^n(s))\geq nc\bar W^{n}(X^n(s))$,

2153: and so 

2154: \[

2155: \bar W^n(X^n(t))-\bar W^n(x)-\int_0^t nc \bar W^{n}(X^n(s))ds

2156: =\int_0^tZ(s)ds+Y(t)

2157: \]

2158: for some nonnegative process $Z$.

2159: Using Gronwall's lemma we obtain that for each $t<\infty$

2160: \[

2161: E^{n,u}_x \bar W^n(X^n(t\wedge \sigma^n))e^{-nc(t\wedge \sigma^n)}\geq \bar W^n(x),

2162: \]

2163: and by the Lebesgue Dominated Convergence Theorem 

2164: \[

2165: E^{n,u}_x e^{-nc\sigma^n}\geq \bar W^n(x).

2166: \]

2167: If we define $u$ in terms of the feedback control that minimizes in (\ref{eq:dpe3}) then all the inequalities above become equalities, thus showing that $\bar W^n=W^n$.

2168: 

2169: 

2170: 

2171: The definition of $W^{n\,}$ implies $W^{n}(x)=\exp \left[ -nV^{n}(x)\right] $

2172: . \ If we insert this into the DPE of $\bar W^{n}$ and multiply by $\exp \left[

2173: nV^{n}(x)\right] $ then the equation 

2174: \begin{eqnarray*}

2175: 0& =& \inf_{u\in U}\left[ \sum_{j=1}^{J}n\lambda _{j}\left( \exp \left[

2176: -nV^{n}\left( x+\frac{1}{n}v_{j}\right) +nV^{n}\left( x\right) \right]

2177: -1\right) \right. \\ && \left. \mbox{}  +\sum_{i=1}^{J}n\mu _{i}u_{i}\left( \exp \left[ -nV^{n}\left( x+

2178: \frac{1}{n}\pi (x,\tilde{v}_{i})\right) +nV^{n}\left( x\right) \right]

2179: -1\right) - nc\right] 

2180: \end{eqnarray*}

2181: results. \ Recall the definition $l(x)=x\log x-x+1$ for $x>0$. We now

2182: divide throughout by $n$ and

2183: use the convex duality relation 

2184: \[

2185: \left[ e^{y}-1\right] =\sup_{x>0}[xy-l(x)]

2186: \]

2187: to represent the terms in the previous display. \ For example, in the sum on 

2188: $j$ we take $x=\bar{\lambda}_{j}/\lambda _{j}$ and $y=-\left[

2189: nV^{n}\left( x+\frac{1}{n}v_{j}\right) -nV^{n}\left( x\right) \right] $.

2190: Representing each term in this way and multiplying by $-1$ produces the

2191: first line in (\ref{eq:dpe}). \ The boundary condition that is the second line in (\ref{eq:dpe})

2192: follows directly from the relation between $W^{n}$ and $V^{n}$.

2193: \qed

2194: 

2195: 

2196: 

2197: \noi{\bf Proof of Lemma \ref{lem:unif}:}

2198:  We reduce the Lipschitz property on $(n^{-1}\Z^J_+)\cap\bar G$

2199: to a Lipschitz property near the boundary.

2200: To this end we use the following coupling.

2201:   For $z\in G^n$, let $u^n(z)$ be a minimizer

2202: in (\ref{eq:dpe3}). Given a point $x$ on the lattice, let $X^x$ denote

2203: the process corresponding to the generator $\calL^{n,u^n}$ and starting at $x$

2204: (see the discussion following (\ref{eq:gens})). 

2205: To simplify the notation we will not explicitly denote the dependence of quantities such as $X^x$ on $n$.

2206: Let $u(t)=u^n(X^x(t))$, and let $F_t$ be the filtration generated by $X^x$.

2207: 

2208: Fix a point $ y \not= x $ and let $ X^y $ denote the queueing process

2209: on this probability space that 

2210: starts at $ y $ and  uses the control $u$.

2211: In other words,

2212: $X^y$ is the image,

2213: under the Skorokhod map,

2214: of $y+X^x(\cdot)-x$.

2215: The evolution of the processes $X^x$ and $X^y$ are identical, save that jumps which would cause $X^y$ to leave $\Z^J_+$ are deleted.

2216: Automatically, $u$ is suboptimal for the control problem starting from $y$.

2217:  Define

2218: \be

2219:   V^n(y ; \policy ) = - n^{-1}\log E_x^{\policy ,n} e^{-nc\sig^y }

2220: \ee

2221: where $ \sig^y $ is the exit time of $  X^y $ from $G$.

2222: Note that due to the coupling we may take expectations with respect

2223: to $ E_x^{u,n} $ rather then with respect to $ E_y^{u,n} $.

2224: Since $(X^y,u)$ is a (possibly suboptimal) controlled Markov process, we have

2225: \be\label{eq:VbyVu}

2226: V^n (x ) - V^n ( y ) \leq V^n (x) - V^n (y ; \policy ) .

2227: \ee

2228: 

2229:  

2230: Define $ \sig = \min \{ \sig^x , \sig^y \} $.

2231: By Theorem~\ref{th:SP} on the Lipschitz continuity

2232: of the Skorokhod map we have

2233: \be

2234: \dist (  X^x (\sig ) , \partial G ) \leq K_1 | x - y | , \quad

2235: \dist (  X^y (\sig ) , \partial G ) \leq K_1 | x - y |,

2236: \ee

2237: since at least one of the processes has left $G$ by $ \sig $.

2238: In the last display, $K_1$ is the constant appearing in (\ref{sp:lip}). We claim that

2239: \be\label{dist:1}

2240: V^n (x ) - V^n ( y ) \leq \sup \{ V^n (z) : z \in S \} \ee

2241: where $ S \doteq \{z\in n^{-1}\Z_+^J\cap \bar G:

2242:  \dist (z , \pl_{co} G ) \leq K_1 |x-y| \} $.

2243: To establish this, note that

2244: \beaa

2245: V^n (x ) - V^n ( y , \policy ) 

2246:   &=& - \frac 1 n \left [ \log E_x^{u,n} e^{-nc \sigma^x } 

2247:                               - \log E^{u,n}_x e^{-nc \sigma^y } \right ] \\

2248:  & \leq & - \frac 1 n \left [ \log E_x^{u,n} \left [ e^{-nc \sigma }

2249:         E_x^{u,n} \left ( e^{-nc (\sigma^x - \sigma )} \left |  X^x (\sigma)

2250:            \right .  \right ) \right ] - \log E_x^{u,n} e^{-nc \sigma} \right ] \\

2251:  & \leq & \sup_{z\in S} - \frac 1 n \left [ \log E_x^{u,n} \left [ e^{-nc \sigma }

2252:                  E_x^{u,n} \left ( e^{-nc (\sigma^x - \sigma )} 

2253:                \left |  X^x (\sigma) = z \right .  \right ) \right ]

2254:               - \log E_x^{u,n} e^{-nc \sigma} \right ]\\

2255: & =     & \sup_{z\in S} - \frac 1 n \left [ \log \left [ E_x^{u,n} 

2256:              \left ( e^{-nc (\sigma^x - \sigma )} \left |  X^x (\sigma)  = z 

2257:             \right .  \right ) E_x^{u,n} e^{-nc \sigma } \right ]

2258:               - \log E_x^{u,n} e^{-nc \sigma} \right ]\\

2259:  & = & \sup_{z\in S} - \frac 1 n \left [ \log E_x^{u,n} \left ( 

2260:            e^{-nc (\sigma^x - \sigma )} \left |  X^x (\sigma)  = z 

2261:                \right .  \right ) \right ]

2262: \eeaa

2263: However, by the strong Markov property,

2264: $$

2265: - \frac 1 n \left [ \log E_x^{u,n} \left ( e^{-nc (\sigma^x - \sigma )} 

2266:                 \left |  X^x (\sigma)  = z \right .  \right ) \right ]

2267: \leq \sup_u - \frac 1 n \log \left ( E_z^{u,n} e^{-nc \sigma^z }\right ) 

2268: = V^n (z)

2269: $$

2270: and together with~(\ref{eq:VbyVu}) we have~(\ref{dist:1}).

2271: 

2272: To prove the lemma, one needs to show that

2273: $|V_n(x)-V_n(y)|\le c_0|x-y|$ for all $n$ and all

2274: $x,y\in(n^{-1}\Z_+^J)\cap\bar G$, where $c_0$ does not depend

2275: on $x,y$ and $n$. It suffices to prove this inequality for

2276: $x,y$ such that $|x-y|=n^{-1}$.

2277: Since the roles of $x $ and $y$ are symmetric, and in view of

2278: (\ref{dist:1}), it suffices to show that

2279: for $\{x\in G:\dist(x,\pl_{co} G)\le K_1n^{-1}\}$,

2280: \begin{equation}\label{eq:55}

2281:  V_n(x)=-n^{-1}\log\inf_uE_x^{u,n}e^{-nc\sig^x}\le c_1n^{-1},

2282: \ee

2283: where $c_1 > 0$ is a constant.

2284: 

2285: Let us first treat the case where $G$ is not a rectangle.

2286: In that case, Condition \ref{cond:G} implies that for

2287: any $x$ with $\dist(x,\pl_{co} G)\le K_1n^{-1}$,

2288: \be\label{imp:1}

2289: \mbox{ there is } i\in\calJ_+\ \mbox{ such that }\ x+c'n^{-1}e_i\not\in G,

2290: \ee

2291: where $c'$ is a constant.

2292: Let such $i$ be fixed.

2293: To show (\ref{eq:55}), it is enough to show that for any

2294: $x$ such that $\dist(x,\pl_{co}G)\le K_1n^{-1}$, and any

2295: $n$ and $u$,

2296: \begin{equation}\label{eq:56}

2297:  E_x^{u,n}e^{-nc\sig^x}\ge c_2>0.

2298: \ee

2299: Recall that $\la_i>0$. Let $S_t$ denote the event that

2300: all service processes and all arrival processes, except for the

2301: one corresponding to $i$, do not increase on $[0,t]$.

2302: Recall that the expected time till a Poisson process of rate $\la$

2303: hits level $K$ is $K/\la$.

2304: Then for any $\al\in(0,1)$

2305: \beaa

2306:  E_x^{u,n}e^{-nc\sig^x} & \ge & \al P_x^{u,n}(e^{-nc\sig^x}>\al)\\

2307:  &=&

2308: \al P_x^{u,n}\lt(\sig^x<-\frac{\log\al}{nc}\rt).

2309: \eeaa

2310: Choosing $t_0=-(\log\al)/nc = 2\tilde c/n\la_i$

2311: and using $P_x^{u,n}(\sig^x<2E\sig^x)\ge1/2$,

2312: \beaa

2313:  E_x^{u,n}e^{-nc\sig^x} & \ge & \al P_x^{u,n}(\sig^x<t_0|S_{t_0})P_x^{u,n}(S_{t_0})\\

2314:  &\ge&

2315: e^{-2 c \tilde c / \la_i } \frac 1 2 c_3

2316: \eeaa

2317: where $c_3 > 0 $ is the probability that a Poisson process with rate

2318: $ n c_4 $ has not jumped by time $ t_0 = 2 \tilde c / \la_i n $.

2319: This proves (\ref{eq:56}), which implies (\ref{eq:55}),

2320: and hence the statement of the lemma holds.

2321: 

2322: In the case where $G$ is a rectangle,

2323: the bound~(\ref{dist:1}) does not suffice since $V^n (x) $ is

2324: discontinuous near $\pl_cG$.

2325: We therefore prove that a similar bound applies, where there

2326: supremum is over $S=\{z\in(n^{-1}\Z_+^J)\cap\bar G:

2327: \dist(z,\pl_oG)\le K_1|x-y|\}$.

2328: To apply the previous argument we need to show that if $  X^x (t)$ is

2329: close to $\pl_cG$,

2330: then neither $  X^x $ nor $ X^y $ will exit (locally) through that boundary.

2331: This is clear for $  X^x $: the only

2332: way for the process to leave $G$ is due to a service to one of the queues,

2333: say, queue $j$, leading to an increase in queue $i$.

2334: However, allowing this service is certainly not optimal: it is better to

2335: avoid this control, as our objective is to increase $ \sigma^x $.

2336: To prevent $  X^y $ from exiting we need to modify the coupling argument as

2337: follows. The control $u^y$ used by $ X^y$ avoids a jump that leads

2338: $  X^y $ to exit through $\pl_cG$ (that is, queue $j$ above will not be

2339: served if $  X_i^y (t) = z_i - 1 $.) Note that this is the only possible

2340: type of jump that leads the process out of $G$.

2341: Moreover, the $ \ell_1 $ distance

2342: between $  X^x $ and $ X^y $ may only decrease due to this change in

2343: control: the control is changed only if $  X^x_i (t) < X^y_i (t) $,

2344: and following the service $  X^x_i $ increases by $1$ so that

2345: $ |  X^x_i (t) -  X^y_i (t) | $ decreases by $1$, while

2346: $  X^x_j $ decreases by $1$.

2347: 

2348: Condition \ref{cond:G} still

2349: implies (\ref{imp:1}), but only for $x$ such that

2350: $$

2351: \mbox{$z_i-x_i\le n^{-1}$, for some $i\in\calJ_+$.}

2352: $$

2353: For such $x$, the argument in the last paragraph holds.

2354: However, for $x$ near $\pl_c G$

2355: there is nothing to prove, since the process never exits through such a

2356: boundary.

2357: \qed

2358: 

2359:  

2360: \noi{\bf Proof of Lemma \ref{lem:bound:m}:}

2361: The first part is an immediate consequence of the fact that

2362:  $ u_i \geq 0 $ and both

2363: $\calL^{n,u,m}V^n(x)$ and $\rho(u,m)$ depend on $u$ as $\sum_iu_i\eta_i$,

2364: where $\eta_i$ is a function of $m_i ,x,n$ but not of $u$.

2365: 

2366: For the second part of the lemma, one can explicitly solve for

2367: $m^n$ in terms of $V^n$, and get $m^n(x,u)=m^n(x)=

2368: ((\bar\la^n_i(x)),(\bar\mu^n_i(x)))$,

2369: where

2370: $$

2371:  \bar\la^n_i(x)=\la_ie^{-n\del_i V^n(x)}, \quad

2372:  \bar\mu^n_i=\mu_ie^{-n\tilde\del_i V^n(x)},

2373: $$

2374: and

2375: $$

2376:  \del_i V^n(x)\doteq V^n(x+n^{-1}v_i)-V^n(x), \quad

2377:  \tilde\del_i V^n(x)\doteq V^n(x+n^{-1}\pi(x,\tilde v_i))-V^n(x).

2378: $$

2379: The result follows from Lemma \ref{lem:unif},

2380: since it shows that there is a constant $b_2$

2381: independent of $x,n$ where

2382: $$

2383:  n\del_i V^n(x) \ge-b_2, \quad

2384:  n\tilde\del_i V^n(x) \ge-b_2.

2385: $$

2386: \qed

2387: 

2388: \noi{\bf Proof of Lemma \ref{lem:cont}:}

2389: We fix $b\in [b^*,\infty]$ and suppress it from the notation throughout

2390: the proof.

2391: Item 1 of the lemma is trivial under Condition \ref{cond:G}.1.

2392: Under Condition \ref{cond:G}.2, by continuity of the functions

2393: $ \phi_i$, we only need to show is that $\pl_{co}G_a$ and $\pl_{co}G$

2394: do not intersect. Consider first $a>0$, and let $x\in\pl_{co}G_a$.

2395: Then $x_i=a+ \phi_i(x_1,\ldots,x_{i-1},x_{i+1},\ldots,x_J)$

2396: for some $i\in\calJ_+$, and therefore

2397: $x$ cannot belong to the closure of $G$. The proof for $a<0$ is

2398: similar.

2399: 

2400: Let $\beta_0[u](t)=m_0$ for all $u,t$, where $m_0$ sets all $\bar\la_i=

2401: \la_i$ and $\bar\mu_i=0$. Then $\rho(u(t),\beta_0[u](t))$ is bounded

2402: by a constant, and the dynamics, unaffected by $u$, follow $ X(t)=

2403: x+\sum_i\la_ie_it$ and leave the bounded set $G$ within a finite time bounded

2404: by ${\rm diam}(G)/\max_i\la_i<\infty$. Therefore

2405: \[

2406: V^- (x)\le \sup_u C (x,u,m_0) = C(x,\beta_0) \le c_1< + \infty .

2407: \]

2408: Similarly,

2409: \[

2410: V^+ (x)\le \sup_\alpha C (x,\alpha (m_0),m_0) \le c_1 < + \infty .

2411: \]

2412: 

2413: It is useful to notice that for all $a\in(0,a_0)$

2414: and $y\in\pl_o G$ there is $i=i_y\in\calJ_+$ such that

2415: $y+2ae_{i_y}\not\in G_a$. Similarly, for all $a\in(-a_0,0)$

2416: and $y\in\pl_o G_a$ there is $i=i_y\in\calJ_+$ such that

2417: $y+2ae_{i_y}\not\in G$.

2418: 

2419: First consider $a>0$ and recall that $ \sigma_a $ (resp., $\sigma $)

2420: is the exit time from $G_a $ (resp., $G$), so that for any fixed $ u$ and

2421: $ \beta [u] $ we have $ \sigma \leq \sigma_a $.

2422: Therefore, since $c$ and $ \rho $ are positive,

2423: \beaa

2424:  V^-_a(x)

2425:  &=    & \inf_\beta\sup_u \int_0^{\sig_a} (c+\rho(u(s), \beta[u](s))ds \\

2426:  &\geq & \inf_\beta\sup_u \int_0^{\sig} (c+\rho(u(s), \beta[u](s))ds \\

2427:  & = & V^- (x) .

2428: \eeaa

2429: Thus to prove the Lipschitz property a one-sided bound suffices.

2430: Recall that $ \phi (\sigma ) $ is the exit point from $ G$ and

2431: for each $\beta$ define the extension $\beta_a$ by

2432: $$

2433:  \beta_a[u]=\lt\{\begin{array}{ll}\beta[u](t) & t\in[0,\sig), \\ 

2434:  \hat m & t\in[\sig,\infty),

2435:  \end{array}\rt.

2436: $$

2437: where $\hat m$ sets all $\bar\mu_j=0$ and

2438: $\bar\la_j=1_{j=i_{\phi(\sig)}}$.

2439: Then for any $\beta$

2440: \beaa

2441:  V^-_a(x)

2442:  &=& \inf_{\beta}\sup_u C_a(x,\beta[u],u) \\

2443:  &\le& \inf_{\beta}\sup_u C_a(x,\beta_a[u],u)\\

2444:  &=& \inf_\beta\sup_u\lt[C(x,\beta[u],u)+\int_\sig^{\sig_a}

2445:  (c+\rho(u(s),\hat m)ds\rt] \\

2446:  &\le&

2447: V^-(x)+c_1a,

2448: \eeaa

2449: where the last line follows since $\rho(u(s),\hat m)$ is bounded

2450: and since by the previous paragraph, $\sig_a-\sig\le 2a$.

2451: Note that $c_1$ does not depend on $b\in[b^*,\infty]$.

2452: For $a<0$ the same argument shows that $V^-(x)\le V^-_a(x)+c_3|a|$,

2453: by interchanging the roles of $G$ and $G_a$.

2454: 

2455: 

2456: For $ V_a^+ (x) $ note that an argument as above gives 

2457: $ V_a^+ (x) \geq V^+ (x) $. For each $m$ define $ m_a $ by

2458: $$

2459:  m_a = \lt\{ \begin{array}{ll} m (t) & t\in[0,\sig), \\ 

2460:  \hat m & t\in[\sig,\infty),  

2461:  \end{array}\rt .

2462: $$

2463: where $ \hat m $ is as above. Let $ \alpha_\epsilon $ be an $\epsilon

2464: $-optimal strategy. Then, since for any fixed $ u$ and

2465: $ m $ we have $ \sigma \leq \sigma_a $,

2466: \beaa

2467:  V^+_a(x)

2468:  &=& \sup_\alpha \inf_m C_a (x,m, \alpha[m]) \\

2469:  & \leq & \inf_m C_a (x,m, \alpha_\epsilon [m]) + \epsilon \\

2470:  & \leq & \inf_m C_a (x,m_a , \alpha_\epsilon [m_a ]) + \epsilon,

2471: \eeaa

2472: since we are taking the infimum over a smaller class of controls.

2473: By the definition of $ C_a $,

2474: \beaa

2475: V^+_a(x)

2476:  & \leq & \inf_m C (x, m , \alpha_\epsilon [m ])

2477:            + \int_\sigma^{\sigma_a}

2478:                (c+\rho(\alpha_\epsilon [m_a ](s),\hat m (s))\, ds + \epsilon \\

2479:  & \leq & \sup_\alpha \inf_m C (x,m, \alpha[m]) + c_2 a + \epsilon

2480: \eeaa

2481: by the previous argument,

2482: where $c_2 $ does not depend on $x$, $ \epsilon $ and $b$.

2483: Since $ \epsilon $ is

2484: arbitrarily small, the proof for $a>0 $ and $ V_a^+ (x) $ is established.

2485: \qed

2486: 

2487: \noi{\bf Proof of Lemma \ref{lem:apriori}:}

2488: We suppress $b$ from the notation, throughout the proof.

2489: It is obvious that one can restrict the infimum

2490: over $\beta\in B$ to the class of strategies

2491: $\beta$ for which $C(x,\beta)\le V^-(x)+1$.

2492: Within this class, for every $\beta$ and $u$,

2493: $$

2494: \sigma c \leq \int_0^\sig[c+\rho(u(t),\beta[u](t))]dt\le V^-(x)+1,

2495: $$

2496: and therefore one always has that $\sig\le T_0\doteq(V(x)+1)/c$.

2497: Lemma \ref{lem:apriori} asserts an upper bound on the cost till time a fixed time $T_0$,

2498: and so we must define the strategy for times $t \in [\sigma, T_0]$.

2499: Let $\hat m$ be an arbitrary fixed element of $M$.

2500: Then the extended $\beta$ is just 

2501: $$

2502:  \hat\beta[u](t)=\lt\{\begin{array}{ll}\beta[u](t) & t<\sig.\\ 

2503:  \hat m & t\ge\sig,

2504:  \end{array}\rt.

2505: $$

2506: With this definition

2507: one has that $C(x,u,\beta[u])=C(x,u,\hat\beta[u])$. One can

2508: therefore further restrict to strategies $\beta$ satisfying

2509: $\hat\beta=\beta$. For such $\beta$, it follows that

2510: $$

2511:  \int_0^{T_0}\rho(u(t),\beta[u](t))dt \le c_1T_0,

2512: $$

2513: where $c_1$ does not depend on $u,\beta$.

2514: The result regarding $V^-$ follows.

2515: 

2516: Regarding $V^+$,

2517: let $m_0$ be a control which sets all $\mu_i$ and $\la_i$ to

2518: zero, except that $\la_{i_0}=1$ for some $i_0\in\calJ_+$.

2519: Then for any $\al\in A$ and $m$ for which

2520: \begin{equation}\label{eq:20}

2521: C(x,\al[m],m)\le

2522: C(x,\al[m_0],m_0),

2523: \end{equation}

2524: one has $c\sig(x,\al[m],m)

2525: \le C(x,\al[m],m)\le

2526: C(x,\al[m_0],m_0)\le c_1<\infty$. Note that $c_1$ can be chosen

2527: independent of $\al$, since the dynamics and running cost

2528: under $m_0$ are independent of $\al$. Clearly, for each $\al$ it suffices

2529: to consider, in optimizing over $m$, only those $m$ that satisfy

2530: (\ref{eq:20}). It follows that it suffices to consider only

2531: those $m$ for which $\s(x,\al[m],m)\le c_1/c$.

2532: This completes the proof of the lemma.

2533: \qed

2534: 

2535: 

2536: \noi{\bf Proof of Lemma \ref{lem:vallip}:}

2537: Fix $ b\in[b^*,\infty]$ which we omit from the notation.

2538: Recall from Lemma \ref{lem:cont} that $V^\pm$ are bounded

2539: on $G$.

2540: We first show that $V^-$ is Lipschitz. Assume first that

2541: Condition \ref{cond:G}.2 holds.

2542: Recall that for $x\in G$,

2543: $$

2544:  V^-(x)=\inf_\beta\sup_u C(x,\beta[u],u).

2545: $$

2546: Let $\beta_\eps^x$ be an $\eps$-optimal strategy starting

2547: from $x$, i.e.,

2548: $$

2549: \sup_u C(x,\beta_\eps^x[u],u)\le V^-(x)+\eps.

2550: $$

2551: For any $z\in G$ let $\s_z=\inf\{t:\phi_z\not\in

2552: G\}$, where $\phi_z$ is the solution to

2553: $\dot\phi=\pi(\phi,v(u,\beta_\eps^x[u]))$, with $\phi(0)=z$.

2554: Note that $C(x,u,\beta_\eps^x[u])=\int_0^{\s_x}[c+\rho(u(t),

2555: \beta_\eps^x[u](t))]dt$ (with possibly $ \sigma_x = \infty $).

2556: Now let $y\in G$. Note that on $[0,\s_x\w\s_y]$, one has by

2557: the Lipschitz property of the Skorokhod map

2558: that $|\phi_x (t) -\phi_y (t)|\le c_1|x-y|$,

2559: where $c_1$ is some constant. 

2560: Recall that we are considering the case of Condition \ref{cond:G}.2.

2561: Therefore, at $\s_x\w \s_y$, both

2562: $\phi_x$ and $\phi_y$ are within a distance of $c_1|x-y|$ of

2563: the boundary $\pl_o G$. Because of the assumptions on the domain $G$,

2564: there exists a constant $c_2$

2565:  such that at time $\s_x\w\s_y$, both

2566: $\phi_x+c_2e_{i^*}\not\in G$ and

2567: $\phi_y+c_2e_{i^*}\not\in G$, where $i^*\in\calJ_+$

2568: and, moreover, $c_2 $ is

2569: independent of $ x,y, b \geq b^* $ and $ i^* $.

2570: 

2571: Define $\beta_\eps^{x,y}$ as

2572: $\beta_\eps^{x,y}[u]=\beta_\eps^x[u]$ on $[0,\sig_x)$, and,

2573: if $\sig_y\ge\sig_x$, set $\beta_\eps^{x,y}[u]=m_0$ on $[\s_x,\s_y]$.

2574: Here, $m_0$ sets all $\bar\mu_i$ and all $\bar\la_i$ to zero, except

2575: that it sets $\bar\la_{i^*}=\la_{i^*}$, where $i^*$ is as above.

2576: Consequently, $ \sig_y < \sig_x + c^\prime | x - y | $ for some 

2577: $ c^\prime > 0 $, and there exists $u_\eps$ such that

2578: \beaa

2579:  V^-(y) &\le& \sup_u C(y,u,\beta_\eps^{x,y}[u]) \\

2580:  &\le&

2581: \int_0^{\s_y}[c+\rho(u_\eps(s),\beta_\eps^{x,y}[u_\eps](s))]ds+\eps\\

2582:  &\le&

2583: \int_0^{\s_x}[c+\rho(u_\eps(s),\beta_\eps^{x,y}[u_\eps](s))]ds

2584:  +1_{\s_x<\s_y}

2585:  \int_{\s_x}^{\s_y}[c+\rho(u_\eps(s),\beta_\eps^{x,y}[u_\eps](s))]ds

2586:  +\eps\\

2587:  &\le&

2588: C(x,u_\eps,\beta_\eps^{x}[u_\eps])+c_3|x-y|+\eps\\

2589:  &\le&

2590: \sup_u C(x,u,\beta_\eps^x[u])+c_3|x-y|+\eps\\

2591:  &\le&

2592: V^-(x)+c_3|x-y|+2\eps.

2593: \eeaa

2594: Since $x,y\in G$ and $\eps>0$ are arbitrary, and $c_3$ does not

2595: depend on them or on $ b$, $V^-$ is Lipschitz, uniformly for

2596: $ b\in[b^*,\infty] $.

2597: 

2598: In case that Condition \ref{cond:G}.1 holds, the same

2599: argument shows that $V^-_a(y)\le V^-(x)+c_3|x-y|+2\eps$,

2600: where $a=c_1|x-y|$. By Lemma \ref{lem:cont}, this implies that

2601: $V^-(y)\le V^-(x)+c_4|x-y|+2\eps$, some constant $c_4$, and therefore

2602: $V^-$ is Lipschitz.

2603: 

2604: Next, consider the upper value

2605: $$

2606:  V^+(x)=\sup_\al\inf_mC(y,\al[m],m)

2607: $$

2608: under Condition \ref{cond:G}.2.

2609: Let $x,y\in G$. Note that there is an $\al_\eps^x$ such that

2610: $$

2611:  V^+(x)\le\inf_m C(x,\al_\eps^x[m],m)+\eps.

2612: $$

2613: and an $m_\eps=m_\eps(x,y)$ for which

2614: \beaa

2615: V^+(y) &\ge& \inf_m C(y,\al_\eps^x[m],m) \\

2616:  &\ge& C(y,\al_\eps^x[m_\eps],m_\eps)-\eps.

2617: \eeaa

2618: Let $\s_z=\inf\{t:\phi_z\not\in G\}$, where $\phi_z$ is the solution

2619: to $\dot\phi=\pi(\phi,v(\al_\eps^x[m_\eps],m_\eps))$, with

2620: $\phi(0)=z$. Let $i^*$ be defined in an analogous way to that in

2621: the first paragraph of the proof. Now

2622: define $\bar m_\eps=\bar m_\eps(x,y)$ as follows. If

2623: $\sig_x\le\sig_y$, let $\bar m_\eps=m_\eps$. If $\sig_y<\sig_x$, let

2624: $\bar m_\eps$ agree with $m_\eps$ on $[0,\sig_y)$ and with

2625: $m_0$ on $[\sig_y,\s_x]$. Here, $m_0$ sets all $\bar\mu_i$

2626: and all $\bar\la_i$ to zero, except that it sets $\bar\la_{i^*}

2627: =\la_{i^*}$.

2628: Since $m_\eps$ and $\bar m_\eps$ agree on $[0,\s_y)$, the restrictions

2629: to $[0,\sig_y]$ of $\al_\eps^x[m_\eps]$ and of $\al_\eps^x[\bar m_\eps]$

2630: agree a.e.\ on $[0,\sig_y]$, and therefore,

2631: $$

2632:  C(y,\al_\eps^x[m_\eps],m_\eps)=C(y,

2633:  \al_\eps^x[\bar m_\eps],\bar m_\eps).

2634: $$

2635: Arguing again by the Lipschitz property of the

2636: Skorokhod map and the definition of $m_0 $,

2637: there is a constant $c_4$ for which

2638: $(\sig_x-\sig_y)^+\le c_4|x-y|$. Hence

2639: \beaa

2640: V^+(y) &\ge& C(y,\al_\eps^x[\bar m_\eps],\bar m_\eps)-\eps \\

2641: &\ge & \int_0^{\sig_x}[c+\rho(\al_\eps^x[\bar m_\eps^x](s),\bar m_\eps^x(s))]

2642: ds - 1_{\sig_y<\sig_x}\int_{\sig_y}^{\sig_x}

2643: [c+\rho(\al_\eps^x[\bar m_\eps^x](s),\bar m_\eps^x(s))]ds -\eps\\

2644: &\ge& C(x,\al_\eps^x[\bar m_\eps],\bar m_\eps)-c_5|x-y|-\eps\\

2645: &\ge& \inf_m C(x,\al_\eps^x[m],m)-c_5|x-y|-\eps\\

2646: &\ge& V^+(x)-c_5|x-y|-2\eps.

2647: \eeaa

2648: Since $c_5$ does not depend on $x,y,\eps$ or $b$, we have that $V^+$

2649: is Lipschitz uniformly for $b\in[b^*,\infty] $.

2650: 

2651: Under Condition \ref{cond:G}.1, the same argument shows that

2652: $V^+_a(y)\ge V^+(x)-c_5|x-y|-2\eps$, where $a=c_1|x-y|$, and

2653: again one argues by Lemma \ref{lem:cont}.

2654: \qed

2655: 

2656: 

2657: \noindent{\bf Proof of Lemma \ref{lem:xbar}:}

2658: The processes are constructed recursively using a sequence

2659: of standard exponential clocks.

2660: Recall that $\calL^{n,u,m}$ is given for every $n$, $u\in U$,

2661: $m\in M$ by

2662: \[

2663: \calL^{n,u,m}f(x) =

2664: \sum_{j=1}^J n \bar \lambda_j [f(x+{n}^{-1} v_j)-f(x)]

2665: + \sum_{i=1}^J n \bar \mu_i u_i [f(x+n^{-1}\pi(x,\tilde v_i))-f(x)].

2666: \]

2667: Given $n$, $x_n$ and $\beta$, we construct a filtered

2668: probability space and three processes, $\bar X(t)$, $\bar u(t)$

2669: and $\bar m(t)$ (to simplify notation, we do not write the superscript

2670: $n$ in the notation of $\bar X^n$, $\bar u^n$ and $\bar m^n$)

2671: such that (a) $\bar X,\bar u$ and $\bar m$

2672: are $(\bar F_t)$-adapted;

2673: (b) $\bar m(t)=\beta[\bar u](t)$ a.e.\ $t\ge0$, a.s.;

2674: (c) $\bar u(\cdot )=u^n(X(\cdot ))$ a.s.\ (where $u^n$ is as in

2675: the statement before the lemma); and (d) for any $f$, the process

2676: $$

2677: f(\bar X(t))-\int_0^t\calL^{n,\bar u(s),\bar m(s)}f(\bar X(s))ds

2678: $$

2679: is an $(\bar F_t)$-martingale.

2680: For (a--d) to hold, it suffices that (a--c) hold, and (e)

2681: on any finite interval the process $\bar X$ jumps finitely many 

2682: times---we denote

2683: the $k$th jump by $\tau_k$ and let $\tau_0=0$; (f) the random times

2684: $(\tau_k)$ are

2685: stopping times on $(\bar F_t)$, and (g) denoting $X_k=\bar X(\tau_k)$,

2686: for any $k$,

2687: $$

2688: \bar E[f(X_{k+1})-f(X_k)|\bar F_{\tau_k}] =

2689:  \sum_{i=1}^J \bar E[A_i^{k,\bar u,\bar m}

2690:  +B_i^{k,\bar u,\bar m}|\bar F_{\tau_k}],

2691: $$

2692: where

2693: $$

2694: A_i^{k,\bar u,\bar m}= n\int_{\tau_k}^{\tau_{k+1}}

2695:    \bar\la_i(s)ds [f(X_k+n^{-1}v_i)-f(X_k)],

2696: $$

2697: $$

2698: B_i^{k,\bar u,\bar m} = n\int_{\tau_k}^{\tau_{k+1}} \bar\mu_i(s)u_i(s)ds

2699:     [f(X_k+n^{-1}\pi(X_k,\tilde v_i))-f(X_k)].

2700: $$

2701: 

2702: The construction is recursive.  On a complete probability

2703: space $(\bar\Om,\bar F,\bar P)$ we are given $2J$ independent

2704: i.i.d.\ standard Poisson processes, denoted

2705: $a_i$ and $b_i$, $i=1,\ldots,J$. Let $T^a_i(k)$ [resp., $T^b_i(k)$]

2706: denote the first time $a_i$ [resp., $b_i$] equals $k$.

2707: For each $\om\in\Om$ we construct recursively a sequence of times

2708: $(\tau_k)$ and the processes $\bar X, \bar u$ and $\bar m$

2709: up to time $\tau_k$. Once these processes are

2710: defined, we will define $(\bar F_t)$, $\bar F_t\subset \bar F$,

2711: $t\ge0$, and verify

2712: that items (a--c), (e--g) are satisfied on

2713: $(\bar\Om,\bar F,(\bar F_t),\bar P)$.

2714: 

2715: We set $\bar X(0)=x_n$ and $\bar u(0)=u^n(x_n)$.

2716: Since $\bar m$ need only be defined

2717: almost everywhere on $[0,\infty)$, we do not define it at zero nor

2718: at any $\tau_k$, $k=1,2,\ldots$.

2719: Now assume that we have constructed $\tau_i$, $i\le k$ as well as

2720: the processes $\bar X$ and $\bar u$ on

2721: $[0, \tau_k]$ and $\bar m$ a.e.\ on $[0,\tau_k]$.

2722: Let

2723: $\hat u^k(t)=u^n(\bar X(t\w \tau_k))$, $t\ge0$. Let also

2724: $\hat m^k=\beta[\hat u^k]$.

2725: With $\hat u^k(\cdot )=(\hat u_i^k(\cdot ))$ and

2726: $\hat m^k(\cdot )=((\hat\la_i^k(\cdot )),(\hat\mu_i^k(\cdot )))$, let

2727: $$

2728: p_i^k(t)=n\int_0^t\hat\la_i^k(s)ds,

2729: \qquad q_i^k(t)=n\int_0^t\hat\mu_i^k(s)\hat u_i^k(s)ds,

2730: \qquad i=1,\ldots,J, \ t\ge0.

2731: $$

2732: Denoting $\Del z(s)=z(s)-z(s-)$, let also

2733: $$

2734:  \tau_{k+1}=\inf\{t>\tau_k: \mbox{ either $\Del a_i(p_i^k(t))>0$

2735:  or $\Del b_i(q_i^k(t))>0$ for some $i=1,\ldots,J$}\},

2736: $$

2737: where $\inf\emptyset=+\infty$.

2738: We first consider the case that $\tau_{k+1}<+\infty$.

2739: In this case,

2740: \be\label{eq:exi}

2741: \mbox{there is $i$ such that either $\Del a_i(p_i^k(\tau_{k+1}))>0$ or

2742: $\Del b_i(q_i^k(\tau_{k+1}))>0$.}

2743: \ee

2744: In the former case we let

2745: $\hat v^k= v_i$; otherwise we let $\hat v^k=\tilde v_i$.

2746: 

2747: The three processes are defined on the next interval

2748: as follows. Let $\bar X(t)=\bar X(\tau_k)$ for  $t\in(\tau_k,\tau_{k+1})$,

2749: and $\bar X(\tau_{k+1})=\bar X(\tau_k)+n^{-1}\pi(\bar X(\tau_k),\hat v^k)$.

2750: Let $\bar u(t)=u^n(\bar X(t))$ for $t\in(\tau_k,\tau_{k+1}]$.

2751: Let $\check u(t)=u^n(\bar X(t\w\tau_{k+1}))$ and define

2752: $\bar m(t)=\beta[\check u](t)$, $t\in[0,\tau_{k+1}]$.

2753: Note that since $\beta$ is a strategy,

2754: this definition of $\bar m$ is consistent with its definition

2755: up to $\tau_k$ since so is the definition of $\bar u$.

2756: For the same reason, for a.e.\ $t\le\tau_{k+1}$,

2757: $\bar m(t)=\hat m^k(t)$.

2758: In particular, the equations for $p_i^k, q_i^k$ still hold

2759: if we replace hats by bars, namely,

2760: \be\label{eq:pq}

2761: p_i^k(t)=n\int_0^t\bar\la_i(s)ds,

2762: \quad q_i^k(t)=n\int_0^t\bar\mu_i(s)\bar u_i(s)ds,

2763: \quad i=1,\ldots,J, \ \tau_k\le t\le \tau_{k+1}.

2764: \ee

2765: Note that the above relations are consistent in the sense that

2766: for a given $k$, they hold not only for

2767: $t\in[\tau_k,\tau_{k+1}]$, but in fact for $t\in[0,\tau_{k+1}]$.

2768: Hence, on the event $\tau_k\to\infty$, one can equivalently

2769: consider the processes

2770: \be\label{def:pq}

2771: p_i(t)=n\int_0^t\bar\la_i(s)ds,

2772: \quad q_i(t)=n\int_0^t\bar\mu_i(s)\bar u_i(s)ds,

2773: \quad i=1,\ldots,J, \ t\ge0.

2774: \ee

2775: This completes the definition of the three processes on $[0,\tau_{k+1}]$.

2776: 

2777: In case that $\tau_{k+1}=+\infty$, the definitions above of

2778: $\bar X$, $\bar u$ and $\bar m$

2779: all apply on $(\tau_k,\tau_{k+1})$ and there is

2780: nothing else to define.

2781: 

2782: To complete the construction of the three processes on

2783: $\bar\Om\times[0,+\infty)$, we must consider the set $\Om_0$

2784: of $\om\in\bar\Om$

2785: for which $\bar T\doteq\sup\tau_k$ is finite.

2786: We show that this set is $\bar P$-null

2787: owing to the fact that the range $\bar M^b $ of $\beta$ consists

2788: of bounded functions.

2789: Suppose $\bar T$ is finite. The construction above defines $\bar X,\bar u$

2790: and $\bar m$

2791: on $[0,\bar T)$. Let $\bar u'(t)=\bar u(t)$ for $t<\bar T$

2792: and define $\bar u'(t)$ arbitrarily

2793: on $[\bar T,+\infty)$ but such that $\bar u'\in\bar U$. Then $\bar m'

2794: =\beta[\bar u']$

2795: agrees with $\bar m$ a.e.\ on $[0,\bar T]$. Since each component of $\bar m'$

2796: is bounded by $b$,

2797: \be\label{eq:boundsum}

2798:  n^{-1}\max_{i=1}^J [p_i(\bar T)\vee q_i(\bar T)]\le

2799:  2J\bar T b <+\infty.

2800: \ee

2801: However, by construction, $\bar T<\infty$ implies that

2802: either $a_i(p_i(t))\to\infty$ or

2803: $b_i(q_i(t))\to\infty$ as $t\uparrow\bar T$, for some $i$.

2804: Hence $\bar T<\infty$ must be a null set.

2805: We let $\bar X,\bar u$ and $\bar m$ be defined arbitrarily on $\Om_0$.

2806: 

2807: The definition of the process $\bar Y$ is similar to that of

2808: $\bar X$, but where $\pi(x,v)$ is replaced by $v$ throughout.

2809: The relation $\bar X=\Gamma(\bar Y)$ is clear from the construction.

2810: 

2811: Define for each $t\ge0$ $\bar F_t$ to be the $\sig$-field

2812: generated by $\{ \bar Y(s), s\in[0,t] \} $. Note that it is

2813: equivalently defined as the $\sig$-field generated by

2814: $\{a_i(p_i(t)), b_i(q_i(t)), i=1,\ldots,J\}$, where

2815: $p_i,q_i$ are as in (\ref{def:pq}).

2816: By construction, $\bar u(t)=u^n(\bar X(t))$,

2817: $t\ge0$ and item (c) holds. Item (b), namely that $\bar m=\beta[\bar u]$,

2818: also holds by construction.

2819: $\bar X$ and $\bar u$ are therefore $(\bar F_t)$-adapted,

2820: and since $\beta$ is a strategy, so is $\bar m$, and item (a) holds.

2821: Items (e) and (f) are trivial. Concerning (g),

2822: let $i^k\in\{1,\ldots,2J\}$ denote the index $i$ satisfying

2823: (\ref{eq:exi}) in case that $\Del a_i(p_i^k(t))>0$ holds,

2824: and let it denote $i+J$ in the case $\Del b_i^k(q_i^k(t))>0$.

2825: It suffices to show that for every $i\in\{1,\ldots,2J\}$,

2826: $$

2827:  \bar P(i^k=i|\bar F_{\tau_k})=

2828:  \lt\{\begin{array}{ll}

2829:  \bar E[\int_{\tau_k}^{\tau_{k+1}}p_i(s)ds|\bar F_{\tau_k}]/Z_k

2830:   & i\le J,\\[.1in]

2831:  \bar E[\int_{\tau_k}^{\tau_{k+1}}q_{i-J}(s)ds|

2832:  \bar F_{\tau_k}]/Z_k & i> J,

2833:  \end{array}\rt.

2834: $$

2835: where $Z_k$ is a normalization factor (not depending on $i$).

2836: For $k=0$ ($\tau_k=0$),

2837: this is a well known property of exponential clocks.

2838: For $k>0$, the same argument holds, merely because conditional on

2839: $\bar F_{\tau_k}$, the processes $\int_{\tau_k}^\cdot p_i(s)ds$,

2840: $\int_{\tau_k}^\cdot q_i(s)ds$ are independent, and moreover,

2841: $a_i(\cdot-\tau_k)-a_i(\tau_k),b_i(\cdot-\tau_k)-b_i(\tau_k)$

2842: are still independent Poisson processes (which is a statement

2843: on the lack of memory for exponential random variables).

2844: 

2845: The proof of the claim regarding the martingale associated with

2846: $\calL_0$ is similar (only simpler). This completes the proof of

2847: the first part of the lemma.

2848: 

2849: Clearly,

2850: $$

2851:    \max_i p_i(T_0)\vee q_i(T_0)\le nT_0b,

2852: $$

2853: where $T_0$ is as in Lemma \ref{lem:apriori}.

2854: Thus, if $N_n=\max\{k:\tau_k\le T_0\}$, then

2855: $$

2856:  N_n\le \sum_i a_i(nT_0b)+b_i(nT_0b),

2857: $$

2858: and (\ref{ineq:Nn}) follows.

2859: \qed

2860: 

2861: \noi{\bf Proof of Lemma \ref{lem:xbar2}:}

2862: The proof is completely

2863: analogous to that of Lemma \ref{lem:xbar}, and is therefore

2864: omitted.

2865: \qed

2866: 

2867: \noi{\bf Proof of Theorem \ref{th:b}:}

2868: By Theorems \ref{th:unique} and \ref{th:solve},

2869: $V^{b,-}=V^{b,+}$ for all $b\in[b^*,\infty)$. As a result,

2870: Theorem \ref{th:limit}, implies that

2871: $V^n\to V^{b,-}$ for all $b\in[b^*,\infty)$, as $n\to \infty$.

2872: In particular,

2873: $V^{b,-}$ does not depend on $b\in[b^*,\infty)$.

2874: It remains to show that for all $x$, $V^{b,-}(x)\to V^-(x)$

2875: and $V^{b,+}(x)\to V^+(x)$ as $b\to\infty$.

2876: 

2877: \noi\uu{\em Proof that $V^{b,-}\to V^-$.}

2878: It is immediate from the definitions that $V^-\le V^{b,-}$.

2879: 

2880: Let $\beta\in B$, and let $\s=\s(x,u,\beta)$ be the exit

2881: time of $\phi$ from $G$ where $\dot\phi=\pi(\phi,v(u,\beta[u]))$,

2882: $\phi(0)=x$. Let $\bar\beta$ be defined by

2883: $$

2884: \bar\beta[u](t)=\lt\{\begin{array}{ll}\min\{b,\beta[u](t)\}

2885:  & t\le\s, \\  \hat m & t>\s,

2886: \end{array}\rt.

2887: $$

2888: where $\hat m$ sets all $\bar\mu_j=0$ and $\bar\la_j=1_{j=i_{\phi(\s)}}$,

2889: and the minimum is componentwise. It is clear that

2890: $\bar\beta$ is a strategy. Let $\bar u$ be any extension

2891: of $u$ to $[0,\infty)$, and denote by $\bar\phi$ and $\bar\s$ the

2892: dynamics and exit time corresponding to $x,\bar\beta,\bar u$. 

2893: Recall that by (\ref{eq:lal}) $b$ is greater than all $\lambda_i$ and $\mu_i$.

2894: Thus

2895: \beaa

2896:  C(x,\bar u,\bar \beta[\bar u]) &=&

2897: \int_0^\s(c+\rho(u,\bar\beta[u])ds

2898:  +1_{\bar\s>\s}\int_\s^{\bar \s}(c+\rho(\bar u,\hat m))ds\\

2899:  &\le&

2900: C(x,u,\beta[u])+c_1(\bar\s-\s)^+.

2901: \eeaa

2902: Moreover, by the Lipschitz property of the Skorokhod map, and denoting

2903: $\beta[u]=(\bar\la_i,\bar\mu_i)$,

2904: $$

2905: (\bar\s-\s)^+

2906: \le c_2|\phi(\s)-\bar\phi(\s)|\le

2907: c_2\int_0^\s\sum_i[(\bar\la_i-b)^+ +(u_i\bar\mu_i-b)^+]ds

2908: $$

2909: Since it is enough to consider $\beta$ for which (for any $u$)

2910: $\la_i$ and $u_i\mu_i$ are uniformly integrable over $[0,\s]$,

2911: we have that $(\bar\s-\s)^+\le \del(b)$, where $\del(b)\to0$

2912: as $b\to\infty$. This shows that

2913: $\lim_{b\to\infty}V^{b,-}(x)\le V^-(x)$.

2914: 

2915: \noi\uu{\em Proof that $V^{b,+}\to V^+$.}

2916: It is immediate that $V^+\le V^{b,+}$.

2917: 

2918: To show that $V^+(x)\ge \lim_{b\to\infty}V^{b,+}(x)$

2919: it is enough to show that for $b\ge b^*$ and $a$ small,

2920: $V^+(x)\ge V^{b,+}_{-a}(x)$.

2921: For any $m\in\bar M$ let $m^b$ denote the pointwise and componentwise

2922: truncation of $m$ at level $b$. For any $\al\in A$, let $\al^b\in A$

2923: be defined by $\al^b[m]=\al[m^b]$.

2924: We will write $m\in\bar M(\al,a)$ if $m,\al,a$ satisfy

2925: $C_a(x,\al[m],m)\le V^+_a(x)+1$.

2926: In the expression for $V^+_a(x)$,

2927: $$

2928:  \sup_\al\inf_m C_a(x,\al[m],m),

2929: $$

2930: it is enough to consider $\al\in A$ and $m\in\bar M(\al,a)$

2931: (including for $a=0$).

2932: For such $\al,m$, the functions

2933: $\bar\la_i,u_i\bar\mu_i$ are uniformly integrable over $[0,T]$.

2934: Let $\al\in A$ and $m\in\bar M(\al^b,0)$. Consider

2935: a truncation of $\bar\la_i$ and $\bar\mu_i$ at $b$.

2936: Denote by $\phi$ [resp., $\phi^b$] the dynamics that correspond to

2937: $(x,\al^b,m)$, [resp., $(x,\al^b,m^b)$].

2938: Then the effect of the truncation

2939: on $\phi$ is such that for all $a>0$ there is $b$ such

2940: that $\sup_{[0,T]}|\phi-\phi^b|\le a$ (by uniform integrability).

2941: In particular, $|\phi^b(\s)-\phi(\s)|\le a$.

2942: Hence, using

2943: the monotonicity of the running cost for large values of

2944: the rates, and that $\al^b[m^b]=\al^b[m]$,

2945: $$

2946:  C_{-a}(x,\al^b[m^b],m^b)\le C(x,\al^b[m],m).

2947: $$

2948: We thus have

2949: $$

2950: C_{-a}(x,\al[m^b],m^b)\le C(x,\al^b[m],m).

2951: $$

2952: Since $m\in\bar M(\al^b,0)$ implies that $m^b\in\bar M(\al^b,-a)$,

2953: \beaa

2954:  \inf_{m\in\bar M^b}C_{-a}(x,\al[m],m) &\le&

2955: \inf_{m:m^b\in\bar M(\al^b,-a)}

2956:  C_{-a}(x,\al[m^b],m^b)

2957:  \\ &\le& \inf_{m\in\bar M(\al^b,0)}C(x,\al^b[m],m)

2958:  \\ &=&

2959: \inf_{m\in\bar M}C(x,\al^b[m],m).

2960: \eeaa

2961: Hence

2962: $$

2963: \sup_{\al\in A}\inf_{m\in\bar M^b}C_{-a}(x,\al[m],m)\le

2964: \sup_{\al\in A}\inf_{m\in\bar M}C(x,\al[m],m).

2965: $$

2966: Taking $a\to0$ by letting $b\to\infty$, we have from

2967: Lemma \ref{lem:cont} that $\lim_bV^{b,+}(x)\le V^+(x)$.

2968: \qed

2969: 

2970: 

2971: \vspace{\baselineskip}

2972: \noindent

2973: {\it 1991 Mathematics Subject Classification.}

2974: Primary 60F10, 60K25;

2975: Secondary 93E20, 60F17.

2976: 

2977: 

2978: \bibliographystyle{plain}

2979: \begin{thebibliography}{99}

2980: 

2981: \bibitem{atadup2} R.~Atar and P.~Dupuis.

2982: A differential game with constrained dynamics.

2983: To appear,

2984: {\it Nonlinear Analysis: theory, methods and applications.}

2985: 

2986: \bibitem{atadupshw}

2987: R. Atar, P. Dupuis and A. Shwartz,

2988: ``Explicit solutions to a network control problem in

2989: the large deviation regime'', 1--18, preprint

2990: 

2991: \bibitem{BDM} J.S.~Baras, A.J.~Dorsey and A.M.~Makowski,

2992: ``Two competing queues with geometric service requirements

2993: and linear costs: the  $ \mu$-c rule is often optimal,"

2994: {\em Adv.\ Appl.\ Prob.\ \bf 17} pp.~186--209, 1985.

2995: 

2996: \bibitem{balday1}

2997: J. Ball, M. Day, T. Yu and P. Kachroo,

2998: \newblock Robust L2-gain control for nonlinear systems with projection 

2999: dynamics and input constraints: an example from traffic control.

3000: \newblock {\em Automatica}, 35:429--444, 1999.

3001: 

3002: \bibitem{balday2}

3003: J. Ball, M. Day, and P. Kachroo,

3004: \newblock  Robust feedback control for a single server queueing system.

3005: \newblock {\em Mathematics of Control, Signals, and Systems}, 12:307--345, 1999.

3006: 

3007: \bibitem{barcap}

3008: M. Bardi and I. Capuzzo-Dolcetta.

3009: {\it Optimal control and viscosity solutions

3010: of Hamilton-Jacobi-Bellman equations.}

3011: Birkhauser, Boston. 1997

3012: 

3013: \bibitem{buddup}

3014: A.~Budhiraja and P.~Dupuis.

3015: \newblock Simple necessary and sufficient conditions for the stability of

3016:   constrained processes.

3017: \newblock {\em SIAM J. Applied Math.}, 59:1686--1700, 1999.

3018: 

3019: \bibitem{capLion} I.~Capuzzo-Dolcetta and P.-L.~Lions,

3020: Hamilton-Jacobi equations with state constraints,

3021: {\em Trans.\ AMS \bf 318} pp.~643--683, 1990.

3022: 

3023: \bibitem{delmey} C.~Dellacherie and P.-A.~Meyer. {\em Probabilit\'es

3024: et potentiel/ Th\'eorie des martingales}, Hermann, Paris, 1980.

3025: 

3026: \bibitem{dupell}

3027: P. Dupuis and R.~S. Ellis.

3028: \newblock {\it A Weak Convergence Approach to the Theory of Large

3029: Deviations}.

3030: \newblock John Wiley \& Sons, New York, 1997.

3031: 

3032: \bibitem{dupkus}

3033: P. Dupuis and H. Kushner.

3034: Minimizing escape probabilities: a large deviations approach. 

3035: {\it SIAM J. Control Optim. 27} (1989), no. 2, 432--445

3036: 

3037: \bibitem{dupish1}

3038: P.~Dupuis and H.~Ishii,

3039: On Lipschitz continuity of the solution mapping to the Skorokhod

3040: Problem, with applications.

3041: {\em Stochastics \bf 35}, pp.~31--62, 1991.

3042: 

3043: \bibitem{dupish2}

3044: P.~Dupuis and H.~Ishii,

3045: On oblique derivative problems for fully

3046: nonlinear second-order elliptic PDE's

3047: on domains with corners.

3048: {\it Hokkiado U. Math. J.} {\bf 20}, pp. 135--164, 1991.

3049: 

3050: 

3051: \bibitem{dupjampet}

3052: P.~Dupuis, M.R. James and I.R.~Petersen.

3053: Robust properties of risk--sensitive control.

3054: {\em Math. of Control, Signals and Systems

3055: \bf 13} pp.~318--332, 2000.

3056: 

3057: \bibitem{dupmce}

3058: P. Dupuis and W. M. McEneaney.

3059: Risk-sensitive and robust escape criteria.

3060: {\it SIAM J. Control Optim. 35} (1997), no. 6, 2021--2049.  

3061: 

3062: \bibitem{dupnag}

3063: P. Dupuis and A. Nagurney.

3064: Dynamical systems and variational inequalities.

3065: {\it Ann. Oper. Res.} 44 (1993), no. 1-4, 9--42. 

3066: 

3067: \bibitem{dupram23}

3068: P.~Dupuis and K.~Ramanan,

3069: Convex duality and the Skorokhod Problem. I, II.

3070: {\em Probability Theory and Related Fields \bf 2},

3071: pp~153--195, 197--236, 1999.

3072: 

3073: \bibitem{ellkal} R.J.~Elliott and N.J.~Kalton.

3074: {\em The existence of value in differential games,}

3075: Memoirs of the American Mathematical

3076: Society, No. 126. American Mathematical Society,

3077: Providence, R.I., iv+67 pp., 1972.

3078: 

3079: \bibitem{flesou}

3080: W. H. Fleming and P. E. Souganidis.

3081: PDE-viscosity solution approach to some problems of large deviations.

3082: {\it Ann. Scuola Norm. Sup. Pisa Cl. Sci.} (4) 13 (1986),

3083: no. 2, 171--192.

3084: 

3085: \bibitem{harrei}

3086: J. M. Harrison and M. I. Reiman.

3087: Reflected Brownian motion on an orthant.

3088: {\it Ann. Probab.} 9 (1981), no. 2, 302--308.

3089: 

3090: \bibitem{Klimov} G.P.~Klimov.

3091: Time sharing service systems I.

3092: {\em Theory Prob.\ Appl.} {\bf 19} pp.~532--551, 1974.

3093: 

3094: \bibitem{reiwil} M.I.~Reiman and R.J.~Williams.

3095: A boundary property of semimartingale reflecting Brownian motions.

3096: {\it Probab. Theory Related Fields} 77 (1988), no. 1, 87--97.

3097: 

3098: \bibitem{roc} R.T.~Rockafellar, {\em Convex analysis.} Princeton, NJ,

3099: 1970.

3100: 

3101: \bibitem{shwe} A.~Shwartz and A.~Weiss,

3102: {\em Large deviations for performance analysis.} Chapman and Hall, 1995.

3103: 

3104: \bibitem{son} H.M.~Soner. Optimal control with state space constraints I.

3105: {\em SIAM J.\ Control Opt.} {\bf 24} pp.~552--561, 1986.

3106: 

3107: \bibitem{W} J.~Walrand,

3108: ``A note on `optimal control of a queueing system with two heterogeneous

3109: servers',''

3110: {\em System Control Lett.\ \bf 4} pp.~131--134, 1984.

3111: 

3112: \end{thebibliography}

3113: 

3114: \end{document}

3115: 

3116: 

3117: 

3118: