1: \documentclass[11pt]{article}

2: 

3: \usepackage{amssymb,amsmath,latexsym}

4: 

5: %\usepackage{showkeys}

6: 

7: \input psfig.sty

8: \psfull

9: 

10: \topmargin      0.25truein

11: \oddsidemargin  0.0truein

12: \evensidemargin 0.0truein

13: \textheight     8.5truein

14: \textwidth      6.5truein

15: 

16: \footskip       0.6truein

17: \headheight     0.0truein

18: \headsep        0.0truein

19: \parskip 0.3cm

20: 

21: \begin{document}

22: 

23: \newtheorem{lemma}{Lemma}

24: \newtheorem{theorem}{Theorem}

25: \newtheorem{corollary}{Corollary}

26: \newtheorem{definition}{Definition}

27: \newtheorem{example}{Example}

28: \newtheorem{proposition}{Proposition}

29: \newtheorem{condition}{Condition}

30: 

31: \newcommand{\bea}{\begin{eqnarray}}

32: \newcommand{\eea}{\end{eqnarray}}

33: \newcommand{\beaa}{\begin{eqnarray*}}

34: \newcommand{\eeaa}{\end{eqnarray*}}

35: \newcommand{\ben}{\begin{enumerate}}

36: \newcommand{\een}{\end{enumerate}}

37: \newcommand{\bi}{\begin{itemize}}

38: \newcommand{\ei}{\end{itemize}}

39: 

40: \newcommand{\lip}{\langle}

41: \newcommand{\lan}{\langle}

42: \newcommand{\rip}{\rangle}

43: \newcommand{\ran}{\rangle}

44: \newcommand{\uu}{\underline}

45: \newcommand{\oo}{\overline}

46: \newcommand{\til}{\tilde}

47: 

48: \newcommand{\La}{\Lambda}

49: \newcommand{\la}{\lambda}

50: \newcommand{\eps}{\epsilon}

51: \newcommand{\vph}{\varphi}

52: \newcommand{\al}{\alpha}

53: \newcommand{\bet}{\beta}

54: \newcommand{\gam}{\gamma}

55: \newcommand{\Gam}{\Gamma}

56: \newcommand{\kap}{\kappa}

57: \newcommand{\Del}{\Delta}

58: \newcommand{\Th}{\Theta}

59: \newcommand{\s}{\sigma}

60: \newcommand{\sig}{\sigma}

61: \newcommand{\Sig}{\Sigma}

62: \newcommand{\del}{\delta}

63: \newcommand{\om}{\omega}

64: \newcommand{\Om}{\Omega}

65: 

66: \newcommand{\N}{{\mathbb N}}

67: \newcommand{\R}{{\mathbb R}}

68: \newcommand{\Z}{{\mathbb Z}}

69: \newcommand{\bbeta}{{\text{\boldmath$\beta$}}}

70: \newcommand{\bdel}{\delta}

71: \newcommand{\bgamma}{\gamma}

72: \newcommand{\bnu}{\nu}

73: \newcommand{\bDel}{\Delta}

74: \newcommand{\y}{y}

75: \newcommand{\x}{x}

76: \newcommand{\X}{X}

77: \newcommand{\bu}{u}

78: \newcommand{\e}{e}

79: \newcommand{\m}{m}

80: \newcommand{\M}{M}

81: \newcommand{\p}{p}

82: \newcommand{\q}{q}

83: \newcommand{\bv}{v}

84: \newcommand{\z}{z}

85: 

86: \newcommand{\calA}{{\cal A}}

87: \newcommand{\calB}{{\cal B}}

88: \newcommand{\calC}{{\cal C}}

89: \newcommand{\calD}{{\cal D}}

90: \newcommand{\calF}{{\cal F}}

91: \newcommand{\calG}{{\cal G}}

92: \newcommand{\calH}{{\cal H}}

93: \newcommand{\calJ}{{\cal J}}

94: \newcommand{\calL}{{\cal L}}

95: \newcommand{\calM}{{\cal M}}

96: \newcommand{\calP}{{\cal P}}

97: \newcommand{\calS}{{\cal S}}

98: \newcommand{\calT}{{\cal T}}

99: \newcommand{\calU}{{\cal U}}

100: \newcommand{\calV}{{\cal V}}

101: \newcommand{\calX}{{\cal X}}

102: \newcommand{\calY}{{\cal Y}}

103: 

104: \newcommand{\proof}{\noindent {\bf Proof:\ }}

105: \newcommand{\proofOf}[1]{\noindent {\bf Proof of #1:\ }}

106: \newcommand{\remark}{\noindent {\bf Remark:\ }}

107: \newcommand{\remarks}{\noindent {\bf Remarks:\ }}

108: \newcommand{\note}{\noindent {\bf Note:\ }}

109: \newcommand{\esssup}{{\rm ess}\sup}

110: \newcommand{\essinf}{{\rm ess}\inf}

111: \newcommand{\pl}{\partial}

112: \newcommand{\noi}{\noindent}

113: \newcommand{\goto}{\to}

114: \newcommand{\ink}{\rule{.5\baselineskip}{.55\baselineskip}}

115: \newcommand{\qed}{\rule{.5\baselineskip}{.55\baselineskip}}

116: 

117: \def\ve{\varepsilon}

118: \def\vr{\varrho}

119: 

120: \newcommand{\diag}{{\rm diag}}

121: \newcommand{\trace}{{\rm trace}}

122: \newcommand{\tr}{{\rm tr}}

123: \newcommand{\w}{\wedge}

124: \newcommand{\dint}{\int\!\!\!\int}

125: \newcommand{\lt}{\left}

126: \newcommand{\rt}{\right}

127: \newcommand{\dist}{{\rm dist}}

128: 

129: \newcommand{\policy}{{u}}

130: \def\OBdry{\partial_o}

131: \def\CBdry{\partial_c}

132: \newcommand{\Sfrac}[2]{{{#1}\slash {#2}}}

133: 

134: \newcommand{\mean}[1]{\langle#1\rangle}

135: 

136: \date{August 14, 2003}

137: \title{

138: Explicit solution for a network control problem\\

139: in the large deviation regime\thanks{This

140: research  was supported in part by the United States--Israel Binational

141: Science Foundation (BSF 1999179)}}

142: \author{Rami Atar\footnote{Department of Electrical Engineering,

143: Technion -- Israel Institute of Technology,

144: Haifa 32000, Israel.

145: {\tt atar@ee.technion.ac.il}.

146: Research of this author also supported

147: in part by the fund for promotion of research at the Technion.},

148: Paul Dupuis\footnote{Lefschetz Center for Dynamical Systems,

149: Division of Applied Mathematics,

150: Brown University,

151: Providence,  R.I.\  02912.

152: {\tt dupuis@dam.brown.edu}.

153: Research of this author also supported in part by

154: the National

155: Science Foundation (NSF-DMS-0072004, NSF-ECS-9979250) and the Army Research Office 

156: (DAAD19-02-1-0425).}

157: \ and Adam Shwartz\footnote{Department of Electrical Engineering,

158: Technion -- Israel Institute of Technology,

159: Haifa 32000, Israel.

160: {\tt adam@ee.technion.ac.il}.

161: Research of this author also supported

162: in part by INTAS grant 265, and in part by the fund for promotion

163: of research at the Technion.}\\[.2in]

164:  }

165: 

166: \maketitle

167: 

168: \begin{abstract}

169: We consider optimal control of a stochastic network, where service is controlled to prevent buffer overflow.

170: We use a risk-sensitive escape time criterion,

171: which in comparison to the ordinary escape time criteria heavily penalizes exits which occur on short time intervals.

172: A limit as the buffer sizes tend to infinity is considered.  In~\cite{ads} we showed that, for a large class of networks,

173: the limit of the normalized cost agrees with the value function of a differential game.  In this game, one player controls the service discipline 

174: (who to serve and whether to serve),

175: and the other player chooses arrival and service rates in the network.

176: The game's value is characterized in \cite{ads} as the unique solution

177: to a Hamilton-Jacobi-Bellman Partial Differential Equation (PDE). 

178: In the current paper we apply this general theory to the important case of a network of queues in tandem.

179: Our main results are: (i) the construction of an explicit solution to the corresponding PDE,  and (ii) drawing out the implications for optimal risk-sensitive and robust regulation of the network.

180: In particular, the following general principle can be extracted.

181: To avoid buffer overflow there is a natural competition between two tendencies.

182: One may choose to serve a particular queue, since that will help prevent its own buffer from overflowing, or one may prefer to stop service, with the goal of preventing overflow of buffers further down the line.

183: The solution to the PDE indicates the optimal choice between these two, specifying

184: the parts of the state space where each queue must be

185: served (so as not to lose optimality), and where it can idle.

186: Referring to those queues which must be served

187: as bottlenecks, one can use the solution to the PDE to explicitly

188: calculate the bottleneck queues as a function of

189: the system's state, in terms of a simple set of equations.

190: \end{abstract}

191: 

192: %\vspace{\baselineskip}

193: \noindent

194: {\it 1991 Mathematics Subject Classification.}

195: Primary 60F10, 60K25;

196: Secondary 49N70, 93E20.

197: 

198: 

199: \section{Introduction}

200: 

201: In a previous work \cite{ads}

202: we considered a stochastic control problem for a Markovian queueing network

203: with deterministic routing, where the service stations may provide service to

204: one or more queues, with each queue being limited by a finite buffer. The control refers to the service

205: discipline at the service stations, and the cost involves the time till one of the

206: queues first reaches its buffer limit. Such a problem can be regarded as the control of

207: a Markov process up to the time it exits a domain, the domain being the rectangle

208: associated with the buffer sizes. The cost is chosen so as to obtain a risk-sensitive

209: (or rare-event) control problem: one considers $E_xe^{-c\s}$ as a criterion

210: to be minimized, where $c>0$ is fixed and $\s$ denotes the exit time (the time when any

211: one of the buffers first overflows).

212: Such a criterion penalizes short exit times more heavily

213: than ordinary escape time criteria (such as $E_x\s$, a criterion to be maximized).

214: While the main result of \cite{ads} is the characterization of a limiting problem as the buffer sizes tend to infinity,

215: the current work focuses on finding explicit solutions to this limit problem and on the

216: interpretation thereof.

217: 

218: There are at least two motivations for the use of risk-sensitive criteria when designing 

219: policies for the control of a network.

220: The first is that in many communication networks performance is measured in terms of

221: the occurrence of rare but critical events.

222: Buffer overflow is a principal example of such an event.

223: The second motivation follows from the connection between risk-sensitive

224: controls and robust controls. Indeed, as discussed in \cite{dupjampet},

225: the optimization of a single fixed stochastic network with respect to a 

226: risk-sensitive cost criteria automatically produces controls with specific and 

227: predictable robust properties. In particular,

228: these controls give good performance for a family of perturbed network models 

229: (where the perturbation is around the design model and the size of the 

230: perturbation is measured by relative entropy), and with respect to a 

231: corresponding ordinary (i.e., not risk-sensitive) cost.

232: 

233: In many problems, one considers the limit of the risk-sensitive problem as a scaling parameter 

234: of the system converges, in the hope that the limit model is more tractable.

235: This is the path followed in \cite{ads}, in the asymptotic regime

236: where time is accelerated and buffer lengths are enlarged by a factor $n$.

237: The limit of the normalized cost was characterized both as the value function of

238: a differential game, and as the solution to a corresponding nonlinear PDE.

239: Both interpretations are important and useful. 

240: It is the interpretation as the value function of a differential game that is key in quantifying the robust aspects of the resulting control policy.

241: However, because the PDE gives a necessary and sufficient characterization of the (a priori unknown) value function,

242: it can provide qualitative information on the structure of the value function and the optimal controls.

243: In particularly favorable circumstances one can go even further,

244: and use the PDE for exact calculation and control policy synthesis.

245: Indeed,

246: if one can by any means guess a proper parametric form of the value function,

247: then it is sometimes possible to use the PDE to verify that this is the correct form,

248: and identify the unknown parameters in the representation for the value function.

249: The instances in the control theory literature where this has been carried out are few and far between,

250: especially when the state space of the system has dimension greater than two.

251: However, 

252: important information is obtained from these instances on the structure of optimal controls.

253: 

254: 

255: 

256: In this paper the focus is on applying the PDE obtained in \cite{ads} in the manner just described.

257: In particular, we treat in detail the general case of a tandem queueing network, and construct an explicit solution to the PDE.  

258: In a system of queues in tandem, each server offers service to exactly one queue, and therefore

259: the service control refers simply to whether each station should provide service or remain idle. 

260: Naturally, it is important to provide service to a queue in order to keep

261: it from reaching the buffer limit. On the other hand, if the next buffer in line is nearly full,

262: it might be necessary to idle the first queue in order to keep the second from overflowing.

263: Besides factoring in how close all buffers are to their respective limits,

264: one must consider the mean service rates and the likelihood of significant deviations from those mean service rates.  Is it likely that the next queue down will stall and simply stop serving for a while?

265: It is also possible that one will have to look even further ahead,

266: and consider buffers further downstream.

267: 

268: Although the optimal control problem for the Markovian queueing system is fully described

269: by a dynamic programming equation \cite{ads}, such equations are typically solved numerically.

270: It is hard to extract any global structural information from the exact equation,

271: and even a numerical solution may not be feasible when buffers are large or the dimension of the state space is moderate. 

272: As discussed previously, the solution to the PDE for the limiting problem turns out to simplify things

273: significantly. Roughly speaking, the PDE indicates the following structure of

274: the asymptotic optimally controlled

275: network. In the interior of the domain

276: (i.e., when all buffers are away from their maximum capacity), and depending on the state of the system,

277: service must be provided at

278: certain service stations, while other stations may idle without causing loss of optimality.

279: In that sense, the limit problem sharpens the control policy by emphasizing

280: the importance of serving those `bottleneck' buffers

281: (it is crucial to serve the bottleneck buffers, and completely

282: unimportant to serve the others).

283: The identification of the bottlenecks is nontrivial, and indeed queues with the smallest service rate are not necessarily  bottlenecks.

284: Instead, as hinted above, identification of the bottlenecks must include consideration of at least the following: (i) the relative closeness of all buffers to their maximum value,

285: (ii) relative mean service rates,

286: and (iii) relative uncertainties in the service rates.

287: We identify a system of algebraic equations whose solution identifies all bottlenecks.

288: 

289: 

290: There is relatively little work on risk-sensitive and robust control of networks.

291: Ball et.\ al.\ have considered a robust formulation for network problems

292: arising in vehicular traffic \cite{balday2}, where the cost structure is qualitatively

293: different. Dupuis studies a robust control problem for networks in a deterministic setting

294: and obtains explicit solutions for the value function \cite{dup}.

295: The cost there is, in a sense, antipodal to the one considered in the current work,

296: namely the time till the system becomes empty (a criterion to be minimized).

297: For other recent work on queueing control in a large deviation regime

298: see Stolyar and Ramanan \cite{storam}, where a

299: single server non-Markovian system (with quite general stationary increments input flows)

300: is studied, and a particularly simple scheduling control policy is shown

301: to be asymptotically optimal (see also an extension of the work in

302: Stolyar~\cite{sto}).

303: 

304: The paper is organized as follows.

305: Section 2 introduces the model and the PDE and states the main result.

306: Section 3 contains discussion and interpretation.

307: In Section 4 we prove the main result.

308: 

309: Notation:

310: The symbol $\vee $ stands for

311: maximum, while $\w $ stands for minimum. Denote scalar product between two

312: vectors as $ \x \cdot \y $.

313: For integers $i\le j$, let $[i,j]\doteq\{1,\ldots,j\}$.

314: 

315: 

316: \section{Model and preliminaries}

317: 

318: \noi\uu{\bf The queueing network control problem.}

319: %

320: We consider the following tandem network.

321: There are $J$ queues and $J$ servers. Customers present at queue $i$

322: at a certain time are said to be of class $i$ at that time.

323: Customers arrive to queue $1$ according to a

324: Poisson process of rate $\la\ge0$. Service at queue $i$ is provided by

325: server $i$ at exponential time

326: with parameter $ \mu_i>0$, mutually independent and independent of the

327: arrivals.

328: After a customer is served by server $i$, it moves to queue

329: $r(i)$, where $r(i) = i+1 $, $i=1,\ldots,J-1$, $ r(J) = 0$,

330: and $i=0$ is used to denote the ``outside''.

331: The state of the network is the vector of queue lengths, denoted

332: by $\X$.

333: Let $\{ \e_i;i=1,\ldots,J\} $ denote the unit coordinate vectors,

334: let $\e_0 = 0 $ and denote

335: \begin{equation}\label{egam}

336: \bgamma_i=\e_i-\e_{r(i)}.

337: \end{equation}

338: Note that $\bgamma_i=\e_i-\e_{i+1}$ for $i=1,\ldots,J-1$, $\bgamma_J

339: =\e_J$, and that

340: following service to queue $i$ the state changes by $-\bgamma_i$.

341: The control is specified by the vector $ u =(u_1,...,u_J)$,

342: where $u_i=1$ if customers in queue $i$ are given service and $u_i=0$

343: otherwise.

344: We next consider the scaled process $\X^n$ under the scaling

345: which accelerates time by a factor of $n$ and down-scales space by

346: the same factor. We are interested in a

347: risk-sensitive cost functional that is associated with

348: exit from a bounded set.

349: Let $G$ be the rectangle defined through

350: \begin{equation}\label{def:G}

351:  G=\{(x_1,\ldots,x_J): 0\le x_1<z_1;\ 0\le x_i\le z_i, \,

352: i = 2 , \ldots , J \},

353: \end{equation}

354: for some $z_i>0$, $i=1,\ldots,J$.

355: Note that $G$ contains parts of, but not all of its boundary.

356: Let

357: \[

358: \sigma^n \doteq \inf \{t:\X^n(t) \not \in G \}.

359: \]

360: %

361: The control problem is to minimize the cost

362: $E_\x e^{-nc\sigma^n}$, where $E_\x$ denotes expectation starting

363: from $\x$, and $c>0$ is a constant.

364: With this cost structure ``risk-sensitivity'' means that atypically 

365: short exit times are weighted heavily by the cost.

366: As a result, even if exit within short time

367: occurs with small probability, it may have a noticeable effect on the cost,

368: and thus a ``good'' control will attempt to avoid such events

369: as much as possible.

370: 

371: %

372: Although this paper do not directly treat the

373: stochastic control problem but the corresponding PDE, we give here the precise

374: formulation of the former, for completeness.

375: For each $n$, the state space is $G^n \doteq n^{-1}\Z_+^J \cap G$. 

376: The control (or action) space is given by 

377: $$

378:  U \doteq \left\{(u_i), i=1,\ldots,J: 0 \le u_i\le1, \

379:  i=1,\ldots,J\right\}.

380: $$

381: One considers state processes $\X^n$ that are similar

382: to those defined for the original queueing network,

383: except that time is accelerated (equivalently, arrival and service rates

384: are multiplied) by a scaling parameter $n$, and space is scaled

385: down by the same factor. More precisely, for $u\in U$ and $f:\Z_+^J\to\R$,

386: let

387: $$

388: \calL^uf(\x)=\la [f(\x+\e_1)-f(\x)]

389: +\sum_{i=1}^Ju_i\mu_i1_{\{\x-\bgamma_i\in\Z_+^J\}}[f(\x-\bgamma_i)-f(\x)],

390: \quad \x\in\Z_+^J.

391: $$

392: For $n\in\N$ let

393: $$

394: \calL^{n,u}f(\x)=n\calL^ug(n\x),

395: $$

396: where $f:n^{-1}\Z_+^J\to\R$ and $g(\cdot)=f(n^{-1}\cdot)$.

397: A {\em controlled Markov process} starting from $\x\in G^n$

398: will consist of a complete filtered

399: probability space $(\Om,\calF,(\calF_t),P_x^{n,u})$,

400: a state process $\X^n$ taking values in  $G^n$ and

401: a control process $u$ taking values in  $U$,

402: such that $\X^n$ is adapted to $\calF_t$,

403: $u$ is measurable and adapted to $\calF_t$,

404: $P_\x^{u,n}(\X^n(0)=\x)=1$,

405: and for every function $f:G^n\to\R$

406: \[

407: f(\X^n(t))-\int_0^t\calL^{n,u(s)}f(\X^n(s))ds

408: \]

409: is an $\calF_t$-martingale.

410: $E_\x^{n,u}$ denotes expectation with respect to $P_\x^{n,u}$.

411: For a parameter $c>0$,

412: the value function for the stochastic control problem is 

413: defined by

414: \begin{equation}\label{eq:control}

415:   V^n(\x) \doteq -\inf n^{-1}\log E_\x^{u,n}e^{-nc\sig_n},

416:   \quad \x\in G^n,

417: \end{equation}

418: where the infimum is over all controlled Markov processes.

419: 

420: 

421: 

422: 

423: \noi\uu{\bf The domain and its boundary.}

424: It is possible for the controller to prevent any but the first queue

425: from exceeding $ z_i $, simply by turning off service to the preceding

426: queue. However, the controller cannot prevent overflow of the first queue.

427: Although it is in principle possible that the dynamics

428: could exit through the portion of the boundary defined by queues

429: $ 2 , \ldots , J$, it is always optimal for

430: the controller to not allow this.

431: Consider the simple two-class network illustrated in Figure \ref{fig1}.

432: The controller can prevent exit through the dashed portion of the 

433: boundary simply by stopping service at the first queue. 

434: As a consequence, there are in general three different types

435: of boundary--the constraining boundary due to non-negativity constraints on queue length,

436: the part of the boundary where exit can be blocked, and the remainder.

437: These three types of boundary behavior result

438: in the PDE in three types of boundary conditions.

439: The three portions of the boundary are explicitly given as

440: $$

441: \pl_c G=\{(x_1,\ldots,x_J): 0\le x_1<z_1,\ \mbox{ and }\ x_i=z_i\ \mbox{for

442: some}\,i> 1\}

443: $$

444: $$

445: \pl_o G=\{(x_1,\ldots,x_J): x_1=z_1, \ \mbox{ and}\ 0\le x_i\le z_i\ \mbox{for

446: all}\ i>1\}

447: $$

448: $$

449: \pl_+G= \{(x_1,\ldots,x_J):

450: \mbox{$x_i<z_i$ for all $i$, and $x_i=0$ for some $i$}\}.

451: $$

452: Note that $\pl_cG$, $\pl_oG$ and $\pl_+G$

453: partition the boundary $\pl G$ of $G$.

454: Also, $\pl_cG\subset G$ while $\pl_oG\cap G=\emptyset$.

455: As usual, we will denote $G^o=G\setminus\pl G$

456: and $\bar G=G\cup\pl G$.

457: 

458: 

459: \begin{figure}

460: \centerline{

461: \begin{tabular}{cc}

462: \psfig{file=fig_net_1.eps}\qquad\qquad\qquad

463: \psfig{file=fig_ex_1.eps}

464: \end{tabular}

465: }

466: \caption{

467: A simple queueing network, a rectangular domain and three

468: types of boundary.

469: Full line: $~$ \hfill $\pl_+G$, dashed line: $\pl_cG$,

470: and dotted line: $\pl_oG$

471: \hfill $~$ }

472: \label{fig1}

473: \end{figure}

474: 

475: 

476: 

477: \noi\uu{\bf The Hamiltonian, PDE and viscosity solutions.}

478: %

479: It is a standard fact that the value functions $V^n$

480: of the stochastic control problem considered above can be characterized

481: by a dynamic programming equation. The results of \cite{ads}

482: show that also the limit of $V^n$ as $n\to\infty$

483: can be characterized by a certain equation, namely a Hamilton-Jacobi-Bellman

484: PDE, that can in fact be regarded as the dynamic programming equation

485: for a certain deterministic game problem. For details on this PDE

486: we need some notation (the reader is referred to \cite{ads} for details

487: on the game).

488: Let $l:\R\to\R_+\cup\{+\infty\}$

489: be defined as

490: $$

491:  l(x)\doteq \lt\{\begin{array}{ll}x\log x-x+1 & x\ge0,  \\ +\infty & x<0,

492:      \end{array}\rt.

493: $$

494: where $0\log0\doteq0$.

495: Define

496: \begin{equation}

497: \label{em}

498:  \M =\{m=(\bar\la,\bar\mu_1,\ldots,\bar\mu_J):

499: \bar\la \geq 0 ,\ \bar \mu_i \geq 0 \}.

500: \end{equation}

501: For $ u\in U$ and $ m\in \M$ define

502: \begin{align*}

503: v ( u, m)    & \doteq \bar \lambda \e_1 - \sum_{i=1}^J

504:                u_i \bar\mu_i \bgamma_i, \\

505: \rho( u, m) & \doteq \lambda l\left( \frac{\bar{\lambda}}{

506: \lambda}\right) +\sum_{i=1}^{J}u_i\mu _il\left( \frac{\bar{\mu}_i}

507: {\mu_i}\right),

508: \end{align*}

509: where $\bgamma_i$ are as in \eqref{egam}.

510: Let

511: \begin{align}\label{def:H1}

512: H(p, u, m) &= c+ p\cdot v ( u, m)+\rho( u, m)\\ \notag

513: &=

514: c+

515: \lt[\bar\la p_1+\la l\lt(\frac{\bar\la}{\la}\rt)\rt]

516: +\sum_{i=1}^J u_i

517: \lt[-\bar\mu_i\bgamma_i\cdot \p+\mu_i l\lt(\frac{\bar\mu_i}{\mu_i}\rt)\rt]

518: \end{align}

519: and let the Hamiltonian be defined as

520: \begin{equation}\label{def:H}

521:  H(p)=\sup_{ u\in U}\inf_{ m\in\M}H(p,u,m)=\inf_{m\in \M}\sup_{u\in U}

522: H(p, u, m),

523: \end{equation}

524: where the last identity, expressing the {\it Isaacs Condition} is proved in \cite{ads}.

525: %

526: The following simplification in the structure of 

527: the Hamiltonian will be useful.

528: Using convexity and the fact that the slope of $l$ at $0^+$ is

529: $-\infty$, the minimum over $ m$ is attained

530: at $\bar\la=\la e^{-p_1}$, $\bar\mu_i=\mu_i e^{\bgamma_i\cdot \p}$.

531: A straightforward calculation then shows that

532: \begin{align}\notag

533: H(\p, u)\doteq \inf_m H(\p, u, m) &=

534: c+[(\la-\bar\la)+u_i(\mu_i-\bar\mu_i)]\\

535: &=

536: c+\la(1-e^{-p_1})

537: +\sum_{i=1}^Ju_i\mu_i(1-e^{\bgamma_i\cdot \p}).

538: \label{eq:116}

539: \end{align}

540: Define $ I (\x) = \{ i : x_i = 0 \} $.

541: The PDE of interest is the following.

542: \begin{equation}\label{eq:pde}

543: \lt\{

544: \begin{array}{ll}

545:  H(DV(\x))=0, & \x\in G^o,\\ \\

546:  DV(\x)\cdot \bgamma_i=0, & i\in I(\x),\ \x\in\pl_+G ,\\ \\

547:  V(\x)=0, & \x\in\OBdry G.

548: \end{array}\rt.

549: \end{equation}

550: Since typically such equations do not possess classical solutions,

551: the framework of viscosity solutions is useful (see \cite{barcap}).

552: This framework allows

553: for functions that are merely continuous to be regarded as solutions.

554: For $\x\in G$,

555: the set of superdifferentials $D^+V(\x)$ and the set of subdifferentials

556: $D^-V(\x)$ are defined as

557: \begin{equation}\label{def:Dplus}

558: D^+V(\x)=\lt\{\p:\limsup_{\y\to \x}\frac{V(\y)-V(\x)-\p\cdot(\y-\x)}{|\y-\x|}\le0\rt\},

559: \end{equation}

560: \begin{equation}\label{def:Dminus}

561: D^-V(\x)=\lt\{\p:\liminf_{\y\to \x}\frac{V(\y)-V(\x)-\p\cdot(\y-\x)}{|\y-\x|}\ge0\rt\}.

562: \end{equation}

563: 

564: \begin{definition}

565: $V$ is a viscosity solution to (\ref{eq:pde}) if

566: \begin{equation}\label{eq:up}

567: H(\p)\vee\max_{i\in I(\x)}\p\cdot\bgamma_i\ge0, \quad

568: \p\in D^+V(\x),\ \x\in G,

569: \end{equation}

570: \begin{equation}\label{eq:down}

571: H(\p)\w\min_{i\in I(\x)}\p\cdot\bgamma_i\le0, \quad

572: \p\in D^-V(\x),\ \x\in G\setminus\pl_cG,

573: \end{equation}

574: and $V(\x)=0$ for $\x\in\pl_oG$.

575: \end{definition}

576: The following is a special case of Theorem 2 of \cite{ads}.

577: \begin{theorem}\label{th:main}

578: There exists a unique Lipschitz viscosity solution $V$ to (\ref{eq:pde}).

579: Moreover,

580: if $\x_n\in G^n$, $n\in\N$ are such that $\x_n\to \x\in G$, then

581: $\lim_{n\to\infty}V^n(\x_n) = V(\x)$.

582: \end{theorem}

583: For $i=1,\ldots,J$, let $\beta_i$ denote the unique positive solution to

584: \begin{equation}\label{eq:100}

585: c+\la(1-e^{\beta_i})+\mu_i(1-e^{-\beta_i})=0.

586: \end{equation}

587: %

588: Set $b_i=\beta_i\sum_{j=1}^i\e_j$.

589: Our main result is the following.

590: \begin{theorem}\label{th:tandem}

591: Assume $c>0$.

592: Then the viscosity solution to the PDE (\ref{eq:pde}) is given by

593: \begin{equation}\label{eq:101}

594: V(\x )=\min_{i=1,\ldots,J}b_i\cdot(\z -\x ).

595: \end{equation}

596: \end{theorem}

597: 

598: 

599: 

600: 

601: \section{Some remarks}\label{secrems}

602: 

603: \subsection{Interpretation}

604: 

605: %

606: As commented above, it is possible to characterize

607: the value function $V^n$

608: as the solution to a dynamic programming equation on the discrete space

609: $G^n$. Moreover, if $V^n$ is available in explicit form, one can

610: use the dynamic programming equation to specify an optimal control

611: policy for the problem. The results above provide an explicit

612: expression only to the limit $V=\lim_n V^n$, and therefore we are unable

613: to specify an optimal policy for the problem with finite $n$.

614: Instead, we shall use the quantity

615: $V$ along with the HJB equation \eqref{eq:pde} to propose a policy for

616: the queueing network problem (and the corresponding value $V^n$).

617: It is plausible that the proposed policy is, in a sense,

618: asymptotically optimal as $n\to\infty$ but we do not attempt

619: to prove such a statement here.

620: Instead, the discussion below could be regarded as a natural

621: interpretation of the result.

622: Moreover, since $V$ can be considered as a good approximation to

623: $V^n$ (for large $n$), it can be used as an initial condition

624: in a value iteration procedure that calculates $V^n$ and an optimal

625: policy.

626: 

627: The solution to the PDE stated here may be interpreted as follows.

628: Let $\x $ be an interior point of $G$, and assume that

629: $V$ is differentiable at $x$.

630: The form \eqref{eq:101} implies that for some $j=j(\x)$, 

631: \begin{equation}\label{e300}

632: DV(\x)=-b_j

633: =-\beta_j\sum_{i=1}^j\e_i \, .

634: \end{equation}

635: Hence by \eqref{eq:116}, the equation $H(-b_j)=0$ has the form

636: $$

637: \sup_{u\in U}\lt[c+\la(1-e^{\beta_j})+u_j\mu_j(1-e^{-\beta_j})\rt]=0.

638: $$

639: It is seen that the supremum is attained at any $u$ for which $u_j=1$

640: and $u_i\in[0,1]$, $i\ne j$. The interpretation in terms of the service policy

641: is that it is important to serve class $j$, while optimality does not depend

642: on the service given to other classes.

643: {\em The station $j=j(x)$ can therefore be regarded as a bottleneck:}

644: it is crucial to serve station $j$ when at state $x$.

645: Since the PDE describes the limit of the stochastic control problem,

646: one expects

647: the bottleneck stations to have a similar property in the stochastic problem:

648: Although it may be optimal for all servers to not idle when the system is near

649: an interior point $\x$, it is significant for the bottleneck server

650: to work while if the other servers idle the cost is affected only by little.

651: 

652: A closer look at \eqref{e300} reveals that all bottleneck stations

653: belong to a certain set $A'$ defined below.

654: More precisely,

655: given an interior point $x$ where $V$ is differentiable

656: and denoting $y=z-x$, using \eqref{eq:101}, we see

657: that \eqref{e300} holds for $j$ if and only if $j$ satisfies

658: \begin{equation}\label{e301}

659: \beta_j(y_1+\cdots+y_j)\le\beta_i(y_1+\cdots+y_i)\quad

660: \forall\ i\ne j.

661: \end{equation}

662: We claim that

663: a necessary condition for $j$ to satisfy \eqref{e301} (for some $y$)

664: is $j\in A'$, where

665: $$

666: A'=\{k\in[1,J]:\mu_k\le\mu_l,\ \mbox{ for all } l<k\},

667: $$

668: and by convention, $1\in A'$.

669: To this end, write the explicit form of the solution to~\eqref{eq:100} as

670: \begin{equation*}

671: e^{\beta_i } = (2\la)^{-1}[ c + \lambda + \mu_i

672:                    + ( c^2 + \lambda^2 + \mu_i^2 

673:                        + 2 c ( \lambda + \mu_i) )^{1/2}] \, ,

674: \end{equation*}

675: and note that the positive solutions $\beta_i$ to

676: \eqref{eq:100} are monotone in $\mu_i$, in the sense that

677: \begin{equation}\label{eq:mon}

678: \mu_i<\mu_j\quad \Longleftrightarrow\quad \beta_i<\beta_j.

679: \end{equation}

680: If \eqref{e301} holds then $\beta_i\le\beta_j$ for $j=1,\ldots,i$,

681: and it follows from \eqref{eq:mon} that $j\in A'$.

682: 

683: It should be emphasized that the condition $j\in A'$ is only

684: necessary for $j$ to be a bottleneck, and being a bottleneck is

685: really a function of the system's state. A more complicated necessary

686: condition is given in Section \ref{sec:proof}, which is of the form

687: $j\in A(x)$.

688: One interesting and perhaps counterintuitive phenomenon is that it is possible

689: that a station is nearly empty while others are far from being empty,

690: and still it is a bottleneck.

691: 

692: 

693: In the rest of this subsection we discuss a different queueing system

694: for which a similar analysis is possible, including an explicit form

695: for the limit of the value function and a clear interpretation of

696: it regarding nearly optimal service policies for the queueing system

697: (see \cite{ads} for proof of the results quoted below).

698: Consider a single server that provides

699: service to $J$ classes (each customer requires service once).

700: Service rate to queue $i$ is exponential with rate $ \mu_i $,

701: and the arrival process to class $i$ is Poisson of parameter $\la_i>0$,

702: $i=1,\ldots,J$. All arrival and service processes are mutually independent.

703: One considers the control space

704: \[

705: U=\left \{ u: \sum_{i=1}^J u_i\le 1, u_i \ge 0, i=1,\ldots,J \right \},

706: \]

707: where $u_i$ represents the fraction of service allocated to class $i$

708: by the single server.

709: Analogously to the problem discussed above, one defines a domain

710: $G=\{(x_1,\ldots,x_J):0\le x_i< z_i,1=1,\ldots,J\}$, scaled queueing

711: processes $X^n$ and exit time $\s^n$, and attempts minimizing the cost

712: $E_xe^{-nc\s^n}$ over an appropriate class of control processes.

713: Let $V^n(x)$ denote the infimum.

714: One can then show that $\lim_nV^n(x_n)$

715: exits whenever $x_n\to x\in G$.

716: Moreover, the limit $V$ is characterized in terms of a PDE

717: of the form \eqref{eq:pde}, where now $\gamma_i=e_i$, $i=1,\ldots,J$,

718: and $H(p)=\sup_{u\in U}H(p,u)$. Here $H(p,u)$ has the form

719: (analogous to \eqref{eq:116}):

720: \begin{equation}\label{216}

721: H(\p,u)=

722: c+\sum_{i=1}^J[\la_i(1-e^{-p_i})+u_i\mu_i(1-e^{p_i})],

723: \end{equation}

724: and the limit $V(x)$

725: can be explicitly calculated provided $c$ is large enough

726: as follows.

727: For $i=1,\ldots,J$, define $ \alpha_i $ as the unique positive

728: solution to $c+\la_i(1-e^{\al_i})+\mu_i(1-e^{-\al_i})=0$, namely

729: $$

730: e^{\alpha_i } = (2\la)^{-1}[ c + \lambda + \mu_i

731:                    + (( c + \lambda + \mu_i )^2

732: 		   - 4 \lambda \mu_i)^{1/2}]\, .

733: $$

734: Then if $c$ is large enough one has

735: \begin{equation}\label{eq:form}

736: V(\x )=\min_{i=1,\ldots,J}\al_i(z_i-x_i).

737: \end{equation}

738: Let $\x$ be a point in the interior of $G$

739: where the gradient of $V$ is well defined.

740: Then the PDE is satisfied in the classical sense at this point.

741: Thus, $H(DV(\x))=H(-\al_j\e_j)=0$, where $j=j(\x)$.

742: Using \eqref{216}, the equation $H(-\al_i\e_j)=0$ takes the form

743: $$

744: \sup_{u\in U}\lt[c+\la_j(1-e^{\al_j})+

745: u_j\mu_j(1-e^{-\al_j})\rt]=0.

746: $$

747: Clearly the supremum is attained at $u_i=1_{i=j}$,

748: $i=1,\ldots,J$.

749: This means that it is optimal to serve class $j(\x)$ at the

750: state $\x$.

751: In the totally symmetric case, where $\mu_i=\mu$, $\la_i=\la$,

752: $z_i=z$ for all $i$, the solution takes the form

753: $V(\x )=\al\min_i (z-x_i)$, and the optimal

754: service discipline can be interpreted as ``serve the longest

755: queue.'' An asymmetric two dimensional example

756: is given in Figure \ref{fig:example}, where the domain $G$

757: is divided into two subdomains $G_1$ and $G_2$ in accordance

758: with the structure (\ref{eq:form}), and the optimal service

759: discipline corresponds to giving priority to

760: class $i$ when the state is within $G_i$, $i=1,2$.

761: This discipline gives priority {\em to the queue with the (weighted)

762: shortest free buffer space.}

763: 

764: 

765: 

766: \begin{figure}

767: \centerline{

768: \begin{tabular}{cc}

769: \psfig{file=fig_net_2.eps}\qquad\qquad\qquad

770: \psfig{file=fig_ex_2.eps}

771: \end{tabular}

772: }

773: \caption{Priority to class $i$ when the state is in $G_i$, $i=1,2$.}

774: \label{fig:example}

775: \end{figure}

776: 

777: 

778: 

779: \subsection{The perturbed rates}

780: 

781: In the asymptotic analysis of rare events it is often the case

782: that most of the probability mass of an event of interest

783: is concentrated on the event that the stochastic process, say $X^n$,

784: nearly follows a certain deterministic path, for large values of $n$.

785: For example,

786: the most likely way a stable M/M/1 queue overflows is by nearly

787: following a deterministic path that is the solution to a fluid model

788: in which arrival and service rates are reversed.

789: This  appears as a consequence of time-reversal arguments in~\cite{sw}.

790: For many other cases where one can compute

791: deterministic paths about which most mass is concentrated see \cite{swbook}.

792: Similarly, in the current stochastic control problem,

793: when the system operates under the optimal control most

794: contribution to the cost is obtained when the process $X^n$ nearly

795: follows a certain deterministic path. This path is now

796: the solution to a fluid model with a control $u$ and perturbed

797: arrival and service rates $m=(\bar\la,\bar\mu_1,\ldots,\bar\mu_J)$.

798: At a point $x$ where the viscosity solution $V$ to \eqref{eq:pde}

799: is differentiable, the correct values of $u$ and $m$ are those that achieve

800: the max-min in $H(DV(x))=\sup_u\inf_mH(DV(x),u,m)$ (cf.\ \eqref{def:H}).

801: 

802: We remark that in the example of

803: tandem queues the optimal perturbed rates

804: satisfy certain relations with the unperturbed rates, as shown in

805: the two equations below.

806: By \eqref{eq:116}, the relation $H(DV)=0$ implies

807: \begin{equation}\label{e302}

808: \bar\la+\sum_{i=1}^Ju_i\bar\mu_i=c+\la+\sum_{i=1}^Ju_i\mu_i \, .

809: \end{equation}

810: Moreover, by \eqref{eq:116}, by the fact that the minimum over $m$

811: is attained at

812: $\bar\la=\la e^{-p_1}$, $\bar\mu_i=\mu_i e^{\bgamma_i\cdot \p}$,

813: and using $\sum_{i=1}^J \bgamma_i = \e_1$, we have

814: \begin{equation}\label{e303}

815: \bar\la\prod_{i=1}^J\bar\mu_i=\la\prod_{i=1}^J\mu_i.

816: \end{equation}

817: Relation \eqref{e302}

818: was noticed by Avram \cite{avr}.

819: The relation \eqref{e303} appears to be new.

820: A generalization of these equations to a more general network

821: is possible and will appear elsewhere.

822: 

823: \remark An interesting relation between the roots $\al_i$ and $\beta_i$

824: and the busy cycle period was

825: pointed out to us by Boxma \cite{box}.

826: For example, if $B$ denotes the busy cycle period for an M/M/1

827: queue under the stationary distribution, then $Ee^{cB}=e^\beta$,

828: where

829: $$

830: c+\la(1-e^{\beta})+\mu(1-e^{-\beta})=0

831: $$

832: (compare with \eqref{eq:100}).

833: 

834: 

835: 

836: \section{Proof of the main result}\label{sec:proof}

837: 

838: Recall that $I(\x )=\{i\in[1,J]:x_i=0\}$, and denote

839: $B(\x )=\{i\in[1,J]:x_i=z_i\}$. Note that $I( \x )$ and $B(\x )$

840: do not intersect. Since points $\x $ for which $x_1=z_1$ are not in $G$,

841: we have $1\not\in B(\x )$ for all $\x \in G$.

842: 

843: We show that the minimum over $i=1,\ldots,J$

844: in \eqref{eq:101} can equivalently be performed over a smaller

845: (state-dependent) set. 

846: As in Section \ref{secrems}, let 

847: \begin{equation}\label{eq:112}

848: A'=\{i\in[1,J]:\mu_i\le\mu_j,\ \mbox{ for all } j<i\},

849: \end{equation}

850: and by convention let $1\in A'$.

851: Let also

852: $$

853: A(x)=\{i\in A':\text{ either $i+1\not\in B(\x)$; or 

854: $j>i$ and $[i+1,j]\subset B(\x )$

855: imply $\mu_i<\mu_j$}\}.

856: $$

857: 

858: \begin{lemma}\label{lem:1}

859: The minimum in~\eqref{eq:101} is obtained over the indices in the

860: set $A(\x)$ defined above.

861: More precisely,

862: \[

863: V(\x)=\min_{i=1,\ldots,J}b_i\cdot(\z-\x)

864: =\min_{i\in A(\x)}b_i\cdot(\z-\x) \, .

865: \]

866: \end{lemma}

867: \proof

868: Fix $ \x \in G $ and recall that $ z_1 - x_1 > 0 $.

869: If

870: \begin{equation}\label{eq:mon1}

871: b_i\cdot(\z -\x )\le\min_{j\ne i}

872: b_j\cdot(\z -\x ),

873: \end{equation}

874: then $\beta_i\le\beta_j$ for $j \in [1,i]$,

875: and it follows from \eqref{eq:mon} that $i\in A'$.

876: Thus

877: \[

878: V (\x ) = \min_{i=1,\ldots,J}b_i\cdot(\z -\x )

879: =\min_{i\in A'}b_i\cdot(\z -\x )

880: \]

881: for all $\x \in G $ (although not every $ i \in A' $ is a minimizer).

882: Fix now $i \in A'$, and suppose that

883: \begin{equation}\label{eq:assume}

884: \mbox{there is some $j>i$ so that 

885: $[i+1,j]\subset B(\x )$ and $\mu_i \geq \mu_j$.}

886: \end{equation}

887: Since $ i \in A'$, if $ i+1 \not\in A' $ then $ \mu_{i+1} > \mu_i $.

888: But then,

889: if in addition $ i+2 \not\in A' $ we have $ \mu_{i+2} > \mu_i $.

890: So, if $ k \not\in A'$ for all $ i+1 \le k \le j-1 $ then

891: $ \mu_k > \mu_i $, and together with $ \mu_j \le \mu_i $ we have

892: that $ j \in A' $. We conclude that under~\eqref{eq:assume},

893: \begin{equation}\label{eq:200}

894: \mbox{there is $k\in[i+1,j]$ such that $k\in A'$.}

895: \end{equation}

896: By~\eqref{eq:assume},

897: $\beta_i \geq \beta_j$ and so

898: \begin{equation}\label{eq:201}

899: b_i \cdot(\z -\x )\geq b_j \cdot(\z -\x ).

900: \end{equation}

901: Therefore, for $k$ as in \eqref{eq:200} and under the assumptions

902: in~\eqref{eq:assume},

903: \[

904: b_i\cdot(\z-\x)\ge b_k\cdot(\z-\x) ,

905: \]

906: and since $k\in A'$, we need not consider $i$ in the minimum.

907: As a result,

908: \eqref{eq:101} is equivalently given as

909: $$

910: V(\x )=\min_{i\in A(\x )}b_i\cdot(\z -\x ),\quad \x \in G,

911: $$

912: where $ A (\x ) $ is obtained from $ A' (\x ) $ by deleting those

913: $ i \in A' (\x ) $ which are followed by empty queues,

914: $ [ i+1 , j ] \subset B (\x ) $, and such that $\mu_i \ge \mu_j $.

915: This leaves in $ A (\x ) $ an index $i$ from $ A' (\x ) $

916: only if either $ z_{i+1} > x_{i+1} $, or 

917: $ z_{k} = x_{k} $ for $ i+1 < k \le j $ implies $ \mu_i < \mu_j $.

918: More formally,

919: \begin{align}

920: \notag

921: A(\x) &=

922: A'\setminus\{i\in A':\exists\ j>i,\ [i+1,j]\subset B(\x),\ \mu_i\ge\mu_j\}\\

923: \label{eq:113}

924: &=

925: \{i\in A':\text{ either $i+1\not\in B(\x)$; or 

926: $j>i$ and $[i+1,j]\subset B(\x )$

927: imply $\mu_i<\mu_j$}\}.

928: \end{align}

929: \qed

930: 

931: 

932: The form of the proposed solution is the minimum of smooth functions.

933: The set of superdifferentials

934: at a point where the function is not smooth

935: can be seen (using the definition \eqref{def:Dplus})

936: to consist of the convex hull of the gradients of the

937: smooth functions defining it, at that point. Also,

938: at the boundary, the fact that there are less constraints introduced

939: by \eqref{def:Dplus} on $p\in D^+V$ than there are when $x$

940: is in the interior, has an effect of enlarging the set further.

941: Thus, for example, the set of

942: superdifferentials of the zero function from $\R_+$ to $\R$ at zero

943: is $\R_+$.

944: Using these considerations and the rectangular structure

945: of the domain, one finds the general form for the superdifferential

946: as follows.

947: Let $\x \in G$ be fixed. Set $A=A(\x )$, $I=I(\x )$, $B=B(\x )$, and

948: $O=\{1,\ldots,J\}\setminus (I\cup B)$.

949: Then any element $\p \in D^+V(\x )$ is given as

950: $$

951: \p =-\sum_{i\in A}\nu_ib_i+\bdel,

952: $$

953: for some $ \{\nu_i \}$, where

954: \begin{equation}\label{eq:102}

955: \del_i\ge0,\ i\in I,\quad

956: \del_i\le0,\ i\in B,\quad

957: \del_i=0,\ i\in O,

958: \end{equation}

959: and

960: \begin{equation}\label{eq:103}

961: \nu_i\ge0,\ i\in A,\quad \nu_i=0,\ i\not\in A,\quad \sum_{i\in A}\nu_i=1.

962: \end{equation}

963: If we denote

964: \begin{equation}\label{eq:105}

965: \del_{J+1}=0,

966: \end{equation}

967: then, using \eqref{eq:116}, the Hamiltonian is expressed as

968: \begin{align}\label{eq:114}

969: \notag

970: H(\p ) &= c+\la(1-e^{-p_1})+\sum_{i=1}^J0\vee\mu_i(1-e^{\p \cdot\bgamma_i})\\

971:      &= c+\la(1-e^{\sum_{i  \in A}\nu_i\beta_i-\del_1})

972:      +\sum_{i=1}^J0\vee\mu_i(1-e^{-\nu_i\beta_i+\del_i-\del_{i+1}}).

973: \end{align}

974: 

975: \proofOf{Theorem~\ref{th:tandem}}

976: 

977: \noi\uu{\it Verifying the PDE for superdifferentials.}

978: To verify \eqref{eq:up}, it suffices to show that

979: $H(\p )\ge0$ whenever

980: $\p \in DV^+(\x )$ and 

981: $\p \cdot\bgamma_i<0$ for all $i\in I$.

982: 

983: \noi\uu{\it Step 1.} We show first that $H(\p )\ge0$ whenever

984: $\p \in DV^+(\x )$ and  $\p \cdot\bgamma_i\le0$ for all $i=1,\ldots,J$.

985: The forms of $b_i$ and $\bgamma_i=\e _i-\e_{i+1}$ imply the last inequality can be rewritten

986: \begin{equation}\label{eq:104}

987: -\nu_i\beta_i+\del_i-\del_{i+1}\le0,\ i=1,\ldots,J.

988: \end{equation}

989: In this case, \eqref{eq:114} becomes

990: \begin{equation}\label{eq:109}

991: H(\p ) = h(\bnu,\bdel)

992:  \doteq c+\la(1-e^{\sum_{i \in A}\nu_i\beta_i-\del_1})

993: +\sum_{i=1}^J\mu_i(1-e^{-\nu_i\beta_i+\del_i-\del_{i+1}}).

994: \end{equation}

995: The constraints we have put on $(\bnu,\bdel)$ define a convex

996: set $S$, namely, the set of $(\bnu,\bdel)\in\R^J\times\R^{J+1}$ satisfying

997: \eqref{eq:102}, \eqref{eq:103}, \eqref{eq:105} and \eqref{eq:104}.

998: The set $S$ is bounded for the following reasons.

999: First, since $\bnu$ is a probability vector (see \eqref{eq:103}) we have $\nu_i\in[0,1]$.

1000: Next, by \eqref{eq:102} and $1\not\in B$,

1001:  $\del_1\ge0$.

1002: Finally,

1003: it follows from \eqref{eq:104}

1004: (using the convention $\sum_s^t=0$ if $t<s$) that 

1005: $$

1006: \del_1-\sum_{j=1}^{i-1}\nu_j\beta_j

1007: \le\del_i\le\sum_{j=i}^J\nu_j\beta_j,

1008: \quad i=1,\ldots, J.

1009: $$

1010: This shows the boundedness.

1011: 

1012: The function $(\bnu,\bdel)\mapsto h(\bnu,\bdel)$ is concave, and therefore to prove $h\ge0$ on $S$ it suffices to check the inequality

1013: on the set $E$ of extreme points of $S$, which contains a finite number of

1014: points since the constraints are linear.

1015: \begin{lemma}

1016: All points $(\bnu,\bdel)\in E$ have the following form:

1017: \begin{align*}

1018: \nu_k & = 1 \quad \text{for some $k$, and} \\

1019: \del_i^{(r,k)} & =\beta_k(1_{i\ge r+1}-1_{i\ge k+1}),\quad

1020: i=1,\ldots,J,\ r=s,\ldots,t-1,

1021: \end{align*}

1022: where $ t, s $ are defined below.

1023: \end{lemma}

1024: \proof

1025: We will first obtain the general form of $\bnu$

1026: and then that of $\bdel$.

1027: 

1028: We claim that for any $(\bnu,\bdel)\in E$, $\bnu$ is of the form

1029: $\bnu=1_k$ (short for $\nu_i=1_{i=k}$, $i=1,\ldots,J$),

1030: for some $k\in A$. Assume this is false.

1031: Then there are $l,m\in A$, $l<m$, for which $\nu_l,\nu_m\in(0,1)$.

1032: We will show that

1033: there is a vector $\bDel_m$, such that replacing $(\bnu,\bdel)$ by

1034: $(\bnu',\bdel')=(\bnu-\eps \e_m,\bdel+\eps\bDel_m)$ maintains

1035: the relations \eqref{eq:102}, \eqref{eq:105} and \eqref{eq:104}

1036: for both $\eps>0$ and $\eps<0$ (provided $|\eps|$ is small).

1037: A similar statement will hold also for

1038: $(\bnu'',\bdel'')=(\bnu+\eps \e_l-\eps \e_m,\bdel-\eps\bDel_l+\eps\bDel_m)$,

1039: and as a result all of \eqref{eq:102}, \eqref{eq:103},

1040: \eqref{eq:105} and \eqref{eq:104}

1041: will hold for both $\eps>0$ and $\eps<0$, a contradiction to

1042: $(\bnu,\bdel)$ being an extreme point of $S$.

1043: The construction of $\bDel_m$ based on $(\bnu,\bdel)$

1044: can be mimicked to construct $\bDel_l$ based on $(\bnu',\bdel')$

1045: (in particular, no use is made of the fact that $\nu_i$ sum to one,

1046: but only that some of its components are within $(0,1)$),

1047: and therefore the latter construction is omitted.

1048: 

1049: Case 1: {\it Inequality \eqref{eq:104} holds as a strict inequality

1050: for $i=m$, i.e., $-\nu_m\beta_m+\del_m-\del_{m+1}<0$.}

1051: Then $-(\nu_m-\eps)\beta_m+\del_m-\del_{m+1}\le0$

1052: also holds (for both $\eps$ positive and negative),

1053: provided that $|\eps|$ is small. Here we take $\bDel_m=0$.

1054: We see that $(\bnu-\eps \e_m,\bdel)$ satisfies the requirements.

1055: 

1056: Case 2: {\it Inequality \eqref{eq:104} holds with equality

1057: for $i=m$, i.e., $-\nu_m\beta_m+\del_m-\del_{m+1}=0$.}

1058: Since $\nu_m\beta_m>0$, either $\del_m>0$ or $\del_{m+1}<0$.

1059: Assume $\del_m>0$ (the case $\del_{m+1}<0$ can be treated analogously,

1060: and is therefore omitted).

1061: Let $m'$ be the smallest $j\in[1,m]$ for which \eqref{eq:104}

1062: holds with equality for all $i\in[j,m]$. For $i\in[m',m-1]$

1063: (the set being empty and the statement void if $m'=m$)

1064: we have $-\nu_i\beta_i+\del_i-\del_{i+1}=0$, hence if

1065: $\del_{i+1}>0$ then $\del_i>0$. Together with the fact that

1066: $\del_m>0$, this shows that

1067: \begin{equation}\label{eq:108}

1068: \del_i>0,\ i\in[m',m].

1069: \end{equation}

1070: Moreover,

1071: \begin{equation}\label{eq:107}

1072: -\nu_{m'-1}\beta_{m'-1}+\del_{m'-1}-\del_{m'}<0

1073: \end{equation}

1074: (the statement being void in case that $m'=1$).

1075: Set $\bDel_m=-\beta_m\sum_{i=m'}^m \e_i$, $\bnu'=\bnu+

1076: \eps \e_{m}$, and $\bdel'=\bdel+\eps\bDel_m$.

1077: The perturbation $\eps\bDel_m$ is chosen to cancel the change in $-\nu_m\beta_m$,

1078: and to preserve the left hand side of \eqref{eq:104}

1079: for all $i>m'-1$. However, by \eqref{eq:107}

1080: the inequality is maintained for $i=m'-1$ as well, by taking

1081: $|\eps|$ sufficiently small. Similarly, by \eqref{eq:108},

1082: \eqref{eq:102} and \eqref{eq:105} are also maintained on taking

1083: $|\eps|$ small.

1084: 

1085: This completes the construction of $\bDel_m$.

1086: As described above, this leads to a contradiction, and we conclude that

1087: any extreme point $(\bnu,\bdel)\in E$ satisfies $\bnu=1_k$ for some $k\in A$.

1088: 

1089: When $\bnu=1_k$ for some $k\in A$, inequalities \eqref{eq:104} can be rewritten as

1090: $$

1091: \del_i\le\del_{i+1},\ i\ne k,\quad \del_k\le \beta_k+\del_{k+1}.

1092: \eqno{(\ref{eq:104}')}

1093: $$

1094: In particular, $0\le\del_1\le\cdots\le\del_k$.

1095: Let $s$ denote the largest $j\le k$ for which $j\in B\cup O$

1096: (and $s=0$ if there is no such $j$). 

1097: The definitions of $O$ and $B$ imply $\delta_s=0$,

1098: and thus

1099: \stepcounter{equation}

1100: $$

1101: \text{$\del_j=0$ for $1\le j\le s$.}

1102: \eqno{(\theequation a)}

1103: $$

1104: Similarly, $\del_{k+1}\le\cdots\le\del_J\le0$,

1105: and if $t$ denotes the least $j\in [k+1,J]$ for which

1106: $j\in I\cup O$ (and $t=J+1$ if empty), then

1107: $$

1108: \del_j=0,\quad t\le j\le J.

1109: \eqno{(\theequation b)}

1110: $$

1111: Hence $\bdel$ must satisfy

1112: $$

1113: \begin{cases}

1114: 0\le\del_{s+1}\le\cdots\le\del_k\le\beta_k+\del_{k+1},

1115: &

1116: \text{when $s\le k-1$},\\

1117: 0\le\del_k

1118: &

1119: \text{when $s=k$}.

1120: \end{cases}

1121: \eqno{(\theequation c)}

1122: $$

1123: In addition,

1124: $$

1125: \begin{cases}

1126: \del_{k+1}\le\cdots\le\del_{t-1}\le0,&

1127: \text{when }t\ge k+2\\

1128: \del_{k+1}\le0 &

1129: \text{when }t=k+1.

1130: \end{cases}

1131: \eqno{(\theequation d)}

1132: $$

1133: We have just shown that \eqref{eq:102}, \eqref{eq:105}

1134: and (\ref{eq:104}') imply (36$a$--$d$). On the other hand, clearly

1135: (36$a$--$d$) implies (\ref{eq:104}').

1136: Moreover, as follows directly from the definition of $s$ and $t$,

1137: \begin{equation}\label{eq:106}

1138: \text{$i\in I$ for all $i\in[s+1,k]$ and $i\in B$

1139: for all $i\in[k+1,t-1]$}.

1140: \end{equation}

1141: This shows that (36$a$--$d$) implies \eqref{eq:102} and \eqref{eq:105}.

1142: Thus \eqref{eq:102}, \eqref{eq:105} and (\ref{eq:104}')

1143: are equivalent to (36$a$--$d$).

1144: Now, the set of $\bdel$ satisfying

1145: the constraints (36$a$--$d$) is easy to analyze.

1146: In particular, it is not hard to see that it has the following

1147: $t-s$ extreme points, indexed by $k$ and $r\in[s,\ldots,t-1]$, namely

1148: \begin{equation}\label{eq:110}

1149: \del_i^{(r,k)}=\beta_k(1_{i\ge r+1}-1_{i\ge k+1}),\quad

1150: i=1,\ldots,J,\ r=s,\ldots,t-1.

1151: \end{equation}

1152: \qed

1153: 

1154: We now calculate $h$ (cf.\ \eqref{eq:109}) at each extreme point.

1155: To this end, note that if $(\bnu,\bdel)=

1156: (1_k,\bdel^{(r,k)})$ then by \eqref{eq:110}

1157: $$

1158: -\nu_i\beta_i+\del_i-\del_{i+1}=

1159: \begin{cases}

1160: -\beta_k & i=r,\\ 0 & i\ne r,

1161: \end{cases}

1162: \qquad i=1,\ldots,J,

1163: $$

1164: and

1165: $$

1166: \del_1=

1167: \begin{cases}

1168: \beta_k & r=0,\\ 0 & r>0.

1169: \end{cases}

1170: $$

1171: Substituting in \eqref{eq:109}, we have the following possibilities.

1172: If $r=0$ then $\del_i^{(r,k)}=\beta_k1_{i\in[1,k]}$

1173: thus $\sum\nu_i\beta_i-\del_1=0$, and

1174: $-\nu_i\beta_i+\del_i-\del_{i+1}=0$, $i=1,\ldots,J$. Hence

1175: $h(1_k,\bdel^{(r,k)})=c>0$.

1176: Otherwise,

1177: \begin{equation}\label{eq:111}

1178: h(1_k,\bdel^{(r,k)})=c+\la(1-e^{\beta_k})+\mu_r(1-e^{-\beta_k}).

1179: \end{equation}

1180: In case that $r=k$, the right hand side of \eqref{eq:111} vanishes owing to the definition of $\beta_k$ (see \eqref{eq:100}).

1181: In case that $r<k$, recall that $k\in A$, and in particular,

1182: $k\in A'$. By \eqref{eq:112} we therefore have $\mu_k\le\mu_r$,

1183: hence by \eqref{eq:100}, $h(1_k,\bdel^{(r,k)})\ge0$.

1184: Finally, consider the case where $k<r\le t-1$. By \eqref{eq:106},

1185: $[k+1,r]\subset B$. Since $k\in A$, \eqref{eq:113} implies

1186: that $\mu_k<\mu_r$, and we again conclude that $h(1_k,\bdel^{(r,k)})>0$.

1187: Having shown that $h(\bnu,\bdel)\ge0$ for all

1188: extreme points of the set $S$, we conclude that the inequality holds

1189: for all $(\bnu,\bdel)\in S$.

1190: 

1191: \noi\uu{\it Step 2.}

1192: We now relax the condition \eqref{eq:104}.

1193: Let then $(\bnu,\bdel)$ satisfy \eqref{eq:102}, \eqref{eq:103},

1194: \eqref{eq:105}, and assume that the inequality in \eqref{eq:104} holds

1195: for all $i\in I$ (relaxing the assumption made in Step 1, that it

1196: holds for all $1\le i\le J$). Let $P=P(\bnu,\bdel)$ denote the set of

1197: $i$ such that

1198: \begin{equation}\label{eq:130}

1199: -\nu_i\beta_i+\del_i-\del_{i+1}>0.

1200: \end{equation}

1201: If $P$ is empty then the results of Step 1 apply and $H(\p )\ge0$.

1202: Hence assume $P$ is not empty.

1203: Let $j=j(\bnu,\bdel)$ be the least element in $P$.

1204: 

1205: Note that $J\not\in P$. For if $J\in P$ then using

1206: \eqref{eq:130} and \eqref{eq:105} we find $\del_J>0$, and therefore $J\in I$. 

1207: However, we get to assume \eqref{eq:104} for all $i \in I$, which means that 

1208: $-\nu_J\beta_J+\del_J-\del_{J+1}\le0$.  This contradicts $J\in P$.

1209: 

1210: We proceed by backward induction on the value of $j(\bnu,\bdel)$.

1211: Note that the sets $B$ and $I$ depend on $x$. The argument below treats

1212: simultaneously all $x\in G$ by considering all

1213: possible sets $B$ and $I$.

1214: 

1215: \noi

1216: {\it Induction step.} Assumption: For all $(I,B,\bnu,\bdel)$

1217: satisfying \eqref{eq:102}, \eqref{eq:103}, \eqref{eq:105},

1218: such that $P(\bnu,\bdel)\cap I=\emptyset$ and $j(\bnu,\bdel)\in

1219: [i+1,i+2,\ldots,J-1]$, one has $H(\p )\ge0$.

1220: Let $(I,B,\bnu,\bdel)$ be such that $P(\bnu,\bdel)\cap I=\emptyset$

1221: and $j(\bnu,\bdel)=i$ ($i\ge1$). Then

1222: $-\nu_i\beta_i+\del_i-\del_{i+1}>0$.

1223: Modify $\del_{i+1}$ by increasing it so as to get equality i.e.,

1224: set $\del_{i+1}'>\del_{i+1}$ so that

1225: $-\nu_i\beta_i+\del_i-\del'_{i+1}=0$.

1226: This modification does not change the value of

1227: $0\vee\mu_i(1-e^{-\nu_i\beta_i+\del_i-\del_{i+1}})$

1228: (but keeps it zero), and it can only decrease (or leave unchanged)

1229: the value of

1230: $0\vee\mu_{i+1}(1-e^{-\nu_{i+1}\beta_{i+1}+\del_{i+1}-\del_{i+2}})$.

1231: Hence by \eqref{eq:114}, the value of $H(\p )$ is only decreased.

1232: At the same time, $j(\bnu,\bdel')>j(\bnu,\bdel)$.

1233: Note that in modifying $\bdel$, \eqref{eq:102} need not hold

1234: for $B$ and $I$.

1235: However, clearly there are other sets, $B'$ and $I'$

1236: with which it holds. For example,

1237: \begin{equation}\label{eq:151}

1238: B'=\{i\in[1,J]:\del'_i<0\},\quad

1239: I'=\{i\in[1,J]:\del'_i\ge0\},\quad

1240: O'=\emptyset.

1241: \end{equation}

1242: Moreover, $1\not\in B'$, since we had

1243: $1\not\in B$ and $\del_1$ was not modified.

1244: By the induction assumption we therefore obtain that $H(\p )\ge0$.

1245: 

1246: \noi {\it Induction base.} We show that for all $(I,B,\bnu,\bdel)$

1247: satisfying \eqref{eq:102}, \eqref{eq:103}, \eqref{eq:105},

1248: such that $P(\bnu,\bdel)\cap I=\emptyset$ and $j(\bnu,\bdel)=J-1$,

1249: one has $H(\p )\ge0$.

1250: We have $-\nu_{J-1}\beta_{J-1}+\del_{J-1}-\del_J>0$.

1251: Similar to before, we set

1252: $\del_J'>\del_J$ so that $-\nu_{J-1}\beta_{J-1}+\del_{J-1}-\del'_J=0$.

1253: We claim that $P(\bnu,\bdel')=\emptyset$. Indeed, since

1254: $P(\bnu,\bdel)=\{J-1\}$, clearly $P(\bnu,\bdel')\subset\{J\}$.

1255: However,

1256: \begin{align*}

1257: -\nu_J\beta_J+\del'_J &=

1258: -\nu_J\beta_J-\nu_{J-1}\beta_{J-1}+\del_{J-1}\\

1259: &\le

1260: 0,

1261: \end{align*}

1262: where we have used the fact that $J-1\in P(\bnu,\bdel)$

1263: implies $J-1\not\in I$, and hence by \eqref{eq:102} $\del_{J-1}\le0$.

1264: This shows that $P(\bnu,\bdel')=\emptyset$.

1265: As in the previous paragraph, because of the modification of $\bdel$,

1266: \eqref{eq:102} need not hold for the $I$ and $B$ we started with,

1267: but there are other sets $I'$ and $B'$ (defined e.g.\ as in \eqref{eq:151})

1268: with which it holds.

1269: The results of Step 1 therefore apply, and therefore $H(\p )\ge0$.

1270: 

1271: This completes the argument by induction and establishes

1272: \eqref{eq:up} for superdifferentials.

1273: 

1274: \noi\uu{\it Verifying the PDE for subdifferentials.}

1275: We are required to show that \eqref{eq:down} holds for

1276: $\p \in D^-V(\x )$, where $\x \in G\setminus\pl_c G$ (and hence $B(\x )=\emptyset$).

1277: Unless $D^-V(\x )$ is empty, the general form of $\p \in D^-V(\x )$, $\x \in G$ is

1278: $$

1279: \p =-b_k+\bdel,

1280: $$

1281: where $k\in A$, and

1282: \begin{equation}\label{eq:115}

1283: \del_i\le0,\ i\in I,\quad

1284: \del_i=0,\ i\not\in I.

1285: \end{equation}

1286: Here we have used the fact that $V$ is the minimum of smooth functions,

1287: and therefore away from the boundary $D^-V(\x )$ can have at most one element,

1288: equal to the gradient of any minimizing function.

1289: It suffices to show that $H(\p )\le0$ whenever $\p \cdot\bgamma_i

1290: =-1_{i=k}\beta_k+\del_i-\del_{i+1}>0$ for all

1291: $i\in I$. 

1292: Using \eqref{eq:114}, \eqref{eq:115} and \eqref{eq:100},

1293: we find

1294: \begin{align*}

1295: H(\p )

1296: &=

1297: c+\la(1-e^{\beta_k-\del_1})+\sum_{i=1}^J0\vee\mu_i(1-e^{-1_{i=k}\beta_k

1298: +\del_i-\del_{i+1}})\\

1299: &=

1300: c+\la(1-e^{\beta_k-\del_1})+\sum_{i\not\in I}\mu_i(1-e^{-1_{i=k}\beta_k

1301: -\del_{i+1}})\\

1302: &\le

1303: c+\la(1-e^{\beta_k})+\sum_{i\not\in I}\mu_i(1-e^{-1_{i=k}\beta_k})\\

1304: &\le

1305: c+\la(1-e^{\beta_k})+\mu_k(1-e^{-\beta_k})\\

1306: &=0.

1307: \end{align*}

1308: \qed

1309: 

1310: \newpage

1311: \bibliographystyle{plain}

1312: \begin{thebibliography}{99}

1313: 

1314: \bibitem{atadup2} R.~Atar and P.~Dupuis,

1315: ``A differential game with constrained dynamics."

1316: {\em Nonlinear Analysis} Vol. 51, No. 7, p. 1105--1130 (2002)

1317: 

1318: \bibitem{ads} R.~Atar, P.~Dupuis and A.~Shwartz,

1319: ``An escape time criterion for queueing networks:

1320: Asymptotic risk-sensitive control via differential games,''

1321: Math. Op. Res., to appear.

1322: 

1323: \bibitem{avr} F.~Avram,

1324: ``On asymptotic optimal control problems for stochastic networks,''

1325: preprint, 2002.

1326: 

1327: \bibitem{box} O. J. Boxma, personal communication.

1328: 

1329: \bibitem{balday2}

1330: J.~Ball, M.~Day, and P.~Kachroo,

1331: ``Robust feedback control for a single server queueing system."

1332: {\em Mathematics of Control, Signals, and Systems \bf 12}, pp.~307--345, 1999.

1333: 

1334: \bibitem{barcap}

1335: M.~Bardi and I.~Capuzzo-Dolcetta,

1336: {\em Optimal control and viscosity solutions

1337: of Hamilton-Jacobi-Bellman equations.}

1338: Birkhauser, Boston, 1997.

1339: 

1340: \bibitem{dup} P.~Dupuis,

1341: {\em Explicit solution to a robust queueing control problem.}

1342: preprint.

1343: 

1344: \bibitem{dupjampet}

1345: P.~Dupuis, M.R.~James and I.R.~Petersen.

1346: ``Robust properties of risk--sensitive control."

1347: {\em Math.\ of Control, Signals and Systems

1348: \bf 13} pp.~318--332, 2000.

1349: 

1350: \bibitem{dupkus}

1351: P.~Dupuis and H.J.~Kushner,

1352: ``Minimizing escape probabilities: a large deviations approach. "

1353: {\em SIAM J.\ Control Optim.\ \bf 27}, pp.~432--445, 1989.

1354: 

1355: \bibitem{ellkal} R.J.~Elliott and N.J.~Kalton,

1356: {\em The existence of value in differential games,}

1357: Memoirs of the American Mathematical

1358: Society No.\ 126. American Mathematical Society,

1359: Providence, R.I., iv+67 pp., 1972.

1360: 

1361: \bibitem{flesou}

1362: W.H.~Fleming and P.E.~Souganidis,

1363: ``PDE-viscosity solution approach to some problems of large deviations."

1364: {\em Ann.\ Scuola Norm.\ Sup.\ Pisa Cl.\ Sci.\ \bf 4}, pp.~171--192, 1986.

1365: 

1366: \bibitem{sw}

1367: A.~Shwartz and A.~Weiss, ``Induced rare events: analysis

1368: via large deviations and time reversal."

1369: {\em Journal of Applied Prob.\ \bf 25}, pp.~667--689, 1993.

1370: 

1371: \bibitem{swbook}

1372: A. Shwartz and A. Weiss.

1373: \newblock {\it Large Deviations for Performance Analysis: Queues,

1374: Communication

1375:   and Computing}.

1376: \newblock Chapman and Hall, New York, 1995.

1377: 

1378: \bibitem{sto}

1379: A.~Stolyar,

1380: {\em Control of end-to-end delay tails in a multiclass network:

1381: LWDF discipline optimality.} Preprint, 2000.

1382: 

1383: \bibitem{storam}

1384: A.~Stolyar and K.~Ramanan,

1385: ``Largest weighted delay first scheduling:

1386: Large deviations and optimality."

1387: {\em Ann.\ Appl.\ Probab.\ \bf 11}, pp.~1--48, 2001.

1388: 

1389: \end{thebibliography}

1390: 

1391: 

1392: \end{document}

1393: 

1394: 

1395: 

1396: