1: \documentclass[10pt,reqno]{amsart}
2:
3: \pagestyle{empty}
4: \usepackage[margin=0.625in,bottom=1in,top=0.75in,twosideshift=0in]{geometry}
5: \linespread{1.0}
6: \usepackage{multicol}
7: \newcommand{\mysubsection}[1]{\subsection{\normalfont \text{ }\emph{#1}}\text{}\indent}
8: \usepackage{amssymb, amsmath}
9: \usepackage{amsthm}
10: \usepackage[usenames]{color}
11: \usepackage{calc}
12: \usepackage{times}
13: \newcounter{myFCounter}[section]
14: \usepackage{graphicx}
15: \newcommand{\myFigure}[4]{
16: \footnotesize
17: \begin{center}
18: \begin{minipage}[!t]{\columnwidth}%
19: \begin{center}\refstepcounter{myFCounter}\vspace{1ex}%
20: \includegraphics[width=#1,keepaspectratio]{#2}\ \\%
21: \parbox{3in}{
22: \begin{center}
23: Fig.\ \arabic{myFCounter}.\ \rm #3
24: \end{center}
25: }\label{#4}
26: \vspace{1ex}
27: \end{center}
28: \end{minipage}
29: \end{center}
30: \normalsize
31: }
32: \usepackage{bm}
33: \renewcommand{\labelenumi}{(\roman{enumi})}
34: \usepackage{comment}
35: %short version
36: \excludecomment{long-version}
37: \includecomment{short-version}
38:
39: %long version
40: %\includecomment{long-version}
41: %\excludecomment{short-version}
42:
43: \usepackage{ifpdf}
44:
45:
46: \newcommand{\esup}{\mathop{\text{ess\;sup}}} % essential supremum
47: \renewcommand{\P}{\mathbb{P}} % probability measure
48: \newcommand{\E}{\mathbb{E}} % expectation
49: \newcommand{\Em}{\mathcal{E}} %space of measurements
50: \newcommand{\V}{\text{Var}} % variance
51: \newcommand{\F}{\mathcal{F}} % filtration
52: \newcommand{\Fb}{\mathbb{F}} % filtration
53: \newcommand{\Mh}{\mathcal{M}} % set of hypotheses
54: \newcommand{\N}{\mathbb{N}} % integers
55: \newcommand{\R}{\mathbb{R}} % real numbers
56: \newcommand{\I}{\bm{1}} % basic indicator function 1
57: \newcommand{\II}[1]{\bm{1}_{\left\{#1\right\}}} % indicator with brackets 1_{A}
58: \newcommand{\bpi}{\pi} % bold \pi
59: \newcommand{\bPi}{\Pi} % bold \Pi
60: \newcommand{\e}{e}
61: \newcommand{\T}{\mathbb{T}}
62: \newcommand{\M}{\mathbb{M}} % Operator (Mf)(pi) = min{h(pi), c(1 - pi_0) + (Tf)(pi)}
63:
64: \newtheorem{theorem}{Theorem}
65: \newtheorem{proposition}[theorem]{Proposition}
66: \newtheorem{lemma}[theorem]{Lemma}
67: \newtheorem{definition}[theorem]{Definition}
68: \newtheorem{corollary}[theorem]{Corollary}
69: \newtheorem{remark}[theorem]{Remark}
70: \newtheorem{conjecture}[theorem]{Conjecture}
71: \newtheorem{notation}[theorem]{Notation}
72: \newtheorem{example}[theorem]{Example}
73: \newtheorem{exercise}{Exercise}
74: \newtheorem*{notes}{Notes}
75: \renewcommand{\thesection}{\Roman{section}}
76: \renewcommand{\thesubsection}{\emph{\Alph{subsection}}}
77: \columnsep 0.25in
78:
79: \usepackage[unicode]{hyperref}
80: \hypersetup{
81: draft %to suppress all hypertext options
82: % %colorlinks=true,
83: % %linkcolor=webbrown,
84: % %filecolor=webbrown,
85: % %citecolor=webgreen,
86: % %breaklinks=true,
87: % %pdfstartview=FitH,
88: % %pdfpagemode=FitWidth,
89: % %pdfpagemode=UseOutlines,
90: % %bookmarksopen=true,
91: % %bookmarksnumbered=true
92: }
93: %\definecolor{webgreen}{rgb}{0,.5,0}
94: %\definecolor{webbrown}{rgb}{.6,0,0}
95: \usepackage{ifpdf}
96:
97:
98: \begin{document}
99:
100: \thispagestyle{empty}
101:
102: \begin{center}
103: \Huge Joint Detection and Identification of an Unobservable Change in the Distribution of a Random Sequence\\
104: \vskip 10pt
105: \large
106: \begin{tabular}[t]{c@{\extracolsep{4em}}c}
107: Savas Dayanik and Christian Goulding & H.\ Vincent Poor \\
108: Dept. of Operations Research and Financial Engineering & School of Engineering and Applied Science \\
109: Princeton University, Princeton, NJ~~08544 & Princeton University, Princeton, NJ~~08544 \\
110: Email: \{sdayanik, cgouldin\}@princeton.edu & Email: poor@princeton.edu
111: \end{tabular}
112: \normalsize
113: \end{center}
114:
115:
116: \begin{multicols}{2}{
117: \textbf{\emph{Abstract--}This paper examines the joint problem of
118: detection and identification of a sudden and unobservable change
119: in the probability distribution function (pdf) of a sequence of
120: independent and identically distributed (i.i.d.)\ random variables
121: to one of finitely many alternative pdf's. The objective is quick
122: detection of the change and accurate inference of the ensuing pdf.
123: Following a Bayesian approach, a new sequential decision strategy
124: for this problem is revealed and is proven optimal. Geometrical
125: properties of this strategy are demonstrated via numerical
126: examples.}
127:
128: \section{Introduction} \label{sec:Introduction}
129:
130: Consider a sequence of i.i.d.\ random variables $X_1, X_2,
131: \ldots$, taking values in some measurable space $(E,\Em)$. The
132: common probability distribution of the $X$'s is initially some
133: known probability measure $\P_0$ on $(E,\Em)$, and then, at some
134: \emph{unobservable} disorder time $\theta$, the common probability
135: distribution changes suddenly to another probability measure
136: $\P_{\mu}$ for some \emph{unobservable} index $\mu\in \Mh
137: \triangleq \{1,\ldots,M\}$. The objective is to detect the change
138: as quickly as possible, and, at the same time, to identify the new
139: probability distribution as accurately as possible, so that the
140: most suitable actions can be taken with the least delay.
141:
142: This problem can be viewed as the fusion of two fundamental areas
143: of sequential analysis: change detection and multi-hypothesis
144: testing. In traditional change detection problems, there is only
145: one change distribution, $\P_1$; therefore, the focus is
146: exclusively on detecting the change time. Whereas, in traditional
147: sequential multi-hypothesis testing problems, there is no change
148: time to consider. Instead, every observation has common
149: distribution $\P_\mu$ for some unknown $\mu$, and the focus is
150: exclusively on the inference of $\mu$. Both of these subproblems
151: have been studied extensively. For recent reviews of these areas,
152: we refer the reader to \cite{MR1210954} and \cite{MR1844531} and
153: the references therein.
154:
155:
156: However, the joint problem involves key trade-off decisions not
157: taken into account by separately applying techniques for these
158: subproblems. While raising an alarm as soon as the change occurs
159: is advantageous for the change detection task, it is undesirable
160: for the identification task because waiting longer provides more
161: observations for inferring the change distribution. Likewise, the
162: unknown change time complicates the identification task, and, as a
163: result, adaptation of existing sequential multi-hypothesis testing
164: algorithms is problematic.
165: \noindent\begin{minipage}[!b]{\columnwidth} \footnotetext{The
166: research of Savas Dayanik was supported by the Air Force Office of
167: Scientific Research, under grant AFOSR-FA9550-06-1-0496. The
168: research of H.\ Vincent Poor was supported in part by the U.S.\
169: Army Pantheon Project.}
170: \end{minipage}
171:
172: Decision strategies for the joint problem have a wide array of
173: applications, such as fault detection and isolation in industrial
174: processes, target detection and identification in national
175: defense, pattern recognition and machine learning, radar and sonar
176: signal processing, seismology, speech and image processing,
177: biomedical signal processing, finance, and insurance. However, the
178: theory has not been broadly developed.
179: Nikiforov~\cite{Nikiforov1995} provides the first results for this
180: problem, showing asymptotic optimality for a certain non-Bayesian
181: approach, and Lai~\cite{Lai2000} generalizes these results through
182: the development of information-theoretic bounds and the
183: application of likelihood methods. In this paper, we follow a
184: Bayesian approach to reveal a new optimal strategy for this
185: problem and we describe an accurate numerical scheme for its
186: implementation.
187:
188: In Sec.\ \ref{sec:Problem-statement} we formulate precisely the
189: problem in a Bayesian framework, and in Sec.\
190: \ref{sec:Reformulation} we show that it can be reduced to an
191: optimal stopping of a Markov process whose state space is the
192: standard probability simplex. In addition, we establish a simple
193: recursive formula that captures the dynamics of the process and
194: yields a sufficient statistic fit for online tracking.
195:
196: In Sec.\ \ref{sec:dynamic-programming-solution} we use optimal
197: stopping theory to substantiate the optimality equation for the
198: value function of the optimal stopping problem. Moreover, we
199: prove that this value function is bounded, concave, and continuous
200: on the standard probability simplex and that the optimal stopping
201: region consists of $M$ non-empty, convex, closed, and bounded
202: subsets. Also, we consider a truncated version of the problem
203: that allows at most $N$ observations from the sequence of random
204: measurements. We establish an explicit bound (inversely
205: proportional to $N$) for the approximation error associated with
206: this truncated problem.
207:
208: In Sec.\ \ref{sec:Special-cases} we show that the separate
209: problems of change detection and sequential multi-hypothesis
210: testing are solved as special cases of the overall joint solution.
211: We illustrate some geometrical properties of the optimal method
212: and demonstrate its implementation by numerical examples for the
213: special cases $M=2$ and $M=3$. Specifically, we show instances in
214: which the $M$ convex subsets comprising the optimal stopping
215: region are connected and instances in which they are not.
216: Likewise, we show that the continuation region (i.e., the
217: complement of the stopping region) need not be connected. We refer
218: the reader to \cite{DGP06} for complete proofs of the results.
219:
220:
221: \section{Problem statement}
222: \label{sec:Problem-statement}
223:
224: Let $(\Omega,\F, \P)$ be a probability space hosting random
225: variables $\theta:\Omega\mapsto\{0,1,\ldots\}$ and
226: $\mu:\Omega\mapsto \Mh \triangleq \{1,\ldots,M\}$ and a process $X
227: = (X_n)_{n\geq1}$ taking values in some measurable space
228: $(E,\Em)$. Suppose that for every $t\ge 1$, $i \in \Mh$, $n\ge 1$,
229: and $(E_k)^n_{k=1}\subseteq \Em $ we have
230: \begin{multline*}
231: \P\left\{\theta=t, \mu=i, X_1 \in E_1,\ldots, X_n \in E_n \right\} \\
232: = (1-p_0) (1-p)^{t-1} p \nu_i \prod_{k=1}^{(t-1) \land n}
233: \P_0(E_k) \prod_{\ell = t\lor 1}^{n} \P_i(E_{\ell})
234: \end{multline*}
235: for some given probability measures $\P_0,\P_1,\ldots,\P_M$ on
236: $(E,\Em)$, known constants $p_0\in[0,1]$, $p\in(0,1)$, and
237: $\nu_i>0,i\in \Mh$ such that $\nu_1+\cdots+\nu_M = 1$, where
238: $x\wedge y\triangleq \min\{x,y\}$ and $x\vee y\triangleq
239: \max\{x,y\}$. Namely, $\theta$ is independent of $\mu$; it has a
240: zero-modified geometric distribution with parameters $p_0$ and $p$
241: in the terminology of
242: %Klugman, Panjer, and Willmot
243: \cite[Sec.\
244: 3.6]{MR1490300}, which reduces to the standard geometric
245: distribution when $p_0=0$.
246:
247: Conditionally on $\theta$ and $\mu$, the random variables $X_n$, $n\ge
248: 1$ are independent; $X_1,\ldots, X_{\theta-1}$ and $X_{\theta},
249: X_{\theta+1},\ldots$ are identically distributed with common
250: distributions $\P_0$ and $\P_{\mu}$, respectively. The probability
251: measures $\P_0,\P_1,\ldots,\P_M$ always admit densities with respect
252: to some sigma-finite measure $m$ on $(E,\Em)$; for example, we can
253: take $m = \P_0+\P_1\cdots+\P_M$. So, we fix $m$ and denote the
254: corresponding densities by $f_0, f_1,\ldots,f_M$, respectively.
255:
256: Suppose now that we observe sequentially the random variables
257: $X_n$, $n\ge 1$. Their common pdf $f_0$ changes at stage $\theta$
258: to some other pdf $f_{\mu}$, $\mu\in \Mh$. Our objective is to
259: detect the change time $\theta$ as quickly as possible \emph{and}
260: to identify the change index $\mu$ as accurately as possible. More
261: precisely, given costs associated with detection delay, false
262: alarm, and false identification of the change index, we seek a
263: strategy that minimizes the expected total change detection
264: \emph{and} identification cost.
265:
266: Let $\mathbb{F} = (\F_n)_{n\geq0}$ denote the natural filtration
267: of the observation process $X$, where
268: \begin{align*}
269: \F_0=\{\varnothing,\Omega\}\quad\text{and}\quad
270: \F_n=\sigma(X_1,\ldots,X_n),\quad n\geq1.
271: \end{align*}
272: A \emph{strategy} $\delta=(\tau, d)$ is a pair consisting of a
273: \emph{stopping time} $\tau$ of the filtration $\mathbb{F}$ and a
274: \emph{terminal decision rule} $d: \Omega \mapsto \Mh$ measurable
275: with respect to the history $\F_{\tau}=\sigma(X_{n\wedge\tau};
276: n\geq1)$ of observation process $X$ through stage $\tau$. Applying
277: a strategy $\delta=(\tau,d)$ consists of announcing at the end of
278: stage $\tau$ that the common pdf has changed from $f_0$ to $f_d$
279: at or before stage $\tau$. Let
280: \begin{align*}
281: \Delta \triangleq \{(\tau,d) \mid \tau\in\mathbb{F}, \text{ and
282: $d\in\F_\tau$ is an $\Mh$-valued r.\ v.}\}
283: \end{align*}
284: denote the collection of all such sequential decision strategies.
285:
286: For every strategy $\delta=(\tau, d)\in\Delta$, we define a
287: \emph{Bayes risk function}
288: \begin{align}
289: R(\delta) = c\,\E[(\tau-\theta)^+] + \E[a_{0
290: d}\II{\tau<\theta}+a_{\mu
291: d}\II{\theta\leq\tau<\infty}]\label{E:BayesRiskUnderP}
292: \end{align}
293: \noindent as the expected diagnosis cost: the sum of the expected
294: detection delay cost and the expected terminal decision cost upon
295: alarm, where $c>0$ and $a_{ij}\ge 0, i\in\{0\}\cup\Mh,j\in\Mh$ are
296: known constants satisfying $a_{ii}=0, i\in\Mh$ (i.e., no cost for
297: a correct terminal decision), and $(x)^+\triangleq\max\{x,0\}$.
298:
299: The problem is to find a sequential decision strategy
300: $\delta=(\tau,d)\in\Delta$ (if it exists) with the \emph{minimum
301: Bayes
302: risk}
303: \begin{align}
304: R^* \triangleq \inf_{\delta\in\Delta} R(\delta).\label{E:UDef1}
305: \end{align}
306:
307:
308: \section{Posterior analysis and formulation as an optimal stopping
309: problem}
310: \label{sec:Reformulation}
311:
312: In this section we show that the Bayes risk function in
313: (\ref{E:BayesRiskUnderP}) can be written as the expected value of the
314: running and terminal costs driven by a certain Markov process. We use
315: this fact to recast the minimum Bayes risk in (\ref{E:UDef1}) as a
316: Markov optimal stopping problem.
317:
318: Let us introduce the posterior probability processes
319: \begin{align*}
320: \Pi_n^{(0)} &\triangleq
321: \P\{\theta>n\,|\,\F_n\}\quad\text{and}\quad%\\
322: \Pi_n^{(i)} \triangleq \P\{\theta\leq n,\mu = i\,|\,\F_n\}
323: \end{align*}
324: for $i\in \Mh, n\geq 0$. Having observed the first $n$
325: observations, $\Pi_n^{(0)}$ is the posterior probability that the
326: change \emph{has not} yet occurred at or before stage $n$, while
327: $\Pi_n^{(i)}$ is the posterior joint probability that the change
328: \emph{has} occurred by stage $n$ and that the hypothesis $\mu=i$
329: is correct. The connection of these posterior probabilities to
330: the loss structure for our problem is established in
331: the next proposition. %Proposition \ref{P:BayesRiskInTermsOfPi}.
332:
333: \begin{proposition}\label{P:BayesRiskInTermsOfPi}
334: For every sequential decision strategy $\delta\in\Delta$, the Bayes
335: risk function (\ref{E:BayesRiskUnderP}) can be expressed in terms of
336: the process $\bPi\triangleq\{ \bPi_n= (\Pi_n^{(0)}, \ldots,
337: \Pi_n^{(M)})\}_{n\geq 0}$ as
338: {
339: \begin{align*}
340: R(\delta) &= \E\!\left[ \sum_{n=0}^{\tau-1}c\,(1\!-\!\Pi_n^{(0)})
341: +\II{\tau<\infty}\!\sum_{j=1}^{M}\II{d=j}\!\sum_{i=0}^{M}
342: a_{ij}\Pi_{\tau}^{(i)}\right]\!\!.
343: \end{align*}}
344: \end{proposition}
345:
346: While our original formulation of the Bayes risk function
347: (\ref{E:BayesRiskUnderP}) was in terms of the values of the
348: unobservable random variables $\theta$ and $\mu$, Proposition
349: \ref{P:BayesRiskInTermsOfPi} gives us an equivalent version of the
350: Bayes risk function in terms of the posterior distributions for
351: $\theta$ and $\mu$. This is particularly effective in light of
352: Proposition \ref{P:PiProperties}, which we state with the aid of some
353: additional notation that is referred to throughout the paper. Let
354: \begin{align*}
355: S^M \triangleq
356: \left\{\bpi=(\pi_0,\pi_1,\ldots,\pi_M)\in[0,1]^{M+1}\,\bigm|\,
357: {\textstyle\sum_{i=0}^M}\pi_i = 1 \right\}
358: \end{align*}
359: denote the standard $M$-dimensional probability simplex. Define
360: the mappings $D_i:S^M \times E \mapsto [0,1], i\in \Mh$ and $D:S^M
361: \times E \mapsto [0,1]$ by
362: \begin{align*}
363: %\label{eq:D-mappings}
364: D_{i}(\bpi,x) &\triangleq \left\{
365: \begin{aligned}
366: &(1-p)\pi_0 f_0(x), && i=0\\
367: &(\pi_i+\pi_0\,p\nu_i) f_i(x), && i\in \Mh
368: \end{aligned}
369: \right\}
370: \end{align*}
371: and $D(\bpi,x)\triangleq\sum_{i=0}^{M}D_{i}(\bpi,x)$, and the
372: operator $\T$ on the collection of bounded functions $f:S^M
373: \mapsto\R$ by
374: \begin{align}
375: \label{E:T-operator}
376: (\T f)(\bpi) &\triangleq\!\int_{E}
377: m(dx)\,D(\bpi,x)\,f\!\left({\textstyle\frac{D_0(\bpi,x)}{D(\bpi,x)},\ldots,
378: \frac{D_M(\bpi,x)}{D(\bpi,x)}}\right)
379: \end{align}
380: for every $\bpi\in S^M$.
381:
382: \begin{proposition}\label{P:PiProperties}
383: (a) The process
384: $\bPi^{(0)}\triangleq\{\Pi_n^{(0)},\F_n\}_{n\geq 0}$ is a
385: supermartingale, and $\E\,\Pi_n^{(0)} \leq (1-p)^n$ for every
386: $n\geq 0$.
387:
388: (b) The process
389: $\bPi^{(i)}\triangleq\{\Pi_n^{(i)},\F_n\}_{n\geq 0}$ is a
390: submartingale for every $i\in \Mh$.
391:
392: (c) The process
393: $\bPi=\{(\Pi_n^{(0)},\ldots,\Pi_n^{(M)})\}_{n\geq 0}$ is a Markov
394: process, and
395: \begin{align}
396: \Pi_{n+1}^{(i)} =
397: \frac{D_i(\bPi_n,X_{n+1})}{D(\bPi_n,X_{n+1})},\quad i\in
398: \{0\}\cup\Mh ,\quad n\geq 0,\label{E:Pi-Dynamics}
399: \end{align}
400: with initial state $\Pi_{0}^{(0)} = 1-p_0$ and
401: $\Pi_{0}^{(i)}=p_0\nu_i$, $i\in \Mh.$ %\label{E:Pi0}
402: Moreover, for every bounded function $f:S^M\mapsto\R$ and $n\geq
403: 0$, we have $\E[f(\bPi_{n+1})|\bPi_n] = (\T f)(\bPi_n)$.
404: \end{proposition}
405:
406: \begin{remark}\label{R:PiProperties}
407: Since $\bPi$ is uniformly bounded, the limit
408: $\lim_{n\rightarrow\infty}\bPi_n$ exists by the martingale
409: convergence theorem. Moreover,
410: $\lim_{n\rightarrow\infty}\Pi_n^{(0)}=0$ a.s.\ by Proposition
411: \ref{P:PiProperties}(a) since $p\in(0,1)$.
412: \end{remark}
413:
414: Now, let the functions $h, h_1,\ldots,h_M$ from $S^M$ into $\R_+$ be
415: defined by
416: \begin{align*}
417: h(\bpi)\triangleq \min_{j\in \Mh} h_j(\bpi) \quad \text{and} \quad
418: h_j(\bpi) \triangleq \sum_{i=0}^{M} \pi_i\, a_{ij},\quad j\in \Mh,
419: \end{align*}
420: respectively. Then, we note that for every $\delta=(\tau,d)\in
421: \Delta$, we have
422: \begin{align*}
423: R(\tau, d) &= \E\left[ \sum_{n=0}^{\tau-1}c(1-\Pi_n^{(0)})
424: +\II{\tau<\infty}\sum_{j=1}^{M}\II{d=j}h_j(\Pi_{\tau})\right]\\
425: &\geq \E\left[ \sum_{n=0}^{\tau-1}c(1-\Pi_n^{(0)})
426: +\II{\tau<\infty}h(\Pi_{\tau})\right] = R(\tau,\tilde{d})
427: \end{align*}
428: where we define on the event $\{\tau<\infty\}$ the terminal decision
429: rule $\tilde{d}$ to be any index satisfying
430: $h_{\tilde{d}}(\Pi_{\tau})=h(\Pi_{\tau})$. In other words, an optimal
431: terminal decision depends only upon the value of the $\bPi$ process at
432: the stage in which we stop. Note also that the functions $h$ and
433: $h_1,\ldots,h_M$ are bounded on $S^M$. Therefore, we have the
434: following:
435:
436: \begin{lemma}\label{L:OSP1}
437: The minimum Bayes risk (\ref{E:UDef1}) reduces to the following
438: optimal stopping of the Markov process $\bPi$:
439: \begin{align*}
440: R^* &= \inf_{(\tau,d)\in\Delta}R(\tau,d) =
441: \inf_{(\tau,\tilde{d})\in\Delta}R(\tau,\tilde{d})\\
442: &= \inf_{\tau\in\mathbb{F}}
443: \,\E\left[\sum_{n=0}^{\tau-1}c\,(1-\Pi_n^{(0)})+\II{\tau<\infty}h(\Pi_\tau)\right].
444: \end{align*}
445: \end{lemma}
446:
447: We simplify this formulation further by showing that it is enough to
448: take the infimum over
449: \begin{align}
450: C \triangleq \{\tau\in\mathbb{F}\,|\,\tau<\infty \text{ a.s. and }
451: \E Y_\tau^-<\infty\},\label{E:C}
452: \end{align}
453: where we define
454: \begin{align*}
455: -Y_n \triangleq \sum_{k=0}^{n-1}c\,(1-\Pi_k^{(0)})+h(\Pi_n),\quad
456: n\geq 0%\label{E:Yn}
457: \end{align*}
458: as the minimum \emph{partial risk} obtained by making the best
459: terminal decision on $\{\tau=n\}$. Since $h(\cdot)$ is bounded on
460: $S^M$, the process $\{Y_n, \F_n; n\ge 0\}$ consists of integrable
461: random variables. So the expectation $\E Y_\tau$ exists for every
462: $\tau\in\mathbb{F}$, and our problem becomes
463: \begin{align}
464: -R^*=\sup_{\tau\in\mathbb{F}}\E
465: Y_\tau.\label{E:OptimizationProblemTau}
466: \end{align}
467:
468: Observe that $\E \tau <\infty$ for every $\tau\in C$ because $\infty >
469: (1/c)\E Y_\tau^- \geq \E (\tau-\theta)^+ \geq \E (\tau-\theta) \geq \E
470: \tau -\E \theta \ge \E\tau - (1/p)$. In fact, we have $\E
471: Y_\tau>-\infty \Leftrightarrow \E Y_\tau^-<\infty \Leftrightarrow \E
472: \tau<\infty$ for every $\tau\in\mathbb{F}$. Since
473: $\sup_{\tau\in\mathbb{F}}\E Y_{\tau} \geq \E Y_0 > -h(\Pi_0) >
474: -\infty$, it is enough to consider $\tau\in\mathbb{F}$ such that
475: $\E\tau <\infty$. Namely, (\ref{E:OptimizationProblemTau}) reduces to
476: \begin{align}
477: -R^*=\sup_{\tau\in C} \E Y_\tau.\label{E:OptimizationProblemC}
478: \end{align}
479:
480:
481: \section{Solution via optimal stopping theory}
482: \label{sec:dynamic-programming-solution}
483:
484: In this section we derive an optimal solution for the problem
485: in~\eqref{E:UDef1} by building on the formulation
486: of~\eqref{E:OptimizationProblemC} via the tools of optimal
487: stopping theory, which are detailed in \cite{MR0331675}.
488:
489: \mysubsection{The optimality equation}\label{sec:Derive-Opt-Eqn}
490:
491: We begin by applying the method of truncation with a view of
492: passing to the limit to arrive at the final result. Define for
493: every pair of integers $n, N$ satisfying $0 \le n \le N$ the
494: sub-collections
495: \begin{align*}
496: C_n &\triangleq \{\tau \vee n\,|\,\tau\in C\}\quad\text{and}\quad
497: C_n^N \triangleq \{\tau \wedge N\,|\,\tau\in C_n\}
498: \end{align*}
499: of stopping times in $C$ of \eqref{E:C} and the families of
500: (truncated) optimal stopping problems
501: \begin{align}
502: \label{E:Vn-and-VnN}
503: -V_n \triangleq \sup_{\tau\in C_n}\E Y_\tau
504: \quad\text{and}\quad
505: -V_n^N \triangleq \sup_{\tau\in C_n^N}\E
506: Y_\tau
507: \end{align}
508: corresponding to $(C_n)_{n\geq 0}$ and $(C_n^N)_{0\leq n\leq N}$,
509: respectively. Note that $C\equiv C_0$ and $R^*\equiv V_0$.
510:
511: To investigate these optimal stopping problems, we introduce
512: versions of the \emph{Snell envelope} of $(Y_n)_{n\geq 0}$ (i.e.,
513: the smallest regular supermartingale dominating $(Y_n)_{n\geq
514: 0}$) corresponding to $(C_n)_{n\geq 0}$ and $(C_n^N)_{0\leq n\leq
515: N}$, respectively, defined by
516: \begin{align*}
517: \gamma_n &\triangleq \esup_{\tau\in C_n} \E [Y_\tau\,|\,\F_n]
518: \quad\text{and}\quad \gamma_n^N \triangleq \esup_{\substack{\tau\in
519: C_n^N}} \E [Y_\tau\,|\,\F_n].
520: \end{align*}
521: Then through the following series of lemmas we point out several
522: useful properties of these Snell envelopes. Finally, we extend
523: these results to an arbitrary initial state vector and establish
524: the optimality equation. Note that each of the ensuing
525: (in)equalities between random variables are in the $\P$-almost
526: sure sense.
527:
528: First, these Snell envelopes provide the following alternative
529: expressions for the optimal stopping problems introduced in
530: \eqref{E:Vn-and-VnN} above.
531:
532: \begin{lemma}\label{L:Vn-equal-expected-gamma}
533: For every $N\ge 0$ and $0\le n\le N$, we have $-V_n = \E \gamma_n$
534: and $-V_n^N = \E \gamma_n^N$.
535: \end{lemma}
536:
537: Second, we have the following backward-induction equations.
538:
539: \begin{lemma}\label{L:backward-induction-eqns}
540: We have $\gamma_n = \max\{Y_n, \E [\gamma_{n+1}\,|\,\F_n]\}$ for
541: every $n\ge 0$. For every $N\ge 1$ and $0\le n \le N-1$, we have
542: $\gamma_N^N = Y_N$ and $\gamma_n^N = \max\{Y_n, \E
543: [\gamma_{n+1}^N\,|\,\F_n]\}$.
544: \end{lemma}
545:
546: We also have that these versions of the Snell envelopes coincide in
547: the limit as $N\rightarrow\infty$. That is,
548:
549: \begin{lemma}\label{L:gamma-equals-gamma-prime}
550: For every $n\geq 0$, we have $\gamma_n = \lim_{N\rightarrow\infty}
551: \gamma_n^N$.
552: \end{lemma}
553:
554: Next, recall from \eqref{E:T-operator} and Proposition
555: \ref{P:PiProperties}(c) the operator $\T$ and let us introduce the
556: operator $\M$ on the collection of bounded functions $f:S^M
557: \mapsto \R_+$ defined by
558: \begin{align*}
559: (\M f)(\bpi) \triangleq
560: \min\{h(\bpi),c(1-\pi_0)+(\T f)(\bpi)\},\quad\bpi\in S^M.
561: \end{align*}
562: Observe that $0\leq \M f \leq h$. That is, $\bpi\mapsto(\M f)(\bpi)$
563: is a nonnegative bounded function. Therefore, $\M^2 f\equiv \M(\M f)$
564: is well-defined. If $f$ is nonnegative and bounded, then $\M^n
565: f\equiv \M(\M^{n-1} f)$ is defined for every $n\ge 1$, with
566: $\M^0 f\equiv f$ by definition. Using operator $\M$, we can express
567: $(\gamma_n^N)_{0\leq n\leq N}$ in terms of the process $\bPi$ as
568: stated in the following lemma.
569:
570: \begin{lemma}\label{L:pg-36}
571: For every $N\ge 0$, and $0\le n \le N$, we have
572: $\gamma_n^N = -c\sum_{k=0}^{n-1}(1-\Pi_k^{(0)})-(\M^{N-n}h)(\Pi_n)$.
573: \end{lemma}
574:
575: The next lemma shows how the optimal stopping problems can be
576: rewritten in terms of the operator $\M$. It also conveys the
577: connection between the truncated optimal stopping problems and the
578: initial state $\bPi_0$ of the $\bPi$ process.
579:
580: \begin{lemma}\label{L:pg-38}
581: We have
582: (a) $V_0^N=(\M^N h)(\bPi_0)$ for every $N\geq 0$, and
583: (b) $V_0={\displaystyle\lim_{N\rightarrow\infty}(\M^N
584: h)(\bPi_0)}$.
585: \end{lemma}
586:
587: Observe that since $\bPi_0\in\F_0=\{\varnothing,\Omega\}$, we have
588: $\P\{\bPi_0=\bpi\}=1$ for some $\bpi\in S^M$. On the other hand, for
589: every $\bpi\in S^M$ we can construct a probability space
590: $(\Omega,\F,\P_{\bpi})$ hosting a Markov process $\bPi$ with the same
591: dynamics as in \eqref{E:Pi-Dynamics} and $\P_{\bpi}\{\bPi_0=\bpi\}=1$.
592: Moreover, on such a probability space, the preceding results remain
593: valid. So, let us denote by $\E_{\bpi}$ the expectation with respect
594: to $\P_{\bpi}$ and rewrite \eqref{E:Vn-and-VnN} as
595: \begin{align*}
596: %\label{E:Vnpi-and-VnNpi}
597: -V_n(\bpi) \triangleq \sup_{\tau\in C_n}\E_{\bpi} Y_\tau
598: \quad \text{and} \quad -V_n^N(\bpi) \triangleq \sup_{\tau\in
599: C_n^N}\E_{\bpi} Y_\tau
600: \end{align*}
601: for every $\bpi\in S^M$. Then Lemma \ref{L:pg-38} implies that
602: \begin{align}
603: \label{eq:value-functions}
604: V_0^N\!(\bpi)=(\M^N h)(\bpi)\!\!\quad\text{ and
605: }\quad\!\!V_0(\bpi)=\lim_{N\rightarrow\infty}(\M^N h)(\bpi)
606: \end{align}
607: for every $\bpi\in S^M$. Taking limits as $N\rightarrow\infty$ of
608: both sides in $(\M^{N+1}h)(\bpi) = \M(\M^N h)(\bpi)$ and applying
609: the monotone convergence theorem on the right-hand side yields
610: $V_0(\bpi) = (\M V_0)(\bpi)$. Hence, we have shown the following
611: result.
612:
613: \begin{proposition}[Optimality equation]\label{P:Dyn-prog-eqn}
614: For every $\bpi\in S^M$, %we have
615: \begin{align}
616: V_0(\bpi)\!=\!(\M V_0)(\bpi) \equiv
617: \min\{h(\bpi),c(1\!-\!\pi_0)\!+\!(\T V_0)(\bpi)\}.\label{E:Dyn-prog-eqn}
618: \end{align}
619: \end{proposition}
620:
621: \begin{remark}
622: By solving $V_0(\bpi)$ for any initial state $\bpi\in S^M$, we
623: capture the solution to the original problem since property (c) of
624: Proposition \ref{P:PiProperties} and \eqref{E:OptimizationProblemC}
625: imply that
626: $R^* = V_0(1-p_0,p_0\nu_1,\ldots,p_0\nu_M)$.
627: \end{remark}
628:
629:
630: \mysubsection{Some properties of the value
631: function}\label{sec:V-properties}
632:
633: Now, we reveal some important properties of the value function
634: $V_0(\cdot)$ of (\ref{eq:value-functions}). These results help us
635: to establish an optimal solution for $V_0(\cdot)$, and hence an
636: optimal solution for $R^*$, in the next subsection.
637:
638: \begin{lemma}\label{L:V-concave}
639: If $g:S^M \mapsto \R$ is a bounded concave function, then so is $\T g$.
640: \end{lemma}
641:
642: \begin{proposition}\label{P:V-concave}
643: The mappings $\bpi \mapsto V_0^N(\bpi), N\geq 0$ and $\bpi \mapsto
644: V_0(\bpi)$ are concave.
645: \end{proposition}
646:
647: \begin{proposition}\label{P:V-convergence-rate}
648: For every $N\ge 1$ and $\bpi\in S^M$, we have
649: \begin{align*}
650: V_0(\bpi)\leq V_0^N(\bpi) \leq
651: V_0(\bpi)+\left(\frac{\|h\|^2}{c}+\frac{\|h\|}{p}\right)\frac{1}{N}.
652: \end{align*}
653: Since $\|h\|\triangleq \sup_{\bpi\in S^M} |h(\bpi)|<\infty$,
654: $\lim_{N\rightarrow\infty} \downarrow V_0^N(\bpi) = V_0(\bpi)$
655: uniformly in $\bpi\in S^M$.
656: \end{proposition}
657:
658: \begin{proposition}\label{P:V0N-continuous}
659: For every $N\ge 0$, the function $V_0^N:S^M\mapsto\R_+$ is
660: continuous.
661: \end{proposition}
662:
663: \begin{corollary}\label{C:V-continuous}
664: The function $V_0:S^M \mapsto \R_+$ is continuous.
665: \end{corollary}
666:
667: Note that $S^M$ is a compact subset of $\R^{M+1}$, so while continuity
668: of $V_0(\cdot)$ on the interior of $S^M$ follows from the concavity of
669: $V_0(\cdot)$ by Proposition \ref{L:V-concave}, Corollary
670: \ref{C:V-continuous} establishes continuity on all of $S^M$, including
671: its boundary.
672:
673:
674: \mysubsection{An optimal sequential decision
675: strategy}\label{sec:optimal-soln}
676:
677: Finally, we describe the optimal stopping region in $S^M$ implied
678: by the value function $V_0(\cdot)$, and we present an optimal
679: sequential decision strategy for our problem. Let us define for
680: every $N\ge 0$,
681: \begin{align*}
682: \Gamma_N &\triangleq \{\bpi\in S^M\,|\, V_0^N(\bpi)=h(\bpi)\},\\
683: \Gamma_N^{(j)} &\triangleq \Gamma_N \cap \{\bpi\in S^M\,|\,
684: h(\bpi)=h_j(\bpi)\}, \; j\in \Mh, \\
685: \Gamma &\triangleq \{\bpi\in S^M\,|\, V_0(\bpi)=h(\bpi)\},\\
686: \Gamma^{(j)} &\triangleq \Gamma \cap \{\bpi\in S^M\,|\,
687: h(\bpi)=h_j(\bpi)\}, \; j\in \Mh.
688: \end{align*}
689: For each $j\in \{0\}\cup \Mh$, let $\e_j\in S^M$ denote the unit
690: vector consisting of zero in every component except for the $j$th
691: component, which is equal to one. Note that
692: $\e_0,\ldots,\e_M$ are the extreme points of the closed
693: convex set $S^M$, and any vector $\bpi=(\pi_0,\ldots,\pi_M)\in S^M$
694: can be expressed in terms of $\e_0,\ldots,\e_M$ as $\bpi =
695: \sum_{j=0}^{M}\pi_j\e_j$.
696:
697: \begin{theorem}\label{T:Gamma-decreasing-subsets}
698: For every $j\in \Mh$, $(\Gamma_N^{(j)})_{N\geq 0}$ is a decreasing
699: sequence of non-empty, closed, convex subsets of $S^M$. Moreover,
700: \begin{gather*}
701: \Gamma_0^{(j)} \supseteq \Gamma_1^{(j)} \supseteq \cdots \supseteq
702: \Gamma^{(j)},\\
703: \Gamma^{(j)}\supseteq \left\{\bpi\in S^M
704: \,|\,h_j(\bpi)\leq\min\{h(\bpi),c(1-\pi_0)\}\right\} \ni
705: \e_j,\\
706: \Gamma = \bigcap_{N=1}^{\infty}\Gamma_N =
707: \bigcup_{j=1}^{M}\Gamma^{(j)},\quad\text{and}\quad
708: \Gamma^{(j)}=\bigcap_{N=1}^{\infty}\Gamma_N^{(j)},\quad
709: j\in \Mh.
710: \end{gather*}
711: Furthermore, $S^M = \Gamma_0 \supseteq \Gamma_1 \supseteq \cdots
712: \supseteq \Gamma \supsetneqq \{\e_1,\ldots,\e_M\}$.
713: \end{theorem}
714:
715: \begin{lemma}\label{L:gamma-n-V}
716: For every $n\geq 0$, we have $\gamma_n =
717: -c\sum_{k=0}^{n-1}(1-\Pi_k^{(0)})-V_0(\Pi_n).$
718: \end{lemma}
719:
720: \begin{theorem}\label{T:sigma-properties}
721: Let $\sigma \triangleq \inf\{n\geq 0 \,|\, \bPi_n \in\Gamma\}$.
722: (a) The stopped process $\{\gamma_{n \wedge\sigma}, \F_n;
723: n\geq 0\}$ is a martingale.
724:
725: (b) The random variable $\sigma$ is an optimal stopping time
726: for $V_0$, and
727:
728: (c) $\E\,\sigma<\infty$.
729: \end{theorem}
730:
731: Therefore, the pair $(\sigma, d^*)$ is an optimal sequential
732: decision strategy for \eqref{E:UDef1}, where the optimal stopping
733: rule $\sigma$ is given by Theorem~\ref{T:sigma-properties}, and,
734: as in the proof of Lemma~\ref{L:OSP1}, the optimal terminal
735: decision rule $d^*$ is given by
736: \begin{align*}
737: d^* = j\! \quad\! \text{ on the event}\! \quad\! \{\sigma=n, \bPi_n\in
738: \Gamma^{(j)}\}\! \quad\! \text{ for every } n\geq 0.
739: \end{align*}
740: Accordingly, the set $\Gamma$ is called the \emph{stopping region}
741: implied by $V_0(\cdot)$, and
742: Theorem~\ref{T:Gamma-decreasing-subsets} reveals its basic
743: structure. We demonstrate the use of these results in the
744: numerical examples of Sec.\ \ref{sec:Special-cases}.
745:
746: Note that we can take a similar approach to prove that the
747: stopping rules $\sigma_N\triangleq\inf\{n\geq 0\,|\, \bPi_n \in
748: \Gamma_{N-n}\}, N\geq 0$ are optimal for the truncated problems
749: $V_0^N(\cdot), N\geq 0$ in (\ref{eq:value-functions}). Thus, for
750: each $N\geq 0$, the set $\Gamma_{N}$ is called the stopping region
751: for $V_0^N(\cdot)$: it is optimal to terminate the experiments in
752: $\Gamma_N$ if $N$ stages are left before truncation.
753:
754:
755:
756: \section{Special cases and examples}\label{sec:Special-cases}
757:
758: \mysubsection{A.\ N.\ Shiryaev's sequential change detection
759: problem}
760:
761: Set $a_{0j}=1$ for $j\in \Mh$ and $a_{ij}=0$ for $i,j\in \Mh$, then
762: the Bayes risk function \eqref{E:BayesRiskUnderP} becomes
763: $R(\delta) = \P\{\tau<\theta\} + c\,\E[(\tau-\theta)^+]$.
764: This is the Bayes risk studied by Shiryaev
765: \cite{MR0155708,MR0468067} to solve the sequential change
766: detection problem.
767:
768: \mysubsection{Sequential multi-hypothesis testing}
769:
770: Set $p_0=1$, then $\theta = 0$ a.s.\ and thus the Bayes risk
771: function \eqref{E:BayesRiskUnderP} becomes
772: $R(\delta) = \E[c\tau + a_{\mu d}\II{\tau<\infty}]$.
773: This gives the sequential multi-hypothesis testing problem studied
774: by Wald and Wolfowitz \cite{MR0034005} and Arrow, Blackwell, and
775: Girshick \cite{MR0032173}; see also \cite{MR597146}.
776:
777:
778: \mysubsection{Two alternatives after the change}
779:
780: In this subsection we consider the special case $M=2$ in which we
781: have only two possible change distributions, $f_1(\cdot)$ and
782: $f_2(\cdot)$. We describe a graphical representation of the
783: stopping and continuation regions for an arbitrary instance of the
784: special case $M=2$. Then we use this representation to illustrate
785: geometrical properties of the optimal method (Sec.\
786: \ref{sec:dynamic-programming-solution}.\ref{sec:optimal-soln}) via
787: model instances for certain choices of the model parameters $p_0$,
788: $p$, $\nu_1$, $\nu_2$, $f_0(\cdot)$, $f_1(\cdot)$, $f_2(\cdot)$,
789: $a_{01}$, $a_{02}$, $a_{12}$, $a_{21}$, and $c$.
790:
791:
792:
793: Let the linear mapping $L:\R^3\mapsto\R^2$ be defined by
794: $L(\pi_0,\pi_1,\pi_2)\triangleq(\tfrac{2}{\sqrt{3}}\pi_1
795: +\tfrac{1}{\sqrt{3}}\pi_2,\pi_2)$. Since $\pi_0=1-\pi_1-\pi_2$ for
796: every $\bpi=(\pi_0,\pi_1,\pi_2)\in S^2\subset\R^3$, we can recover
797: the preimage $\bpi$ of any point $L(\bpi)\in L(S^2)\subset\R^2$.
798: For every point $\bpi=(\pi_0,\pi_1,\pi_2)\in S^2$, the coordinate
799: $\pi_i$ is given by the Euclidean distance from the image point
800: $L(\bpi)$ to the edge of the image triangle $L(S^2)$ that is
801: \emph{opposite} the image point $L(\e_i)$, for each $i=0,1,2$. For
802: example, the distance from the image point $L(\bpi)$ to the edge
803: of the image triangle opposite the lower-left-hand corner
804: $L(1,0,0)=(0,0)$ is the value of the preimage coordinate $\pi_0$.
805: See Fig.\ \ref{F:S2-to-2D}.
806:
807: \ifpdf
808: \myFigure{3.5in}{imgS2to2D.pdf}{Linear mapping $L$ of the
809: standard two-dimensional probability simplex $S^2$ from the
810: positive orthant of $\R^3$ into the positive quadrant of
811: $\R^2$.}{F:S2-to-2D}
812: \fi
813:
814: Therefore, we can work with the mappings $L(\Gamma)$ and
815: $L(S^2\setminus\Gamma)$ of the stopping region $\Gamma$ and the
816: continuation region $S^2\setminus\Gamma$, respectively.
817: Accordingly, we depict the decision region for each instance in
818: this subsection using the two-dimensional representation as in the
819: right-hand-side of Fig.\ \ref{F:S2-to-2D} and we drop the
820: $L(\cdot)$ notation when labeling various parts of each figure to
821: emphasize their source in $S^2$.
822:
823: Each of the examples in this section have the following model
824: parameters in common:
825: \begin{gather*}
826: p_0=\tfrac{1}{50},\quad p=\tfrac{1}{20},\quad
827: \nu_1=\nu_2=\tfrac{1}{2},\\
828: f_0\!=\!\left(\tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4},
829: \tfrac{1}{4}\right)\!,
830: f_1\!=\!\left(\tfrac{4}{10}, \tfrac{3}{10}, \tfrac{2}{10},
831: \tfrac{1}{10}\right)\!,
832: f_2\!=\!\left(\tfrac{1}{10}, \tfrac{2}{10}, \tfrac{3}{10},
833: \tfrac{4}{10}\right)\!.
834: \end{gather*}
835: We vary the delay cost and false alarm/identification costs to
836: illustrate certain geometrical properties of the continuation and
837: stopping regions. See Figs.\ \ref{F:2D1}, \ref{F:2D2}, and
838: \ref{F:2D3}.
839:
840: \ifpdf
841: \myFigure{3.5in}{img2D1.pdf}{Illustration of
842: \emph{connected} stopping regions and the effects of false-alarm
843: costs. (a) and (b): $a_{12}=a_{21}=3,\,c=1$. (a):
844: $a_{01}=a_{02}=10$. (b): $a_{01}=a_{02}=50$.}{F:2D1}
845: \fi
846:
847: \ifpdf
848: \myFigure{3.5in}{img2D2.pdf}{Illustration of
849: \emph{disconnected} stopping regions and the effects of asymmetric
850: false-identification costs. (a) and (b): $a_{01}=a_{02}=10,\,c=1$.
851: (a): $a_{12}=a_{21}=10$. (b): $a_{12}=16,a_{21}=4$.}{F:2D2}
852: \fi
853:
854: \ifpdf
855: \myFigure{3.5in}{img2D3.pdf}{Illustration of a
856: \emph{disconnected} continuation region and the effects of
857: variation in the delay cost. (a) and (b):
858: $a_{01}=14,a_{02}=20,a_{12}=a_{21}=8$. (a): $c=1$. (b):
859: $c=2$.}{F:2D3}
860: \fi
861:
862: These figures have certain features in common. On each subfigure
863: there is a dashed line representing those states $\bpi\in S^2$ at
864: which $h_1(\bpi)=h_2(\bpi)$. Also, each subfigure shows a sample
865: path of $(\bPi_n)_{n=0}^{\sigma}$ and the realizations of $\theta$
866: and $\mu$ for the sample. The shaded area, including its solid
867: boundary, represents the optimal stopping region, while the
868: unshaded area represents the continuation region.
869:
870: Specifically, these figures show instances in which the $M=2$
871: convex subsets comprising the optimal stopping region are
872: connected (Fig.\ \ref{F:2D1}) and instances in which they are not
873: (Figs.\ \ref{F:2D2} and \ref{F:2D3}(a)). Fig.\ \ref{F:2D3}(b)
874: shows an instance in which the continuation region is
875: disconnected.
876:
877: An implementation of the optimal strategy as described in Sec.\
878: \ref{sec:dynamic-programming-solution}.\ref{sec:optimal-soln} is
879: as follows: Initialize the statistic $\bPi=(\bPi_n)_{n\geq 0}$ by
880: setting $\bPi_0=(1-p_0,p_0\nu_1,p_0\nu_2)$ as in part (c) of
881: Proposition \ref{P:PiProperties}. Use the dynamics of
882: \eqref{E:Pi-Dynamics} to update the statistic $\bPi_n$ as each
883: observation $X_n$ is realized. Stop taking observations when the
884: statistic $\bPi_n$ enters the stopping region
885: $\Gamma=\Gamma^{(1)}\cup\Gamma^{(2)}$ for the first time, possibly
886: before the first observation is taken (i.e., $n=0$). The optimal
887: terminal decision is based upon whether the statistic $\bPi_n$ is
888: in $\Gamma^{(1)}$ or $\Gamma^{(2)}$ upon stopping. Each of the
889: sample paths in Figs.\ \ref{F:2D1}, \ref{F:2D2}, and \ref{F:2D3}
890: were generated via this algorithm. As Fig.\ \ref{F:2D1} shows, the
891: sets $\Gamma^{(1)}$ and $\Gamma^{(2)}$ can intersect on their
892: boundaries and so it is possible to stop in their intersection. In
893: this case, either of the decisions $d=1$ or $d=2$ is optimal.
894:
895: We use value iteration of the optimality
896: equation~\eqref{E:Dyn-prog-eqn} over a fine discretization of
897: $S^2$ to compute $V_0(\cdot)$ and generate the decision region for
898: each subfigure. The resulting discretized decision region is
899: mapped into the plane via $L$. See~\cite[Ch.\ 3]{MR2182753} for
900: techniques of computing the value function via the optimality
901: equation such as value iteration.
902:
903:
904: \mysubsection{Three alternatives after the change}
905:
906:
907: In this subsection we consider the special case $M=3$ in which we
908: have three possible change distributions, $f_1(\cdot)$,
909: $f_2(\cdot)$, and $f_3(\cdot)$. Here, the continuation and
910: stopping regions are subsets of $S^3\subset\R^4$. Similar to the
911: two-alternatives case, we introduce the mapping of
912: $S^3\subset\R^4$ into $\R^3$ via
913: $(\pi_0,\pi_1,\pi_2,\pi_3)\mapsto$
914: \begin{align*}
915: {\textstyle
916: \left(\sqrt{\tfrac{3}{2}}\pi_1
917: +\tfrac{1}{2}\sqrt{\tfrac{3}{2}}\pi_2
918: +\tfrac{1}{2}\sqrt{\tfrac{3}{2}}\pi_3,
919: \tfrac{3}{2}\sqrt{\tfrac{1}{2}}\pi_2 +
920: \tfrac{1}{2}\sqrt{\tfrac{1}{2}}\pi_3, \pi_3\right)}.
921: \end{align*}
922: Then we use this representation---actually a rotation of it---to
923: illustrate in Fig.\ \ref{F:3D} an instance with the following
924: model parameters:
925: \begin{gather*}
926: p_0=\tfrac{1}{50},\quad p=\tfrac{1}{20},\quad
927: \nu_1=\nu_2=\nu_3=\tfrac{1}{3}, \\
928: f_0=\left(\tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4},
929: \tfrac{1}{4}\right),\quad
930: f_1=\left(\tfrac{4}{10}, \tfrac{3}{10}, \tfrac{2}{10},
931: \tfrac{1}{10}\right),\\
932: f_2=\left(\tfrac{1}{10}, \tfrac{2}{10}, \tfrac{3}{10},
933: \tfrac{4}{10}\right),\quad
934: f_3=\left(\tfrac{3}{10}, \tfrac{2}{10}, \tfrac{2}{10},
935: \tfrac{3}{10}\right),\\
936: c=1,\quad a_{0j}=40,\quad a_{ij}=20,\quad i,j=1,2,3.
937: \end{gather*}
938:
939:
940: Fig.\ \ref{F:3D} can be interpreted in a manner similar to the
941: figures of the previous subsection. In this case, for every point
942: $\bpi=(\pi_0,\pi_1,\pi_2,\pi_3)\in S^3$, the coordinate $\pi_i$ is
943: given by the (Euclidean) distance from the image point $L(\bpi)$
944: to the face of
945: \begin{minipage}{3.5in} \ifpdf \myFigure{3.5in}{img3D1.pdf}{Illustration of the optimal
946: decision region for an instance of the special case $M=3$. A
947: sample path of the process $\bPi$ is shown for which $\theta=6$
948: and $\mu=3$.}{F:3D} \fi\text{ }\\
949: \end{minipage}
950: the image tetrahedron $L(S^3)$ that is opposite the image corner
951: $L(\e_i)$, for each $i=0,1,2,3$.
952:
953: \vskip 15pt
954:
955: \bibliography{82a}
956: \bibliographystyle{unsrt}
957: }
958: \end{multicols}
959:
960: \end{document}
961: