1: \documentstyle[psfig]{llncs}

2: \begin{document}

3: \begin{center}

4: {\large {\bf A Numerical Example on the\\ Principles of Stochastic

5: Discrimination}} 

6: \vspace{0.1in}\\

7: Tin Kam Ho \vspace{0.025in}\\

8: Bell Laboratories, Lucent Technologies \vspace{0.05in}\\

9: July 15, 2003 \vspace{0.5in}\\

10: {\bf Abstract}

11: \vspace{0.1in}\\

12: \parbox{4.5in}{

13: Studies on ensemble methods for classification suffer from the

14: difficulty of modeling the complementary strengths of the 

15: components.  Kleinberg's theory of stochastic discrimination (SD)

16: addresses this rigorously via mathematical notions of enrichment, 

17: uniformity, and projectability of an ensemble.  We explain these concepts

18: via a very simple numerical example that captures the 

19: basic principles of the SD theory and method.  We focus on a fundamental

20: symmetry in point set covering that is the key observation

21: leading to the foundation of the theory.  We believe a better

22: understanding of the SD method will lead to developments of

23: better tools for analyzing other ensemble methods.

24: }

25: \end{center}

26: 

27: \section{Introduction}

28: 

29: Methods for classifier combination, or ensemble learning, can be divided

30: into two categories: 1) {\em decision optimization} methods that try

31: to obtain {\em consensus} among a {\em given} set of classifiers to

32: make the best decision; 2) {\em coverage optimization} methods that

33: try to {\em create} a set of classifiers that work best with a {\em

34: fixed} decision combination function to cover all possible cases.

35: 

36: Decision optimization methods rely on the assumption that the given

37: set of classifiers, typically of a small size, contain sufficient

38: expert knowledge about the application domain, and each of them 

39: excels in a subset of all possible input.  A decision combination

40: function is chosen or trained to exploit the individual strengths

41: while avoiding their weaknesses.  Popular combination functions

42: include majority/plurality votes\cite{kn:lam97}, sum/product

43: rules\cite{kn:kittler98}, rank/confidence

44: score combination\cite{kn:ho94}, and probabilistic methods\cite{kn:huang95}.

45: While numerous successful applications of these methods have been

46: reported,  the joint capability of the classifiers sets an intrinsic

47: limitation on decision optimization that the combination functions

48: cannot overcome.  A challenge in this approach is to find out the

49: ``blind spots'' of the ensemble and to obtain a classifier that covers

50: them.

51: 

52: Coverage optimization methods use an automatic and systematic

53: mechanism to generate new classifiers with the hope of covering all

54: possible cases. A fixed, typically simple, function is used for

55: decision combination.  This can take the form of training set

56: subsampling, such as stacking \cite{kn:wolpert92},

57: bagging\cite{kn:breiman96}, and boosting\cite{kn:freund96}, feature

58: subspace projection\cite{kn:ho98}, superclass/subclass

59: decomposition\cite{kn:dietterich95b}, or other forms of random perturbation of

60: the classifier training procedures \cite{kn:hansen90}.

61: Open questions in these methods are

62: 1) how many classifiers are enough?  2) what kind of differences

63: among the component classifiers yields the best combined accuracy?

64: 3) how much limitation is set by the form of the component

65: classifiers?

66: 

67: Apparently both categories of ensemble methods run into some dilemma.

68: Should the component classifiers be weakened in order to achieve a

69: stronger whole?  Should some accuracy be sacrificed for the known

70: samples to obtain better generalization for the unseen cases?  Do we

71: seek agreement, or differences among the component classifiers?

72: 

73: A central difficulty in studying the performance of these ensembles

74: is how to model the complementary strengths among the classifiers.  Many

75: proofs rely on an assumption of statistical independence of

76: component classifiers' decisions.  But rarely is there any attempt to

77: match this assumption with observations of the decisions.  Often,

78: global estimates of the component classifiers' accuracies are used in

79: their selection, while in an ensemble what matter more are the local

80: estimates, plus the relationship between the local accuracy

81: estimates on samples that are close neighbors in the feature

82: space.\footnote{there is more discussion on these difficulties in a

83: recent review\cite{kn:ho02}.} 

84: 

85: Deeper investigation of these issues leads back to three major concerns

86: in choosing classifiers: discriminative power, use of complementary

87: information, and generalization power.  A complete theory on ensembles

88: must address these three issues simultaneously.  Many current

89: theories rely,  either explicitly or implicitly, on ideal assumptions

90: on one or two of these issues, or have them omitted entirely,  and are

91: therefore incomplete.

92: 

93: Kleinberg's theory and method of stochastic discrimination

94: (SD)\cite{kn:kleinberg90} \cite{kn:kleinberg96} is the first attempt

95: to explicitly address these issues simultaneously from a mathematical

96: point of view.   In this theory, rigorous notions are made for

97: discriminative power, complementary information, and generalization

98: power of an ensemble.  A fundamental symmetry is observed between

99: the probability of a fixed model covering a point in a given set and

100: the probability of a fixed point being covered by a model in a given

101: ensemble.  The theory establishes that,  these three conditions

102: are sufficient for an ensemble to converge, with increases in its size, to

103: the most accurate classifier for the application.

104: 

105: Kleinberg's analysis uses a set-theoretic abstraction to remove all the

106: algorithmic details of classifiers, features, and training procedures.

107: It considers only the classifiers' decision regions in the form of point

108: sets, called weak models, in the feature space.  A collection of

109: classifiers is thus just a sample from the power set of the feature

110: space.   If the sample satisfies a uniformity condition, i.e., if its

111: coverage is unbiased for any local region of the feature space, then a

112: symmetry is observed between two probabilities (w.r.t. the feature

113: space and w.r.t. the power set, respectively)  of the same event that a

114: point of a particular class is covered by a component of the sample.

115: Discrimination between classes is achieved by requiring some minimum

116: difference in each component's inclusion of points of different

117: classes,  which is trivial to satisfy.  By way of this symmetry,

118: it is shown that if the sample of weak models is large,  the

119: discriminant function, defined on the coverage of the models on a

120: single point and the class-specific differences within each model,

121: converges to poles distinct by class with diminishing variance.

122: 

123: We believe that this symmetry is the key to the discussions on classifier

124: combination.  However,  since the theory was developed from a fresh,

125: original, and independent perspective on the problem of learning,  there

126: have not been many direct links made to the existing theories.

127: As the concepts are new, the claims are high, the published

128: algorithms appear simple,  and the details of more sophisticated

129: implementations are not known, the method has been poorly understood

130: and is sometimes referred to as mysterious. 

131: 

132: It is the goal of this lecture to illustrate the basic concepts

133: in this theory and remove the apparent mystery.

134: We present the principles of stochastic discrimination

135: with a very simple numerical example.  The example is so chosen

136: that all computations can be easily traced step-by-step by hand or with

137: very simple programs.  We use Kleinberg's notation wherever

138: possible to make it easier for the interested readers to follow up on

139: the full theory in the original papers.

140: The emphasis in this note is on explaining the concepts of

141: uniformity and enrichment, and the behavior of the discriminant when

142: both conditions are fulfilled.  For the details of the mathematical

143: theory and outlines of practical algorithms, please refer to the original

144: publications\cite{kn:kleinberg90}\cite{kn:kleinberg96}\cite{kn:kleinberg00}

145: \cite{kn:kleinberg00b}.

146: 

147: \section{Symmetry of Probabilities Induced by Uniform Space Covering}

148: 

149: The SD method is based on a fundamental symmetry in point set covering.

150: To illustrate this symmetry,  we begin with a simple observation.

151: Consider a set $S=\{a, b, c\}$ 

152: and all the subsets with two elements $s_1=\{a,b\}$, $s_2=\{a,c\}$,

153: and $s_3=\{b,c\}$.  By our choice,  each of these subsets has

154: captured $2/3$ of the elements of $S$.  We call this ratio $r$.  Let

155: us now look at each member of $S$, and check how many of these three

156: subsets have included that member.  For example, $a$ is in two of

157: them, so we say $a$ is captured by $2/3$ of these subsets.  We will

158: obtain the same value $2/3$ for all elements of $S$.  This value is

159: the same as $r$. This is a consequence of the fact that we have used

160: all such 2-member subsets and we have not biased this collection

161: towards any element of $S$.  With this observation, we begin a larger example.

162: 

163: Consider a set of 10 points in a one-dimensional feature space $F$.

164: Let this set be called $A$.  Assume that $F$ contains only points in

165: $A$ and nothing else.  Let each point in $A$ be identified as $q_0$,

166: $q_1$, ..., $q_9$ as follows. 

167: \begin{center}

168: \begin{tabular}{cccccccccc}

169: .&.&.&.&.&.&.&.&.&.\\

170: $q_0$ &$q_1$ &$q_2$ &$q_3$ &$q_4$ &$q_5$ &$q_6$ &$q_7$ &$q_8$ &$q_9$\\

171: \end{tabular}

172: \end{center}

173: 

174: Now consider the subsets of $F$.  Let the collection of

175: all such subsets be ${\cal M}$, which is the power set of $F$.

176: We call each member $m$ of ${\cal M}$ a {\em model},  and we restrict our

177: consideration to only those models that contain 5 points in $A$,

178: therefore each model has a size that is 0.5 of the size of $A$.  Let

179: this set of models be called $M_{0.5,A}$.  Some members of $M_{0.5,A}$

180: are as follows. 

181: \begin{center}

182: \begin{tabular}{ccccc}

183: \{$q_0$, &$q_1$, &$q_2$, &$q_3$, &$q_4$ \}\\

184: \{$q_0$, &$q_1$, &$q_2$, &$q_3$, &$q_5$ \}\\

185: \{$q_0$, &$q_1$, &$q_2$, &$q_3$, &$q_6$ \}\\

186: ...&&&&\\

187: \end{tabular}

188: \end{center}

189: 

190: There are $C(10,5) = 252$ members in $M_{0.5,A}$.  Let $M$ be a

191: pseudo-random permutation of members in $M_{0.5,A}$ as listed

192: in Table \addtocounter{table}{+1}\thetable\addtocounter{table}{-1}

193: in the Appendix.  We identify models in this sequence by a single

194: subscript such that $M = m_1, m_2, ..., m_{252}$.

195: We expand a collection $M_t$ by including more and more members of

196: $M_{0.5,A}$ in the order of the sequence $M$ as follows.

197: $M_1=\{m_1\}$, $M_2=\{m_1,m_2\}$, ..., $M_t=\{m_1,m_2,...m_t\}$.

198: 

199: Since each model covers some points in $A$,  for each member $q$ in $A$,

200: we can count the number of models in $M_t$ that include $q$,

201: call this count $N(q,M_t)$, and calculate the ratio of this count over

202: the size of $M_t$, call it $Y(q,M_t)$.  That is, $Y(q,M_t) =

203: Prob_{\cal M} (q\in m | m \in M_t)$.  As $M_t$ expands, this ratio

204: changes and we show these changes for each $q$ in Table

205: \addtocounter{table}{+2}\thetable\addtocounter{table}{-2} in the Appendix.

206: The values of $Y(q,M_t)$ are plotted in 

207: Figure \addtocounter{figure}{1}\thefigure\addtocounter{figure}{-1}.

208: As is clearly visible in the Figure, the values of $Y(q,M_t)$ converge

209: to 0.5 for each $q$.  Also notice that because of the randomization,

210: we have expanded $M_t$ in a way that $M_t$ is not biased towards any

211: particular $q$,  therefore the values of $Y(q,M_t)$ are similar

212: after $M_t$ has acquired a certain size (say, when $t=80$).

213: When $M_t$=$M_{0.5,A}$, every point $q$ is covered by the

214: same number of models in $M_t$, and their values of $Y(q,M_t)$ are

215: identical and is equal to 0.5,  which is the ratio of the size of each

216: $m$ relative to $A$ (recall that we always include 5 points from $A$

217: in each $m$).

218: 

219: Formally, when $t=252$, $M_t = M_{0.5,A}$, from the

220: perspective of a fixed $q$, the probability of it being contained in a

221: model $m$ from $M_t$ is 

222: \[ Prob_{\cal M} (q \in m | m \in M_{0.5,A}) = 0.5. \]

223: We emphasize that this probability is a measure in the space ${\cal M}$

224: by writing the probability as $Prob_{\cal M}$.

225: On the other hand,  by the way each $m$ is constructed, we know that

226: from the perspective of a fixed $m$,

227: \[ Prob_F (q \in m| q \in A) ~=~ 0.5. \]

228: Note that this probability is a measure in the space $F$.

229: We have shown that these two probabilities, w.r.t. two different

230: spaces, have identical values.  In other words,

231: let the membership function of $m$ be $C_m(q)$, i.e., $C_m(q) =

232: 1 ~{\rm iff}~ q \in m$,  the random variables

233: $\lambda q C_m(q)$ and $\lambda m C_m(q)$ have the same probability

234: distribution, when $q$ is restricted to $A$ and $m$ is restricted to

235: $M_{0.5,A}$.  This is because both variables can have values that are either

236: 1 or 0, and they have the value 1 with the same probability (0.5 in this case).

237: This symmetry arises from the fact that the collection of models $M_{0.5,A}$

238: covers the set $A$ uniformly, i.e.,  since we have used all members of

239: $M_{0.5,A}$, each point $q$ have the same chance to be included

240: in one of these models.   If any two points in a set $S$ have

241: the same chance to be included in a collection of models,

242: we say that this collection is $S$-uniform.  It can be shown, by a simple

243: counting argument, that uniformity leads to the symmetry of 

244: $Prob_{\cal M} (q \in m | m \in M_{0.5,A})$ and

245: $Prob_F (q \in m| q \in A)$, and hence

246: distributions of $\lambda q C_m(q)$ and $\lambda m C_m(q)$.

247: 

248: The observation and utilization of

249: this duality are central to the theory of stochastic discrimination.

250: A critical point of the SD method is to enforce such a uniform

251: cover on a set of points.  That is, to construct a collection of

252: models in a balanced way so that the uniformity (hence the duality) is

253: achieved without exhausting all possible models from the space.

254: 

255: \begin{figure}[h]

256: \centering

257: \begin{tabular}{c}

258: \psfig{file=figs/Yinsen.ps,height=3in,clip=}\\

259: \end{tabular}

260: \caption{Plot of $Y(q,M_t)$ versus $t$.  Each line represents the

261: trace of $Y(q,M_t)$ for a particular $q$ as $M_t$ expands.}

262: \end{figure}

263: 

264: \section{Two-Class Discrimination}

265: 

266: Let us now label each point $q$ in $A$ by one of two classes $c_1$ 

267: (marked by ``x'') and $c_2$ (marked by ``o'') as follows.

268: 

269: \begin{center}

270: \begin{tabular}{cccccccccc}

271: x&x&x&o&o&o&o&x&x&o\\

272: $q_0$ &$q_1$ &$q_2$ &$q_3$ &$q_4$ &$q_5$ &$q_6$ &$q_7$ &$q_8$ &$q_9$\\

273: \end{tabular}

274: \end{center}

275: 

276: This gives a training set $TR_i$ for each class $c_i$.  In

277: particular,  

278: \[ TR_1 = \{q_0,q_1,q_2,q_7,q_8\},\]

279: and

280: \[ TR_2 = \{q_3,q_4,q_5,q_6,q_9\}. \]

281: How can we build a classifier for $c_1$ and $c_2$ using models from

282: $M_{0.5,A}$?

283: First, we evaluate each model $m$ by how well it has captured the

284: members of each class.  Define ratings $r_i$ ($i=1,2$) for each $m$ as

285: \[ r_i(m) = Prob_F (q \in m | q \in TR_i). \]

286: For example, consider model $m_1 = \{q_3,q_5,q_6,q_8,q_9\}$, where $q_8$ is

287: in $TR_1$ and the rest are in $TR_2$.  $TR_1$ has 5 members and 1 is

288: in $m_1$, therefore $r_1(m_1) = 1/5 = 0.2$.  $TR_2$ has (incidentally,

289: also) 5 members and 4 of them are in $m_1$, therefore $r_2(m_1) = 4/5

290: = 0.8$.  Thus these ratings represent the quality of the models as a 

291: description of each class.  A model with a rating $1.0$ for a class is

292: a perfect model for that class.  We call the difference between $r_1$

293: and $r_2$ the {\em degree of enrichment} of $m$ with respect to classes

294: $(1,2)$, i.e., $d_{12} = r_1 - r_2$.  A model $m$ is {\em enriched} if

295: $d_{12} \neq 0$.  Now we define, for all enriched models $m$,

296: 

297: \[ X_{12}(q,m) = \frac{C_m(q) - r_2(m)}{r_1(m) - r_2(m)}, \]

298: 

299: \noindent and let $X_{12}(q,m)$ be $0$ if $d_{12}(m) = 0$.

300: For a given $m$, $r_1$ and $r_2$ are fixed, and the value of $X(q,m)$

301: for each $q$ in $A$ can have one of two values depending on whether

302: $q$ is in $m$.  For example, for $m_1$, $r_1 = 0.2$ and $r_2 = 0.8$,

303: so $X(q,m) = -1/3$ for points $q_3,q_5,q_6,q_8,q_9$, and 

304: $X(q,m) = 4/3$ for points $q_0,q_1,q_2,q_4,q_7$.

305: Next, for each set $M_t = \{m_1,m_2,...,m_t\}$, we define a

306: discriminant 

307: 

308: \[ Y_{12}(q,M_t) = \frac{1}{t} \sum_{k=1}^{t} X_{12}(q,m_k). \]

309: 

310: As the set $M_t$ expands,  the value of $Y_{12}$ changes for each $q$.

311: We show, 

312: in Table \addtocounter{table}{+3}\thetable\addtocounter{table}{-3}

313: in the Appendix, the values of $Y_{12}$ for each $M_t$ and each $q$,

314: and for each new member $m_t$ of $M_t$, $r_1,r_2$, and the two values

315: of $X_{12}$.  The values of $Y_{12}$ for each $q$ are plotted in

316: Figure \addtocounter{figure}{1}\thefigure\addtocounter{figure}{-1}.

317: 

318: \begin{figure}[h]

319: \centering

320: \begin{tabular}{c}

321: \psfig{file=figs/Y01.ps,height=3in,clip=}\\

322: \end{tabular}

323: \caption{Plot of $Y_{12}(q,M_t)$ versus $t$.  Each line represents the

324: trace of $Y_{12}(q,M_t)$ for a particular $q$ as $M_t$ expands.}

325: \end{figure}

326: 

327: In Figure \thefigure \, we see two separate trends.  All those

328: points that belong to class $c_1$ have their $Y_{12}$ values

329: converging to 1.0, and all those in $c_2$ converging to 0.0.  Thus

330: $Y_{12}$ can be used with a threshold 

331: to classify an arbitrary point $q$.  We can assign $q$ to class $c_1$

332: if $Y_{12}(q,M_t) > 0.5$,  and to class $c_2$ if $Y_{12}(q, M_t) <

333: 0.5$, and remain undecided when $Y_{12}(q,M_t) = 0.5$.  Observe that

334: this classifier is fairly accurate far before $M_t$ has expanded to

335: the full set $M_{0.5,A}$.  We can also change the two poles of $Y_{12}$ to 1.0

336: and -1.0 respectively by simply rescaling and shifting $X_{12}$:

337: 

338: \[ X_{12}(q,m) = 2(\frac{C_m(q) - r_2(m)}{r_1(m) - r_2(m)}) - 1. \]

339: 

340: How did this separation of trends happen?  Let us now take a closer look at the

341: models in each $M_t$ and see how many of them cover each point $q$.

342: For a given $M_t$, among its members, there can be different values of

343: $r_1$ and $r_2$.  But because of our choices of the

344: sizes of $TR_1$, $TR_2$, and $m$,  we have only a small set of

345: distinct values that $r_1$ and $r_2$ can have.  Namely, since each

346: model has 5 points, there are only six possibilities as follows. 

347: 

348: \begin{center}

349: \begin{tabular}{l cccccc}

350: no. of points from $TR_1$ & 0   & 1   & 2   & 3   & 4   & 5   \\

351: no. of points from $TR_2$ & 5   & 4   & 3   & 2   & 1   & 0   \\

352: $r_1$                     & 0.0 & 0.2 & 0.4 & 0.6 & 0.8 & 1.0 \\

353: $r_2$                     & 1.0 & 0.8 & 0.6 & 0.4 & 0.2 & 0.0 \\

354: \end{tabular}

355: \end{center}

356: 

357: Note that in a general setting $r_1$ and $r_2$ do not have to sum up

358: to 1.  If we included models of a larger size, say, one with 10

359: points,  we can have both $r_1$ and $r_2$ equal to 1.0.  We have

360: simplified matters by using models of a fixed size and training sets

361: of the same size.  According to the values of $r_1$ and $r_2$, in this

362: case we have only 6 different kinds of models.

363: 

364: Now we take a detailed look at the coverage of each point $q$ by

365: each kind of models, i.e., models of a particular rating (quality) for

366: each class.   Let us count how many of the models of each value of

367: $r_1$ and $r_2$ cover each point $q$, and call this $N_{M_t,r_1,TR_1}(q)$ and

368: $N_{M_t,r_2,TR_2}(q)$ respectively.  We can normalize this count by 

369: the number of models having each value of $r_1$ or $r_2$,

370: and obtain a ratio $f_{M_t,r_1,TR_1}(q)$ and 

371: $f_{M_t,r_2,TR_2}(q)$ respectively.   Thus, for each point $q$, we

372: have ``a profile of coverage'' by models of each value of ratings

373: $r_1$ and $r_2$ that is described by these ratios. For example, point $q_0$ at

374: $t=10$ is only covered by 5 models ($m_2,m_3,m_5,m_8,m_{10}$) in $M_{10}$, and

375: from Table 3 we know that $M_{10}$ has various numbers of models in

376: each rating as summarized in the following table.

377: 

378: \begin{center}

379: \begin{tabular}{l r r r r r r}

380: $r_1 $                               & 0.0 & 0.2 & 0.4 & 0.6 & 0.8 & 1.0 \\

381: no. of models in $M_{10}$ with $r_1$ &   0 &   2 &   2 &   4 &   2 &  0 \\

382: $N_{M_{10},r_1,TR_1}(q_0)$           &   0 &   0 &   0 &   3 &   2 &  0 \\

383: $f_{M_{10},r_1,TR_1}(q_0)$           &   0 &   0 &   0 & 0.75 & 1.0 & 0 \\

384: \\

385: $r_2 $                               & 0.0 & 0.2 & 0.4 & 0.6 & 0.8 & 1.0 \\

386: no. of models in $M_{10}$ with $r_2$ &   0 &   2 &   4 &   2 &   2 & 0 \\

387: $N_{M_{10},r_2,TR_2}(q_0)$           &   0 &   2 &   3 &   0 &   0 & 0\\

388: $f_{M_{10},r_2,TR_2}(q_0)$           &   0 &  1.0 & 0.75 & 0 &   0 & 0\\

389: \end{tabular}

390: \end{center}

391: 

392: We show such profiles for each point $q$ and each set $M_t$ in

393: Figure \addtocounter{figure}{1}\thefigure\addtocounter{figure}{-1}

394: (as a function of $r_1$) and

395: Figure \addtocounter{figure}{2}\thefigure\addtocounter{figure}{-2}

396: (as a function of $r_2$) respectively.

397: 

398: Observe that as $t$ increases,  the profiles of coverage for each point $q$

399: converge to two distinct patterns.  In Figure 3, the profiles for

400: points in $TR_1$ converge to a diagonal $f_{M_t,r_1,TR_1} = r_1$, and in

401: Figure 4, those for points in $TR_2$ also converge to a diagonal

402: $f_{M_t,r_2,TR_2} = r_2$.  That is, when $M_t = M_{0.5,A}$,  we have

403: for all $q$ in $TR_1$ and for all $r_1$,

404: $Prob_{\cal M} (q \in m | m \in M_{r_1,TR_1}) = r_1$, and 

405: for all $q$ in $TR_2$ and for all $r_2$,

406: $Prob_{\cal M} (q \in m | m \in M_{r_2,TR_2}) = r_2$.

407: Thus we have the symmetry in place for both $TR_1$ and $TR_2$.

408: This is a consequence of $M_t$ being both $TR_1$-uniform and $TR_2$-uniform. 

409: 

410: The discriminant $Y_{12}(q,M_t)$ is a summation over all models $m$ in

411: $M_t$, which can be decomposed into the sums of terms corresponding to

412: different ratings $r_i$ for either $i=1$ or $i=2$.  To understand what

413: happens with the points in $TR_1$, we can decompose their $Y_{12}$ by

414: values of $r_1$.  Assume that there are $t_x$ models in $M_t$ that

415: have $r_1 = x$.  Since we have only 6 distinct values for $x$,

416: $M_t$ is a union of 6 disjoint sets,  and $Y_{12}$ can be decomposed as

417: 

418: \vspace{0.1in}

419: \begin{tabular}{lll}

420: \hspace{-0.3in}

421: $Y_{12}(q,M_t)$ = & $\frac{t_{0.0}}{t} [\frac{1}{t_{0.0}}

422: \sum_{k_{0.0}=1}^{t_{0.0}}  X_{12}(q,m_{k_{0.0}})] ~~+ $

423:                   & $\frac{t_{0.2}}{t} [\frac{1}{t_{0.2}}

424: \sum_{k_{0.2}=1}^{t_{0.2}}  X_{12}(q,m_{k_{0.2}})] ~~+ $\\

425:                   & $\frac{t_{0.4}}{t} [\frac{1}{t_{0.4}}

426: \sum_{k_{0.4}=1}^{t_{0.4}}  X_{12}(q,m_{k_{0.4}})] ~~+ $

427:                   & $\frac{t_{0.6}}{t} [\frac{1}{t_{0.6}}

428: \sum_{k_{0.6}=1}^{t_{0.6}}  X_{12}(q,m_{k_{0.6}})] ~~+ $\\

429:                   & $\frac{t_{0.8}}{t} [\frac{1}{t_{0.8}}

430: \sum_{k_{0.8}=1}^{t_{0.8}}  X_{12}(q,m_{k_{0.8}})] ~~+ $

431:                   & $\frac{t_{1.0}}{t} [\frac{1}{t_{1.0}}

432: \sum_{k_{1.0}=1}^{t_{1.0}}  X_{12}(q,m_{k_{1.0}})]$. \\

433: \end{tabular}

434: \vspace{0.1in}

435: 

436: The factor in the square bracket of each term is the expectation of values of

437: $X_{12}$ corresponding to that particular rating $r_1 = x$.  Since

438: $r_1$ is the same for all $m$ contributing to that term,  by our

439: choice of sizes of $TR_1$, $TR_2$, and the models, $r_2$ is also the

440: same for all those $m$ relevant to that term.  Let that value of $r_2$

441: be $y$, we have, for each (fixed) $q$, each value of $x$ and the

442: associated value $y$, 

443: 

444: \[ E(X_{12}(q,m_x)) = E(\frac{C_{m_x}(q) - y}{x - y}) = 

445: \frac{E(C_{m_x}(q))-y}{x-y} = \frac{x-y}{x-y} = 1. \]

446: 

447: The second to the last equality is a consequence of the uniformity of

448: $M_t$:  because the collection $M_t$ (when $t=252$) covers $TR_1$

449: uniformly,  we have for each value $x$, 

450: $Prob_{\cal M} (q \in m | m \in M_{x,TR_1}) = x$, and

451: since $C_{m_x}(q)$ has only two values (0 or 1), and

452: $C_{m_x}(q) = 1 ~{\rm iff}~ q \in m$, we have the expected value of 

453: $C_{m_x}(q)$ equal to $x$.  Therefore 

454: 

455: \[ Y_{12}(q,M_t) = \frac{t_{0.0} + t_{0.2} + t_{0.4} + t_{0.6} +

456: t_{0.8} + t_{1.0}}{t} = 1. \]

457: 

458: In a more general case,  the values of $r_2$ are not

459: necessarily equal for all models with the same value for $r_1$, so

460: we cannot take $y$ and $x-y$ out as constants.  But

461: then we can further split the term by the values of $r_2$, and proceed

462: with the same argument.

463: 

464: A similar decomposition of $Y_{12}$ into terms corresponding to

465: different values of $r_2$ will show that $Y_{12}(q,M_t) = 0$ for those

466: points in $TR_2$.

467: 

468: \section{Projectability of Models}

469: We have built a classifier and shown that it works for $TR_1$

470: and $TR_2$.  How can this classifier work for an arbitrary point that

471: is not in $TR_1$ or $TR_2$?  Suppose that the feature space $F$

472: contains other points $p$ (marked by ``,''), and that each $p$ is

473: close to some training point $q$ (marked by ``.'') as follows.

474: 

475: \begin{center}

476: \begin{tabular}{cccccccccc}

477: .,&.,&.,&.,&.,&.,&.,&.,&.,&.,\\

478: $q_0,p_0$& $q_1,p_1$& $q_2,p_2$& $q_3,p_3$& $q_4,p_4$&

479: $q_5,p_5$& $q_6,p_6$& $q_7,p_7$& $q_8,p_8$& $q_9,p_9$\\

480: \end{tabular}

481: \end{center}

482: 

483: We can take the models $m$ as regions in the space that cover the

484: points $q$ in the same manner as before.  Say, if each point $q_i$

485: has a particular value of the feature $v$ (in our one-dimensional

486: feature space) that is $v(q_i)$.  We can define a model by ranges of values for

487: this feature, e.g., in our example $m_1$ covers

488: $q_3,q_5,q_6,q_8,q_9$, so we take

489: 

490: \begin{center}

491: \begin{tabular}{ll}

492: $m_1 = $ & $\{q | \frac{v(q_2)+v(q_3)}{2} < v(q) <

493: \frac{v(q_3)+v(q_4)}{2}\} \cup $\\

494: & $\{q | \frac{v_(q_4)+v(q_5)}{2} < v(q) < \frac{v(q_6)+v(q_7)}{2} \} \cup$ \\

495: & $\{q | \frac{v_(q_7)+v(q_8)}{2} < v(q) \}. $\\

496: \end{tabular}

497: \end{center}

498: 

499: Thus we can tell if an arbitrary point $p$ with value $v(p)$ for this

500: feature is inside or outside this model.

501: 

502: We can calculate the model's ratings in exactly the same way as

503: before,  using only the points $q$.  But now the same 

504: classifier works for the new points $p$,  since we can use the new

505: definitions of models to determine if $p$ is inside or outside each model.

506: Given the proximity relationship as above,

507: those points will be assigned to the same class as their closest

508: neighboring $q$.  If these are indeed the true classes for the points

509: $p$,  the classifier is perfect for this new set.  In the SD terminology, if we

510: call the two subsets of points $p$ that should be labeled as two

511: different classes $TE_1$ and $TE_2$,  i.e., $TE_1 =

512: \{p_0,p_1,p_2,p_7,p_8\}$, $TE_2 = \{p_3,p_4,p_5,p_6,p_9\}$, we say

513: that $TR_1$ and $TE_1$ are $M_t$-indiscernible,  and similarly $TR_2$

514: and $TE_2$ are also $M_t$-indiscernible.   This is to say,  from the

515: perspective of $M_t$, 

516: there is no difference between $TR_1$ and $TE_1$,  or $TR_2$ and

517: $TE_2$, therefore all the properties of $M_t$ that are observed using

518: $TR_1$ and $TR_2$ can be projected to $TE_1$ and $TE_2$.

519: The central challenge of an SD method is to maintain projectability,

520: uniformity, and enrichment of the collection of models at the same

521: time. 

522: 

523: \section{Developments of SD Theory and Algorithms}

524: 

525: \subsection{Algorithmic Implementations}

526: 

527: The method of stochastic discrimination constructs a classifier by

528: combining a large number of simple discriminators that are called {\em

529: weak models}.   A weak model is simply a subset of the feature space.

530: Conceptually,  the classifier is constructed by a three-step process:

531: (1) weak model generation, (2) weak model evaluation, and (3) weak

532: model combination.   The generator enumerates weak models in an

533: arbitrary order and passes them on to the evaluator.   The evaluator

534: has access to the training set.   It rates and filters the weak models

535: according to their capability in capturing points of each class,

536: and their contribution to satisfying the uniformity condition.

537: The combinator then produces a discriminant function that depends on

538: a point's membership status with respect to each model, and the

539: models' ratings.  At classification,  a point is assigned to the class

540: for which this discriminant has the highest value.  Informally, the 

541: method captures the intuition of gaining wisdom from graded random

542: guesses.  

543: 

544: \subsubsection*{Weak model generation.}

545: 

546: Two guidelines should be observed in generating the weak models:

547: 

548: (1) {\em projectability}:   A weak model should be able to capture more than

549: one point so that the solution can be projectable to points not

550: included in the training set.

551: Geometrically,  this means that a useful model must be of

552: certain minimum size,  and it should be able to capture points that

553: are considered {\em neighbors} of one another.

554: To guarantee similar accuracies of the classifier (based on

555: similar ratings of the weak models) on both training and testing data,  

556: one also needs an assumption that the training data are {\em representative}. 

557: Data representativeness and model projectability are two sides of

558: the same question.  More discussions of this can be found in

559: \cite{kn:berlind94}.

560: A weak model defines a {\em neighborhood} in the space,  and

561: we need a training sample in a neighborhood of every unseen sample.

562: Otherwise,  since our only knowledge of the class boundaries is from the given

563: training set,  there can be no basis for any inference concerning

564: regions of the feature space where no training samples are given.

565: 

566: (2) {\em simplicity of representation}:  A weak model should have a

567: simple representation.   That means, the membership of an

568: arbitrary point with respect to a model must be cheaply computable.

569: To illustrate this, consider representing a model as a listing of all

570: the points it contains.  This is practically useless since the

571: resultant solution could be as expensive as an exhaustive template

572: matching using all the points in the feature space.   An example of a

573: model with a simple representation is a half-plane in a

574: two-dimensional feature space.

575: 

576: Conditions (1) and (2) restrict the type of weak models yet by no

577: means reduce the number of candidates to any tangible limit.

578: To obtain an unbiased collection of the candidates with minimum effort,

579: random sampling with replacement is useful.

580: The training of the method thus relies on a stochastic process

581: which, at each iteration, generates a weak model that satisfies the

582: above conditions.

583: 

584: A convenient way to generate weak models randomly is to use a type of

585: model that can be described by a small number of parameters.

586: The values of the parameters can be chosen pseudo-randomly.

587: Some example types of models that can be generated this way include

588: (1) half-spaces bounded by a threshold on a randomly selected feature

589: dimension; 

590: (2) half-spaces bounded by a hyperplane of equi-distance to two

591: randomly selected points;

592: (3) regions bounded by two parallel hyperplanes perpendicular to a

593: randomly selected axis. 

594: (4) hypercubes centered at randomly selected points with edges of

595: varying lengths; 

596: (5) balls (based on the city-block metric) centered at randomly

597: selected points with randomly selected radii; and

598: (6) balls (based on the Euclidean metric) centered at a randomly

599: selected points with randomly selected radii.

600: A model can also be a union or intersection of several regions of

601: these types.  An implementation of SD using hyper-rectangular boxes as

602: weak models is described in \cite{kn:ho96}.

603: 

604: A number of heuristics may be used in creating these models.

605: These heuristics specify the way random points are chosen from the space,

606: or set limits on the maximum and minimum sizes of the models.

607: By this we mean restricting the choice of random points to, for instance,

608: points in the space whose coordinates fall inside the range of those of the

609: training samples,  or restricting the radii of the balls

610: to, for instance, a fraction of the range of values in a particular feature

611: dimension.  The purpose of these heuristics is to speed up the search

612: for acceptable models by confining the search within the most

613: interesting regions,  or to guarantee a minimum model size.

614: 

615: \subsubsection*{Enrichment enforcement.}

616: 

617: The enrichment condition is relatively easy to enforce,  as models

618: biased towards one class are most common.

619: But since the strength of the biases ($|d_{ij}(m)|$)

620: determines the rate at which accuracy increases,  we tend to prefer to

621: use models with an enrichment degree further away from zero.

622: 

623: One way to implement this is to use a threshold on the enrichment degree

624: to select weak models from the random stream so that they are of some

625: minimum quality.  In this way, one will be able to

626: use a smaller collection of models to yield a classifier of the

627: same level of accuracy.  However,  there are tradeoffs involved in

628: doing this.  For one thing,  models of higher rating are less likely

629: to appear in the stream, and so more random models have to be explored

630: in order to find sufficient numbers of higher quality weak models.  And

631: once the type of model is fixed and the value of the threshold is

632: set,  there is a risk that such models may never be found.

633: 

634: Alternatively,  one can use the most enriched model found in a

635: pre-determined number of trials.  This also makes the time needed for

636: training more predictable,  and it permits a tradeoff between training

637: time and quality of the weak models.

638: 

639: In enriching the model stream, it is important to remember that if the

640: quality of weak models selected is allowed to get too high, there is a

641: risk that they will become training set specific, that is, less

642: likely to be projectable to unseen samples.  This could present a problem

643: since the projectability of the final classifier is directly based on

644: the projectability of its component weak models.  

645: 

646: \subsubsection*{Uniformity promotion.}

647: 

648: The uniformity condition is much more difficult to satisfy.

649: Strict uniformity requires that every point be covered by the same

650: number of weak models of every combination of per-class ratings.

651: This is rather infeasible for continuous and unconstrained ratings.

652: 

653: One useful strategy is to use only weak models of a particular rating.

654: In such cases, the ratings $r_i(m)$ and $r_j(m)$ are the same for all

655: models $m$ enriched for the discrimination between classes $i$ and

656: $j$,  so we need only to make sure that each point is included in the

657: same number of models.  To enforce this,  models can be created in

658: groups such that each group partitions the entire space into a set of

659: non-overlapping regions.  An example is to use leaves of a fully-split

660: decision tree,  where each leave is perfectly enriched for one class,

661: and each point is covered by exactly one leave of each tree.  For any pairwise

662: discrimination between classes $i$ and $j$,  we can use only those

663: leaves of the trees that contain only points of class $i$.  In other

664: words,  $r_i(m)$ is always 1 and $r_j(m)$ is always 0.

665: Constraints are put in the tree-construction process to guarantee

666: some minimum projectability.

667: 

668: With other types of models,  a first step to promote uniformity is

669: to use models that are unions of small regions with simple boundaries.

670: The component regions may be scattered throughout the space.

671: These models have simple representations but can describe complicated

672: class boundaries.   They can have some minimum size and hence good

673: projectability.  At the same time,  the scattered locations of

674: component regions do not tend to cover large areas repeatedly.

675: 

676: A more sophisticated way to promote uniformity involves defining a measure

677: of the lack of uniformity and an algorithm to minimize such a measure.

678: The goal is to create or retain more models located in areas where the

679: coverage is thinner.    An example of such a measure is the count of those

680: points that are covered by less-than-average number of previously

681: retained models.   For each point $x$ in the class $c_0$ to be

682: positively enriched, we calculate, out of all previous models used for

683: that class, how many of them have covered $x$.

684: If the coverage is less than the average for class $c_0$,  we call $x$

685: a weak point.   When a new model is created,  we check how many

686: such weak points are covered by the new model.   

687: The ratio of the set of covered weak points to the set of all the weak

688: points is used as a merit score of how well this model improves

689: uniformity.   We can accept only those models with a score over a

690: pre-set threshold,  or take the model with the best score found in

691: a pre-set number of trials.

692: One can go further to introduce a bias to the model generator so that

693: models covering the weak points are more likely to be created.   The

694: later turns out to be a very effective strategy that led to good

695: results in our experiments.

696: 

697: \subsection{Alternative Discriminants and Approximate Uniformity}

698: 

699: The method outlined above allows for rich possibilities of variation at

700: the algorithmic level.  The variations may be in the design of the

701: weak model generator, or in ways to enforce the enrichment and uniformity

702: conditions.  It is also possible to change the definition of the

703: discriminant,  or to use different kinds of ratings.   

704: 

705: A variant of the discriminating function is studied in detail in

706: \cite{kn:berlind94}.  In this variant,  we define the ratings by

707: 

708: \[ {r'}_i (m) = {  | m \cap TR_i |  \over {|m \cap TR|}  }, \]

709: 

710: \noindent for all $i$.  It is an estimate of the posterior probability that a

711: point belongs to class $i$ given the condition that it is included in

712: model $m$.   The discriminant for class $i$ is defined to be:

713: 

714: \[ W_i(q) = {{ \sum_{k=1,...,p_i} C_m(q) {r'}_i(m) } \over 

715: 	{ \sum_{k=1,...,p_i} C_m(q)}}. \]

716: 

717: \noindent where $p_i$ is the number of models accumulated for class $i$.

718: 

719: It turns out that, with this discriminant, the classifier also approaches

720: perfection asymptotically provided that an additional {\em symmetry}

721: condition is satisfied.   The symmetry condition requires that the

722: ensemble includes the same number of models for all permutations of

723: $(r'_1, r'_2,..., r'_n)$.  It prevents biases created by using more

724: $(i,j)$-enriched models than $(j,i)$-enriched models for all pairs

725: $(i,j)$ \cite{kn:berlind94}.   Again, this condition may be enforced

726: by using only certain particular permutations of the $r'$ ratings

727: \cite{kn:ho95}.   This alternative discriminant is convenient

728: for multi-class discrimination problems.

729: 

730: The SD theory established the mathematical concepts of 

731: enrichment, uniformity, and projectability of a weak model ensemble.

732: Bounds on classification accuracy are developed based on

733: strict requirements on these conditions,  which is 

734: a mathematical idealization.  In practice, there are often

735: difficult tradeoffs among the three conditions.

736: Thus it is important to understand how much of the 

737: classification performance is affected when these conditions

738: are weakened.  This is the subject of study in \cite{kn:chen98},

739: where notions of near uniformity and weak indiscernibility

740: are introduced and their implications are studied.

741: 

742: \subsection{Structured Collections of Weak Models}

743: 

744: As a constructive procedure, the method of stochastic discrimination

745: depends on a detailed control of the uniformity of model coverage,

746: which is outlined but not fully published in the

747: literature \cite{kn:kleinberg00}.  The method of random subspaces 

748: followed these ideas but attempted a different approach.   Instead of

749: obtaining weak discrimination and projectability through simplicity of

750: the model form, and forcing uniformity by sophisticated algorithms,  the

751: method uses complete, locally pure partitions as given in fully split

752: decision trees \cite{kn:ho98} or nearest neighbor classifiers \cite{kn:ho98a}

753: to achieve strong discrimination and uniformity,  and then explicitly

754: forces different generalization patterns on the component classifiers.

755: This is done by training large capacity component classifiers such as

756: nearest neighbors and decision trees to fully fit the data,  but

757: restricting the training of each classifier to a coordinate subspace

758: of the feature space where all the data points are projected,  so

759: that classifications remain invariant in

760: the complement subspace.  If there is no ambiguity in the subspaces,  the

761: individual classifiers maintain maximum accuracy on the training data,

762: with no cases deliberately chosen to be sacrificed,  and thus the

763: method does not run into the paradox of sacrificing some training

764: points in the hope for better generalization accuracy.

765: This is to create a collection of weak models in a structured way.

766: 

767: However the tension among the three factors persists.   There is

768: another difficult tradeoff in how much discriminating power to retain

769: for the component classifiers.  Can every one use only a single

770: feature dimension so as to maximize invariance in the complement

771: dimensions?   Also,  projection to coordinate subspaces sets parts of

772: the decision boundaries parallel to the coordinate axes.  Augmenting

773: the raw features by simple transformations \cite{kn:ho98} introduces

774: more flexibility, but it may still be insufficient for an arbitrary

775: problem.  Optimization of generalization performance will continue to

776: depend on a detailed control of the projections to suit a particular

777: problem. 

778: 

779: \section{Conclusions}

780: 

781: The theory of stochastic discrimination identifies three and only

782: three sufficient conditions for a classifier to achieve maximum

783: accuracy for a problem.   These are just the three elements long

784: believed to be important in pattern recognition:  discrimination

785: power, complementary information,  and generalization ability.

786: It sets a foundation for theories of ensemble learning.

787: Many current questions on classifier combination can have an answer

788: in the arguments of the SD theory:

789: What is good about building the classifier on weak models instead of

790: strong models?  Because weak models are easier to obtain,  and their

791: smaller capacity renders them less sensitive to sampling errors in

792: small training sets \cite{kn:vapnik82} \cite{kn:vapnik98}, thus

793: they are more likely to have similar coverage on the unseen points

794: from the same problem.  Why are many models needed?   Because the

795: method relies on the law of large numbers to reduce the variance of

796: the discriminant on each single point.  How should these models

797: complement each other?  The uniformity condition specifies exactly what

798: kind of correlation is needed among the individual models. 

799: 

800: Finally, we emphasize that the accuracy of SD methods is not achieved

801: by intentionally limiting the VC dimension of the complete system; the

802: combination of many weak models can have a very large VC dimension.

803: It is a consequence of the symmetry relating probabilities in the two

804: spaces, and the law of large numbers.  It is a structural property of

805: the topology.   The observation of this symmetry and its relationship

806: to ensemble learning is a deep insight of Kleinberg's that we believe

807: can lead to better understanding of other ensemble methods.

808: 

809: \section*{Acknowledgements}

810: The author thanks Eugene Kleinberg for many discussions over the past

811: decade on the theory of stochastic discrimination, its comparison

812: to other approaches, and perspectives on the fundamental issues in

813: pattern recognition. 

814: 

815: \begin{thebibliography}{99}

816: 

817: \bibitem{kn:berlind94}

818: R. Berlind,

819: {\em An Alternative Method of Stochastic Discrimination with

820: Applications to Pattern Recognition},

821: Doctoral Dissertation,  Department of Mathematics,  State University

822: of New York at Buffalo,  1994.

823: 

824: \bibitem{kn:breiman96}

825: L. Breiman, ``Bagging predictors,''  {\em Machine Learning}, {\bf 24},

826: 1996, 123-140. 

827: 

828: \bibitem{kn:chen98}

829: D. Chen, 

830: ``Estimates of Classification Accuracies for Kleinberg's Method of

831: Stochastic Discrimination in Pattern Recognition,'' Ph.D. Thesis, SUNY

832: at Buffalo, 1998. 

833: 

834: \bibitem{kn:dietterich95b}

835: T.G. Dietterich, G. Bakiri, ``Solving multiclass learning problems via

836: error-correcting output codes,'' {\em Journal of Artificial Intelligence

837: Research}, {\bf 2}, 1995, 263-286.

838: 

839: \bibitem{kn:freund96}

840: Y. Freund, R.E. Schapire, 

841: ``Experiments with a New Boosting Algorithm,''

842: {\em Proceedings of the Thirteenth International Conference on Machine

843: Learning}, Bari, Italy, July 3-6, 1996, 148-156.

844: 

845: \bibitem{kn:hansen90}

846: L.K. Hansen, P. Salamon, ``Neural network ensembles,''

847: {\em IEEE Transactions on Pattern Analysis and Machine

848: Intelligence}, {\bf PAMI-12}, 10, October 1990, 993-1001.

849: 

850: \bibitem{kn:ho95}

851: T.K. Ho,

852: Random Decision Forests,

853: {\em Proceedings of the 3rd International Conference on Document Analysis and

854: Recognition},  Montreal,  Canada, August 14-18, 1995, 278-282.

855: 

856: \bibitem{kn:ho02}

857: T.K. Ho, 

858: Multiple classifier combination: Lessons and next steps,

859: in A. Kandel, H. Bunke, (eds.), {\em Hybrid Methods in Pattern

860: Recognition},  World Scientific,  2002.

861: 

862: \bibitem{kn:ho96}

863: T.K. Ho,  E.M. Kleinberg,

864: Building Projectable Classifiers of Arbitrary Complexity,

865: {\em Proceedings of the 13th International Conference on Pattern Recognition},

866: Vienna, Austria, August 25-30, 1996, 880-885.

867: 

868: \bibitem{kn:ho98}

869: T.K. Ho,  The random subspace method for constructing decision forests,

870: {\em IEEE Transactions on Pattern Analysis and Machine

871: Intelligence},  {\bf 20}, 8,  August 1998, 832-844.

872: 

873: \bibitem{kn:ho98a}

874: T.K. Ho,

875: Nearest Neighbors in Random Subspaces,

876: {\em Proceedings of the Second International

877: Workshop on Statistical Techniques in Pattern Recognition}, Sydney,

878: Australia,  August 11-13, 1998, 640-648.

879: 

880: \bibitem{kn:ho94}

881: T.K. Ho,  J. J. Hull, S.N. Srihari,

882: Decision Combination in Multiple Classifier Systems,

883: {\em IEEE Transactions on Pattern Analysis and Machine

884: Intelligence}, {\bf PAMI-16}, 1, January 1994, 66-75.

885: 

886: \bibitem{kn:huang95}

887: Y.S. Huang, C.Y. Suen, A method of combining multiple experts for the

888: recognition of unconstrained handwritten numerals,

889: {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},

890: {\bf PAMI-17}, 1, January 1995, 90-94.

891: 

892: \bibitem{kn:kittler98}

893: J. Kittler, M. Hatef, R.P.W. Duin, J. Matas,

894: On combining classifiers, 

895: {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},

896: {\bf PAMI-20}, 3, March 1998, 226-239.

897: 

898: \bibitem{kn:kleinberg90}

899: E.M. Kleinberg,

900: Stochastic Discrimination,

901: {\em Annals of Mathematics and Artificial Intelligence},

902: {\bf 1}, 1990, 207-239.

903: 

904: \bibitem{kn:kleinberg96}

905: E.M. Kleinberg,

906: An overtraining-resistant stochastic modeling method for pattern

907: recognition,

908: {\em Annals of Statistics}, {\bf 4}, 6, December 1996, 2319-2349.

909: 

910: \bibitem{kn:kleinberg00}

911: E.M. Kleinberg,

912: On the algorithmic implementation of stochastic discrimination,

913: {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},

914: {\bf PAMI-22}, 5, May 2000, 473-490.

915: 

916: \bibitem{kn:kleinberg00b}

917: E.M. Kleinberg,

918: A mathematically rigorous foundation for supervised learning,

919: in J. Kittler, F. Roli, (eds.), {\em Multiple Classifier Systems}, Lecture

920: Notes in Computer Science 1857, Springer, 2000, 67-76.

921: 

922: \bibitem{kn:lam97}

923: L. Lam, C.Y. Suen,

924: Application of majority voting to pattern recognition,

925: {\em IEEE Transactions on Systems, Man, and Cybernetics}, {\bf

926: SMC-27}, 5, September/October 1997, 553-568.

927: 

928: \bibitem{kn:vapnik82}

929: V. Vapnik, {\em Estimation of Dependences Based on Empirical Data},

930: Springer-Verlag, 1982.

931: 

932: \bibitem{kn:vapnik98}

933: V. Vapnik, {\em Statistical Learning Theory},

934: John Wiley \& Sons, 1998.

935: 

936: \bibitem{kn:wolpert92}

937: D.H. Wolpert, Stacked generalization, {\em Neural Networks}, {\bf 5},

938: 1992, 241-259.

939: 

940: \end{thebibliography}

941: 

942: \begin{figure}[p]

943: \centering

944: \begin{tabular}{cc}

945: \psfig{file=figs/uq0r1.ps,height=1.5in,clip=}&

946: \psfig{file=figs/uq5r1.ps,height=1.5in,clip=} \\

947: $q_0 \in TR_1$ & $q_5 \in TR_2$ \\

948: \psfig{file=figs/uq1r1.ps,height=1.5in,clip=}&

949: \psfig{file=figs/uq6r1.ps,height=1.5in,clip=}\\

950: $q_1 \in TR_1$ & $q_6 \in TR_2$ \\

951: \psfig{file=figs/uq2r1.ps,height=1.5in,clip=}&

952: \psfig{file=figs/uq7r1.ps,height=1.5in,clip=}\\

953: $q_2 \in TR_1$ & $q_7 \in TR_1$ \\

954: \psfig{file=figs/uq3r1.ps,height=1.5in,clip=}&

955: \psfig{file=figs/uq8r1.ps,height=1.5in,clip=}\\

956: $q_3 \in TR_2$ & $q_8 \in TR_1$ \\

957: \psfig{file=figs/uq4r1.ps,height=1.5in,clip=}&

958: \psfig{file=figs/uq9r1.ps,height=1.5in,clip=}\\

959: $q_4 \in TR_2$ & $q_9 \in TR_2$ \\

960: \end{tabular}

961: \caption{$f_{M_t,r_1,TR_1}(q)$ for each point $q$ and set $M_t$.

962: In each plot, the $x$ axis is $t$ that ranges from 0 to 252, the $y$

963: axis is $r$ that ranges from $0$ to $1$, and the $z$ axis is 

964: $f_{M_t,r_1,TR_1}$.}

965: \end{figure}

966: 

967: \begin{figure}[p]

968: \centering

969: \begin{tabular}{cc}

970: \psfig{file=figs/uq0r2.ps,height=1.5in,clip=}&

971: \psfig{file=figs/uq5r2.ps,height=1.5in,clip=}\\

972: $q_0 \in TR_1$ & $q_5 \in TR_2$ \\

973: \psfig{file=figs/uq1r2.ps,height=1.5in,clip=}&

974: \psfig{file=figs/uq6r2.ps,height=1.5in,clip=}\\

975: $q_1 \in TR_1$ & $q_6 \in TR_2$ \\

976: \psfig{file=figs/uq2r2.ps,height=1.5in,clip=}&

977: \psfig{file=figs/uq7r2.ps,height=1.5in,clip=}\\

978: $q_2 \in TR_1$ & $q_7 \in TR_1$ \\

979: \psfig{file=figs/uq3r2.ps,height=1.5in,clip=}&

980: \psfig{file=figs/uq8r2.ps,height=1.5in,clip=}\\

981: $q_3 \in TR_2$ & $q_8 \in TR_1$ \\

982: \psfig{file=figs/uq4r2.ps,height=1.5in,clip=}&

983: \psfig{file=figs/uq9r2.ps,height=1.5in,clip=}\\

984: $q_4 \in TR_2$ & $q_9 \in TR_2$ \\

985: \end{tabular}

986: \caption{$f_{M_t,r_2,TR_2}(q)$ for each point $q$ and set $M_t$.

987: In each plot, the $x$ axis is $t$ that ranges from 0 to 252, the $y$

988: axis is $r$ that ranges from $0$ to $1$, and the $z$ axis is 

989: $f_{M_t,r_2,TR_2}$.}

990: \end{figure}

991: 

992: \clearpage

993: \newpage

994: \noindent{\large {\bf Appendix.}}

995: 

996: Listing of an awk script to calculate a model's ratings.

997: The script takes a file where each line contains the point indices of

998: the elements of a model (as in Table 1).

999: 

1000: \vspace{-0.1in}

1001: {\scriptsize

1002: \begin{verbatim}

1003: # <awk.rating>

1004: # run:  awk -f awk.rating list_of_models > models.rating

1005: # input:

1006: # 3 5 6 8 9

1007: # 0 1 2 6 8

1008: # ...

1009: BEGIN{

1010: sizeTR1 = 5;

1011: sizeTR2 = 5;

1012: # full space = (x x x o o o o x x o)

1013: c[0] = 1;  c[1] = 1;  c[2] = 1;  c[3] = 2;  c[4] = 2;  

1014: c[5] = 2;  c[6] = 2;  c[7] = 1;  c[8] = 1;  c[9] = 2;

1015: }

1016: { cover1 = 0;   

1017:   cover2 = 0;   

1018:   sizeM = NF;

1019:   for (i=1; i<=NF; ++i) if (c[$i] == 1) cover1 ++; else cover2 ++;

1020:   r1 = cover1/sizeTR1;

1021:   r2 = cover2/sizeTR2;

1022:   enr = r1 - r2;

1023:   if (enr == 0.0) { xin = 0.0; xout = 0.0;}

1024:   else {

1025:      xin  = (1.0 - r2)/enr;

1026:      xout = (0.0 - r2)/enr; 

1027:   }

1028:   printf ``model %s r1 %.2f r2 %.2f enr = r1-r2 = %.2f xin %.2f xout %.2f\n'',\

1029:   $0,r1,r2,enr,xin,xout;

1030: }

1031: \end{verbatim}

1032: }

1033: 

1034: Listing of an awk script to calculate the discriminant $Y_{12}$ for

1035: each point $q$, given a list of models and their ratings (output of

1036: awk.rating).

1037: 

1038: \vspace{-0.1in}

1039: {\scriptsize

1040: \begin{verbatim}

1041: # <awk.discrim>

1042: # run: awk -f awk.discrim models.rating 

1043: # input:

1044: # model 3 5 6 8 9  r1 0.20 r2 0.80 enr = r1-r2 = -0.60 xin -0.33 xout 1.33

1045: # model 0 1 2 6 8  r1 0.80 r2 0.20 enr = r1-r2 = 0.60 xin 1.33 xout -0.33

1046: # ...

1047: { 

1048:   nm++;

1049:   xin = $17;

1050:   xout = $19;

1051: #

1052: # check for each point q, whether model m (this row) covers q,

1053: # and increment y[q] accordingly

1054: #

1055:   for (q=0; q<10;++q) {

1056:       inmodel = 0;

1057:       for (i=2; i<=6; ++i) {

1058:           if ($i == q) inmodel = 1;

1059:       }

1060:       if (inmodel) {

1061:          y[q] = (y[q]*(nm-1) + xin)/nm; }

1062:       else {      

1063:          y[q] = (y[q]*(nm-1) + xout)/nm;}

1064:   }

1065:   printf ``Y(M,q)_(|M|= %3d ) ``, nm;

1066:   for (q=0; q<10;++q) { printf ``%5.2f ``, y[q];} 

1067:   printf ``\n'';

1068: }

1069: \end{verbatim}

1070: }

1071: 

1072: \begin{table}[htbp]

1073: {\small

1074: \center

1075: \caption{Models $m_t$ in $M_{0.5,A}$ in the order of $M = m_1, m_2,

1076: ..., m_{252}$.  Each model

1077: is shown with its elements denoted by the indices $i$ of $q_i$ in $A$.

1078: For example, $m_1 = \{q_3,q_5,q_6,q_8,q_9\}$. }

1079: \begin{tabular}{||l|c||l|c||l|c||l|c||l|c||l|c||} \hline

1080: $m_t$ & elements &

1081: $m_t$ & elements &

1082: $m_t$ & elements &

1083: $m_t$ & elements &

1084: $m_t$ & elements &

1085: $m_t$ & elements \\ \hline

1086: $m_{1}$&35689& $m_{43}$&12689& $m_{85}$&24578& $m_{127}$&01469& $m_{169}$&02468& $m_{211}$&02458 \\

1087: $m_{2}$&01268& $m_{44}$&04569& $m_{86}$&23568& $m_{128}$&03679& $m_{170}$&35678& $m_{212}$&13457 \\

1088: $m_{3}$&04789& $m_{45}$&01245& $m_{87}$&01267& $m_{129}$&04579& $m_{171}$&03589& $m_{213}$&24689 \\

1089: $m_{4}$&25689& $m_{46}$&01458& $m_{88}$&01257& $m_{130}$&01237& $m_{172}$&34679& $m_{214}$&03478 \\

1090: $m_{5}$&02679& $m_{47}$&15679& $m_{89}$&05679& $m_{131}$&24789& $m_{173}$&12346& $m_{215}$&23589 \\

1091: $m_{6}$&34578& $m_{48}$&12457& $m_{90}$&24589& $m_{132}$&45689& $m_{174}$&12458& $m_{216}$&24679 \\

1092: $m_{7}$&13459& $m_{49}$&02379& $m_{91}$&04589& $m_{133}$&16789& $m_{175}$&35789& $m_{217}$&02456 \\

1093: $m_{8}$&01238& $m_{50}$&02568& $m_{92}$&12467& $m_{134}$&13479& $m_{176}$&02358& $m_{218}$&05689 \\

1094: $m_{9}$&12347& $m_{51}$&12357& $m_{93}$&13578& $m_{135}$&02349& $m_{177}$&35679& $m_{219}$&12789 \\

1095: $m_{10}$&01579& $m_{52}$&14678& $m_{94}$&02369& $m_{136}$&13469& $m_{178}$&13458& $m_{220}$&02346 \\

1096: $m_{11}$&34589& $m_{53}$&12678& $m_{95}$&12469& $m_{137}$&03678& $m_{179}$&01459& $m_{221}$&23489 \\

1097: $m_{12}$&03459& $m_{54}$&23567& $m_{96}$&04567& $m_{138}$&23679& $m_{180}$&03479& $m_{222}$&23467 \\

1098: $m_{13}$&23459& $m_{55}$&02789& $m_{97}$&14679& $m_{139}$&46789& $m_{181}$&14789& $m_{223}$&12489 \\

1099: $m_{14}$&02457& $m_{56}$&24567& $m_{98}$&13467& $m_{140}$&01468& $m_{182}$&23678& $m_{224}$&14589 \\

1100: $m_{15}$&02368& $m_{57}$&13569& $m_{99}$&45678& $m_{141}$&03689& $m_{183}$&03456& $m_{225}$&25678 \\

1101: $m_{16}$&02689& $m_{58}$&01259& $m_{100}$&03469& $m_{142}$&02478& $m_{184}$&13456& $m_{226}$&12579 \\

1102: $m_{17}$&01368& $m_{59}$&23479& $m_{101}$&34789& $m_{143}$&23457& $m_{185}$&01568& $m_{227}$&03458 \\

1103: $m_{18}$&13589& $m_{60}$&03579& $m_{102}$&45679& $m_{144}$&02347& $m_{186}$&01578& $m_{228}$&01569 \\

1104: $m_{19}$&14579& $m_{61}$&12368& $m_{103}$&01358& $m_{145}$&01289& $m_{187}$&01678& $m_{229}$&45789 \\

1105: $m_{20}$&23468& $m_{62}$&23578& $m_{104}$&01379& $m_{146}$&01369& $m_{188}$&12367& $m_{230}$&12358 \\

1106: $m_{21}$&26789& $m_{63}$&02345& $m_{105}$&01236& $m_{147}$&01356& $m_{189}$&12345& $m_{231}$&02579 \\

1107: $m_{22}$&15678& $m_{64}$&01479& $m_{106}$&01679& $m_{148}$&12379& $m_{190}$&25679& $m_{232}$&01457 \\

1108: $m_{23}$&04578& $m_{65}$&03569& $m_{107}$&13689& $m_{149}$&02569& $m_{191}$&02367& $m_{233}$&05789 \\

1109: $m_{24}$&04679& $m_{66}$&01346& $m_{108}$&12479& $m_{150}$&34678& $m_{192}$&01256& $m_{234}$&01247 \\

1110: $m_{25}$&02459& $m_{67}$&24568& $m_{109}$&14568& $m_{151}$&24569& $m_{193}$&13679& $m_{235}$&03467 \\

1111: $m_{26}$&12569& $m_{68}$&01359& $m_{110}$&15689& $m_{152}$&03578& $m_{194}$&04689& $m_{236}$&12359 \\

1112: $m_{27}$&01269& $m_{69}$&12459& $m_{111}$&01258& $m_{153}$&02359& $m_{195}$&04568& $m_{237}$&02567 \\

1113: $m_{28}$&06789& $m_{70}$&01239& $m_{112}$&12389& $m_{154}$&01234& $m_{196}$&12578& $m_{238}$&12356 \\

1114: $m_{29}$&01689& $m_{71}$&24678& $m_{113}$&03568& $m_{155}$&01345& $m_{197}$&12468& $m_{239}$&02469 \\

1115: $m_{30}$&01248& $m_{72}$&01347& $m_{114}$&23689& $m_{156}$&02348& $m_{198}$&03468& $m_{240}$&13468 \\

1116: $m_{31}$&12456& $m_{73}$&01467& $m_{115}$&23478& $m_{157}$&03457& $m_{199}$&34569& $m_{241}$&02479 \\

1117: $m_{32}$&13579& $m_{74}$&04678& $m_{116}$&34568& $m_{158}$&02357& $m_{200}$&12369& $m_{242}$&36789 \\

1118: $m_{33}$&34689& $m_{75}$&12589& $m_{117}$&23569& $m_{159}$&01235& $m_{201}$&13489& $m_{243}$&13568 \\

1119: $m_{34}$&12679& $m_{76}$&01348& $m_{118}$&14689& $m_{160}$&01378& $m_{202}$&12567& $m_{244}$&02467 \\

1120: $m_{35}$&12568& $m_{77}$&14569& $m_{119}$&23789& $m_{161}$&14567& $m_{203}$&02489& $m_{245}$&01589 \\

1121: $m_{36}$&34579& $m_{78}$&01789& $m_{120}$&01246& $m_{162}$&23458& $m_{204}$&02678& $m_{246}$&01478 \\

1122: $m_{37}$&01389& $m_{79}$&01367& $m_{121}$&23579& $m_{163}$&56789& $m_{205}$&13567& $m_{247}$&15789 \\

1123: $m_{38}$&23469& $m_{80}$&12478& $m_{122}$&01456& $m_{164}$&34567& $m_{206}$&01357& $m_{248}$&01349 \\

1124: $m_{39}$&24579& $m_{81}$&25789& $m_{123}$&23456& $m_{165}$&01249& $m_{207}$&01278& $m_{249}$&02356 \\

1125: $m_{40}$&02589& $m_{82}$&01489& $m_{124}$&03789& $m_{166}$&03489& $m_{208}$&02578& $m_{250}$&14578 \\

1126: $m_{41}$&01567& $m_{83}$&03567& $m_{125}$&05678& $m_{167}$&02389& $m_{209}$&12348& $m_{251}$&13789 \\

1127: $m_{42}$&13478& $m_{84}$&12349& $m_{126}$&13678& $m_{168}$&12378& $m_{210}$&01279& $m_{252}$&02378 \\

1128: \hline

1129: \end{tabular}

1130: }

1131: \end{table}

1132: 

1133: \begin{table}[htbp]

1134: \caption{Ratio of coverage of each point $q$ by members of $M_t$ as

1135: $M_t$ expands.}

1136: \begin{center}

1137: {\tiny

1138: \begin{tabular}{||l|cccccccccc|cccccccccc||}\hline

1139:       & \multicolumn{10}{c|}{$N(M_t,q)$}

1140:       & \multicolumn{10}{c||}{$Y(M_t,q)$} \\ \hline

1141: $M_t$ & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1142:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$

1143:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1144:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1145: $M_{1}$&0&0&0&1&0&1&1&0&1&1&0.00&0.00&0.00&1.00&0.00&1.00&1.00&0.00&1.00&1.00\\

1146: $M_{2}$&1&1&1&1&0&1&2&0&2&1&0.50&0.50&0.50&0.50&0.00&0.50&1.00&0.00&1.00&0.50\\

1147: $M_{3}$&2&1&1&1&1&1&2&1&3&2&0.67&0.33&0.33&0.33&0.33&0.33&0.67&0.33&1.00&0.67\\

1148: $M_{4}$&2&1&2&1&1&2&3&1&4&3&0.50&0.25&0.50&0.25&0.25&0.50&0.75&0.25&1.00&0.75\\

1149: $M_{5}$&3&1&3&1&1&2&4&2&4&4&0.60&0.20&0.60&0.20&0.20&0.40&0.80&0.40&0.80&0.80\\

1150: $M_{6}$&3&1&3&2&2&3&4&3&5&4&0.50&0.17&0.50&0.33&0.33&0.50&0.67&0.50&0.83&0.67\\

1151: $M_{7}$&3&2&3&3&3&4&4&3&5&5&0.43&0.29&0.43&0.43&0.43&0.57&0.57&0.43&0.71&0.71\\

1152: $M_{8}$&4&3&4&4&3&4&4&3&6&5&0.50&0.38&0.50&0.50&0.38&0.50&0.50&0.38&0.75&0.62\\

1153: $M_{9}$&4&4&5&5&4&4&4&4&6&5&0.44&0.44&0.56&0.56&0.44&0.44&0.44&0.44&0.67&0.56\\

1154: $M_{10}$&5&5&5&5&4&5&4&5&6&6&0.50&0.50&0.50&0.50&0.40&0.50&0.40&0.50&0.60&0.60\\

1155: $M_{11}$&5&5&5&6&5&6&4&5&7&7&0.45&0.45&0.45&0.55&0.45&0.55&0.36&0.45&0.64&0.64\\

1156: $M_{12}$&6&5&5&7&6&7&4&5&7&8&0.50&0.42&0.42&0.58&0.50&0.58&0.33&0.42&0.58&0.67\\

1157: $M_{13}$&6&5&6&8&7&8&4&5&7&9&0.46&0.38&0.46&0.62&0.54&0.62&0.31&0.38&0.54&0.69\\

1158: $M_{14}$&7&5&7&8&8&9&4&6&7&9&0.50&0.36&0.50&0.57&0.57&0.64&0.29&0.43&0.50&0.64\\

1159: $M_{15}$&8&5&8&9&8&9&5&6&8&9&0.53&0.33&0.53&0.60&0.53&0.60&0.33&0.40&0.53&0.60\\

1160: $M_{16}$&9&5&9&9&8&9&6&6&9&10&0.56&0.31&0.56&0.56&0.50&0.56&0.38&0.38&0.56&0.62\\

1161: $M_{17}$&10&6&9&10&8&9&7&6&10&10&0.59&0.35&0.53&0.59&0.47&0.53&0.41&0.35&0.59&0.59\\

1162: $M_{18}$&10&7&9&11&8&10&7&6&11&11&0.56&0.39&0.50&0.61&0.44&0.56&0.39&0.33&0.61&0.61\\

1163: $M_{19}$&10&8&9&11&9&11&7&7&11&12&0.53&0.42&0.47&0.58&0.47&0.58&0.37&0.37&0.58&0.63\\

1164: $M_{20}$&10&8&10&12&10&11&8&7&12&12&0.50&0.40&0.50&0.60&0.50&0.55&0.40&0.35&0.60&0.60\\

1165: $M_{21}$&10&8&11&12&10&11&9&8&13&13&0.48&0.38&0.52&0.57&0.48&0.52&0.43&0.38&0.62&0.62\\

1166: $M_{22}$&10&9&11&12&10&12&10&9&14&13&0.45&0.41&0.50&0.55&0.45&0.55&0.45&0.41&0.64&0.59\\

1167: $M_{23}$&11&9&11&12&11&13&10&10&15&13&0.48&0.39&0.48&0.52&0.48&0.57&0.43&0.43&0.65&0.57\\

1168: $M_{24}$&12&9&11&12&12&13&11&11&15&14&0.50&0.38&0.46&0.50&0.50&0.54&0.46&0.46&0.62&0.58\\

1169: $M_{25}$&13&9&12&12&13&14&11&11&15&15&0.52&0.36&0.48&0.48&0.52&0.56&0.44&0.44&0.60&0.60\\

1170: $M_{26}$&13&10&13&12&13&15&12&11&15&16&0.50&0.38&0.50&0.46&0.50&0.58&0.46&0.42&0.58&0.62\\

1171: $M_{27}$&14&11&14&12&13&15&13&11&15&17&0.52&0.41&0.52&0.44&0.48&0.56&0.48&0.41&0.56&0.63\\

1172: $M_{28}$&15&11&14&12&13&15&14&12&16&18&0.54&0.39&0.50&0.43&0.46&0.54&0.50&0.43&0.57&0.64\\

1173: $M_{29}$&16&12&14&12&13&15&15&12&17&19&0.55&0.41&0.48&0.41&0.45&0.52&0.52&0.41&0.59&0.66\\

1174: $M_{30}$&17&13&15&12&14&15&15&12&18&19&0.57&0.43&0.50&0.40&0.47&0.50&0.50&0.40&0.60&0.63\\

1175: $M_{31}$&17&14&16&12&15&16&16&12&18&19&0.55&0.45&0.52&0.39&0.48&0.52&0.52&0.39&0.58&0.61\\

1176: $M_{32}$&17&15&16&13&15&17&16&13&18&20&0.53&0.47&0.50&0.41&0.47&0.53&0.50&0.41&0.56&0.62\\

1177: $M_{33}$&17&15&16&14&16&17&17&13&19&21&0.52&0.45&0.48&0.42&0.48&0.52&0.52&0.39&0.58&0.64\\

1178: $M_{34}$&17&16&17&14&16&17&18&14&19&22&0.50&0.47&0.50&0.41&0.47&0.50&0.53&0.41&0.56&0.65\\

1179: $M_{35}$&17&17&18&14&16&18&19&14&20&22&0.49&0.49&0.51&0.40&0.46&0.51&0.54&0.40&0.57&0.63\\

1180: $M_{36}$&17&17&18&15&17&19&19&15&20&23&0.47&0.47&0.50&0.42&0.47&0.53&0.53&0.42&0.56&0.64\\

1181: $M_{37}$&18&18&18&16&17&19&19&15&21&24&0.49&0.49&0.49&0.43&0.46&0.51&0.51&0.41&0.57&0.65\\

1182: $M_{38}$&18&18&19&17&18&19&20&15&21&25&0.47&0.47&0.50&0.45&0.47&0.50&0.53&0.39&0.55&0.66\\

1183: $M_{39}$&18&18&20&17&19&20&20&16&21&26&0.46&0.46&0.51&0.44&0.49&0.51&0.51&0.41&0.54&0.67\\

1184: $M_{40}$&19&18&21&17&19&21&20&16&22&27&0.47&0.45&0.53&0.42&0.47&0.53&0.50&0.40&0.55&0.68\\

1185: $M_{41}$&20&19&21&17&19&22&21&17&22&27&0.49&0.46&0.51&0.41&0.46&0.54&0.51&0.41&0.54&0.66\\

1186: $M_{42}$&20&20&21&18&20&22&21&18&23&27&0.48&0.48&0.50&0.43&0.48&0.52&0.50&0.43&0.55&0.64\\

1187: $M_{43}$&20&21&22&18&20&22&22&18&24&28&0.47&0.49&0.51&0.42&0.47&0.51&0.51&0.42&0.56&0.65\\

1188: $M_{44}$&21&21&22&18&21&23&23&18&24&29&0.48&0.48&0.50&0.41&0.48&0.52&0.52&0.41&0.55&0.66\\

1189: $M_{45}$&22&22&23&18&22&24&23&18&24&29&0.49&0.49&0.51&0.40&0.49&0.53&0.51&0.40&0.53&0.64\\

1190: $M_{46}$&23&23&23&18&23&25&23&18&25&29&0.50&0.50&0.50&0.39&0.50&0.54&0.50&0.39&0.54&0.63\\

1191: $M_{47}$&23&24&23&18&23&26&24&19&25&30&0.49&0.51&0.49&0.38&0.49&0.55&0.51&0.40&0.53&0.64\\

1192: $M_{48}$&23&25&24&18&24&27&24&20&25&30&0.48&0.52&0.50&0.38&0.50&0.56&0.50&0.42&0.52&0.62\\

1193: $M_{49}$&24&25&25&19&24&27&24&21&25&31&0.49&0.51&0.51&0.39&0.49&0.55&0.49&0.43&0.51&0.63\\

1194: $M_{50}$&25&25&26&19&24&28&25&21&26&31&0.50&0.50&0.52&0.38&0.48&0.56&0.50&0.42&0.52&0.62\\

1195: $M_{51}$&25&26&27&20&24&29&25&22&26&31&0.49&0.51&0.53&0.39&0.47&0.57&0.49&0.43&0.51&0.61\\

1196: $M_{52}$&25&27&27&20&25&29&26&23&27&31&0.48&0.52&0.52&0.38&0.48&0.56&0.50&0.44&0.52&0.60\\

1197: $M_{53}$&25&28&28&20&25&29&27&24&28&31&0.47&0.53&0.53&0.38&0.47&0.55&0.51&0.45&0.53&0.58\\

1198: $M_{54}$&25&28&29&21&25&30&28&25&28&31&0.46&0.52&0.54&0.39&0.46&0.56&0.52&0.46&0.52&0.57\\

1199: $M_{55}$&26&28&30&21&25&30&28&26&29&32&0.47&0.51&0.55&0.38&0.45&0.55&0.51&0.47&0.53&0.58\\

1200: $M_{56}$&26&28&31&21&26&31&29&27&29&32&0.46&0.50&0.55&0.38&0.46&0.55&0.52&0.48&0.52&0.57\\

1201: $M_{57}$&26&29&31&22&26&32&30&27&29&33&0.46&0.51&0.54&0.39&0.46&0.56&0.53&0.47&0.51&0.58\\

1202: $M_{58}$&27&30&32&22&26&33&30&27&29&34&0.47&0.52&0.55&0.38&0.45&0.57&0.52&0.47&0.50&0.59\\

1203: $M_{59}$&27&30&33&23&27&33&30&28&29&35&0.46&0.51&0.56&0.39&0.46&0.56&0.51&0.47&0.49&0.59\\

1204: $M_{60}$&28&30&33&24&27&34&30&29&29&36&0.47&0.50&0.55&0.40&0.45&0.57&0.50&0.48&0.48&0.60\\

1205: $M_{61}$&28&31&34&25&27&34&31&29&30&36&0.46&0.51&0.56&0.41&0.44&0.56&0.51&0.48&0.49&0.59\\

1206: $M_{62}$&28&31&35&26&27&35&31&30&31&36&0.45&0.50&0.56&0.42&0.44&0.56&0.50&0.48&0.50&0.58\\

1207: $M_{63}$&29&31&36&27&28&36&31&30&31&36&0.46&0.49&0.57&0.43&0.44&0.57&0.49&0.48&0.49&0.57\\

1208: $M_{64}$&30&32&36&27&29&36&31&31&31&37&0.47&0.50&0.56&0.42&0.45&0.56&0.48&0.48&0.48&0.58\\

1209: $M_{65}$&31&32&36&28&29&37&32&31&31&38&0.48&0.49&0.55&0.43&0.45&0.57&0.49&0.48&0.48&0.58\\

1210: $M_{66}$&32&33&36&29&30&37&33&31&31&38&0.48&0.50&0.55&0.44&0.45&0.56&0.50&0.47&0.47&0.58\\

1211: $M_{67}$&32&33&37&29&31&38&34&31&32&38&0.48&0.49&0.55&0.43&0.46&0.57&0.51&0.46&0.48&0.57\\

1212: $M_{68}$&33&34&37&30&31&39&34&31&32&39&0.49&0.50&0.54&0.44&0.46&0.57&0.50&0.46&0.47&0.57\\

1213: $M_{69}$&33&35&38&30&32&40&34&31&32&40&0.48&0.51&0.55&0.43&0.46&0.58&0.49&0.45&0.46&0.58\\

1214: $M_{70}$&34&36&39&31&32&40&34&31&32&41&0.49&0.51&0.56&0.44&0.46&0.57&0.49&0.44&0.46&0.59\\

1215: $M_{71}$&34&36&40&31&33&40&35&32&33&41&0.48&0.51&0.56&0.44&0.46&0.56&0.49&0.45&0.46&0.58\\

1216: $M_{72}$&35&37&40&32&34&40&35&33&33&41&0.49&0.51&0.56&0.44&0.47&0.56&0.49&0.46&0.46&0.57\\

1217: $M_{73}$&36&38&40&32&35&40&36&34&33&41&0.49&0.52&0.55&0.44&0.48&0.55&0.49&0.47&0.45&0.56\\

1218: $M_{74}$&37&38&40&32&36&40&37&35&34&41&0.50&0.51&0.54&0.43&0.49&0.54&0.50&0.47&0.46&0.55\\

1219: $M_{75}$&37&39&41&32&36&41&37&35&35&42&0.49&0.52&0.55&0.43&0.48&0.55&0.49&0.47&0.47&0.56\\

1220: $M_{76}$&38&40&41&33&37&41&37&35&36&42&0.50&0.53&0.54&0.43&0.49&0.54&0.49&0.46&0.47&0.55\\

1221: $M_{77}$&38&41&41&33&38&42&38&35&36&43&0.49&0.53&0.53&0.43&0.49&0.55&0.49&0.45&0.47&0.56\\

1222: $M_{78}$&39&42&41&33&38&42&38&36&37&44&0.50&0.54&0.53&0.42&0.49&0.54&0.49&0.46&0.47&0.56\\

1223: $M_{79}$&40&43&41&34&38&42&39&37&37&44&0.51&0.54&0.52&0.43&0.48&0.53&0.49&0.47&0.47&0.56\\

1224: $M_{80}$&40&44&42&34&39&42&39&38&38&44&0.50&0.55&0.53&0.42&0.49&0.53&0.49&0.47&0.47&0.55\\

1225: $M_{81}$&40&44&43&34&39&43&39&39&39&45&0.49&0.54&0.53&0.42&0.48&0.53&0.48&0.48&0.48&0.56\\

1226: $M_{82}$&41&45&43&34&40&43&39&39&40&46&0.50&0.55&0.52&0.41&0.49&0.52&0.48&0.48&0.49&0.56\\

1227: $M_{83}$&42&45&43&35&40&44&40&40&40&46&0.51&0.54&0.52&0.42&0.48&0.53&0.48&0.48&0.48&0.55\\

1228: $M_{84}$&42&46&44&36&41&44&40&40&40&47&0.50&0.55&0.52&0.43&0.49&0.52&0.48&0.48&0.48&0.56\\

1229: \hline

1230: \end{tabular}

1231: }

1232: \end{center}

1233: \end{table}

1234: 

1235: \clearpage

1236: \begin{table}[htbp]

1237: \center\parbox{5in}{(Cont'd) Ratio of coverage of each point $q$ by

1238: members of $M_t$ as $M_t$ expands.}

1239: {\tiny

1240: \begin{tabular}{||l|cccccccccc|cccccccccc||}\hline

1241:       & \multicolumn{10}{c|}{$N(M_t,q)$}

1242:       & \multicolumn{10}{c||}{$Y(M_t,q)$} \\ \hline

1243: $M_t$ & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1244:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$

1245:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1246:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1247: $M_{85}$&42&46&45&36&42&45&40&41&41&47&0.49&0.54&0.53&0.42&0.49&0.53&0.47&0.48&0.48&0.55\\

1248: $M_{86}$&42&46&46&37&42&46&41&41&42&47&0.49&0.53&0.53&0.43&0.49&0.53&0.48&0.48&0.49&0.55\\

1249: $M_{87}$&43&47&47&37&42&46&42&42&42&47&0.49&0.54&0.54&0.43&0.48&0.53&0.48&0.48&0.48&0.54\\

1250: $M_{88}$&44&48&48&37&42&47&42&43&42&47&0.50&0.55&0.55&0.42&0.48&0.53&0.48&0.49&0.48&0.53\\

1251: $M_{89}$&45&48&48&37&42&48&43&44&42&48&0.51&0.54&0.54&0.42&0.47&0.54&0.48&0.49&0.47&0.54\\

1252: $M_{90}$&45&48&49&37&43&49&43&44&43&49&0.50&0.53&0.54&0.41&0.48&0.54&0.48&0.49&0.48&0.54\\

1253: $M_{91}$&46&48&49&37&44&50&43&44&44&50&0.51&0.53&0.54&0.41&0.48&0.55&0.47&0.48&0.48&0.55\\

1254: $M_{92}$&46&49&50&37&45&50&44&45&44&50&0.50&0.53&0.54&0.40&0.49&0.54&0.48&0.49&0.48&0.54\\

1255: $M_{93}$&46&50&50&38&45&51&44&46&45&50&0.49&0.54&0.54&0.41&0.48&0.55&0.47&0.49&0.48&0.54\\

1256: $M_{94}$&47&50&51&39&45&51&45&46&45&51&0.50&0.53&0.54&0.41&0.48&0.54&0.48&0.49&0.48&0.54\\

1257: $M_{95}$&47&51&52&39&46&51&46&46&45&52&0.49&0.54&0.55&0.41&0.48&0.54&0.48&0.48&0.47&0.55\\

1258: $M_{96}$&48&51&52&39&47&52&47&47&45&52&0.50&0.53&0.54&0.41&0.49&0.54&0.49&0.49&0.47&0.54\\

1259: $M_{97}$&48&52&52&39&48&52&48&48&45&53&0.49&0.54&0.54&0.40&0.49&0.54&0.49&0.49&0.46&0.55\\

1260: $M_{98}$&48&53&52&40&49&52&49&49&45&53&0.49&0.54&0.53&0.41&0.50&0.53&0.50&0.50&0.46&0.54\\

1261: $M_{99}$&48&53&52&40&50&53&50&50&46&53&0.48&0.54&0.53&0.40&0.51&0.54&0.51&0.51&0.46&0.54\\

1262: $M_{100}$&49&53&52&41&51&53&51&50&46&54&0.49&0.53&0.52&0.41&0.51&0.53&0.51&0.50&0.46&0.54\\

1263: $M_{101}$&49&53&52&42&52&53&51&51&47&55&0.49&0.52&0.51&0.42&0.51&0.52&0.50&0.50&0.47&0.54\\

1264: $M_{102}$&49&53&52&42&53&54&52&52&47&56&0.48&0.52&0.51&0.41&0.52&0.53&0.51&0.51&0.46&0.55\\

1265: $M_{103}$&50&54&52&43&53&55&52&52&48&56&0.49&0.52&0.50&0.42&0.51&0.53&0.50&0.50&0.47&0.54\\

1266: $M_{104}$&51&55&52&44&53&55&52&53&48&57&0.49&0.53&0.50&0.42&0.51&0.53&0.50&0.51&0.46&0.55\\

1267: $M_{105}$&52&56&53&45&53&55&53&53&48&57&0.50&0.53&0.50&0.43&0.50&0.52&0.50&0.50&0.46&0.54\\

1268: $M_{106}$&53&57&53&45&53&55&54&54&48&58&0.50&0.54&0.50&0.42&0.50&0.52&0.51&0.51&0.45&0.55\\

1269: $M_{107}$&53&58&53&46&53&55&55&54&49&59&0.50&0.54&0.50&0.43&0.50&0.51&0.51&0.50&0.46&0.55\\

1270: $M_{108}$&53&59&54&46&54&55&55&55&49&60&0.49&0.55&0.50&0.43&0.50&0.51&0.51&0.51&0.45&0.56\\

1271: $M_{109}$&53&60&54&46&55&56&56&55&50&60&0.49&0.55&0.50&0.42&0.50&0.51&0.51&0.50&0.46&0.55\\

1272: $M_{110}$&53&61&54&46&55&57&57&55&51&61&0.48&0.55&0.49&0.42&0.50&0.52&0.52&0.50&0.46&0.55\\

1273: $M_{111}$&54&62&55&46&55&58&57&55&52&61&0.49&0.56&0.50&0.41&0.50&0.52&0.51&0.50&0.47&0.55\\

1274: $M_{112}$&54&63&56&47&55&58&57&55&53&62&0.48&0.56&0.50&0.42&0.49&0.52&0.51&0.49&0.47&0.55\\

1275: $M_{113}$&55&63&56&48&55&59&58&55&54&62&0.49&0.56&0.50&0.42&0.49&0.52&0.51&0.49&0.48&0.55\\

1276: $M_{114}$&55&63&57&49&55&59&59&55&55&63&0.48&0.55&0.50&0.43&0.48&0.52&0.52&0.48&0.48&0.55\\

1277: $M_{115}$&55&63&58&50&56&59&59&56&56&63&0.48&0.55&0.50&0.43&0.49&0.51&0.51&0.49&0.49&0.55\\

1278: $M_{116}$&55&63&58&51&57&60&60&56&57&63&0.47&0.54&0.50&0.44&0.49&0.52&0.52&0.48&0.49&0.54\\

1279: $M_{117}$&55&63&59&52&57&61&61&56&57&64&0.47&0.54&0.50&0.44&0.49&0.52&0.52&0.48&0.49&0.55\\

1280: $M_{118}$&55&64&59&52&58&61&62&56&58&65&0.47&0.54&0.50&0.44&0.49&0.52&0.53&0.47&0.49&0.55\\

1281: $M_{119}$&55&64&60&53&58&61&62&57&59&66&0.46&0.54&0.50&0.45&0.49&0.51&0.52&0.48&0.50&0.55\\

1282: $M_{120}$&56&65&61&53&59&61&63&57&59&66&0.47&0.54&0.51&0.44&0.49&0.51&0.53&0.47&0.49&0.55\\

1283: $M_{121}$&56&65&62&54&59&62&63&58&59&67&0.46&0.54&0.51&0.45&0.49&0.51&0.52&0.48&0.49&0.55\\

1284: $M_{122}$&57&66&62&54&60&63&64&58&59&67&0.47&0.54&0.51&0.44&0.49&0.52&0.52&0.48&0.48&0.55\\

1285: $M_{123}$&57&66&63&55&61&64&65&58&59&67&0.46&0.54&0.51&0.45&0.50&0.52&0.53&0.47&0.48&0.54\\

1286: $M_{124}$&58&66&63&56&61&64&65&59&60&68&0.47&0.53&0.51&0.45&0.49&0.52&0.52&0.48&0.48&0.55\\

1287: $M_{125}$&59&66&63&56&61&65&66&60&61&68&0.47&0.53&0.50&0.45&0.49&0.52&0.53&0.48&0.49&0.54\\

1288: $M_{126}$&59&67&63&57&61&65&67&61&62&68&0.47&0.53&0.50&0.45&0.48&0.52&0.53&0.48&0.49&0.54\\

1289: $M_{127}$&60&68&63&57&62&65&68&61&62&69&0.47&0.54&0.50&0.45&0.49&0.51&0.54&0.48&0.49&0.54\\

1290: $M_{128}$&61&68&63&58&62&65&69&62&62&70&0.48&0.53&0.49&0.45&0.48&0.51&0.54&0.48&0.48&0.55\\

1291: $M_{129}$&62&68&63&58&63&66&69&63&62&71&0.48&0.53&0.49&0.45&0.49&0.51&0.53&0.49&0.48&0.55\\

1292: $M_{130}$&63&69&64&59&63&66&69&64&62&71&0.48&0.53&0.49&0.45&0.48&0.51&0.53&0.49&0.48&0.55\\

1293: $M_{131}$&63&69&65&59&64&66&69&65&63&72&0.48&0.53&0.50&0.45&0.49&0.50&0.53&0.50&0.48&0.55\\

1294: $M_{132}$&63&69&65&59&65&67&70&65&64&73&0.48&0.52&0.49&0.45&0.49&0.51&0.53&0.49&0.48&0.55\\

1295: $M_{133}$&63&70&65&59&65&67&71&66&65&74&0.47&0.53&0.49&0.44&0.49&0.50&0.53&0.50&0.49&0.56\\

1296: $M_{134}$&63&71&65&60&66&67&71&67&65&75&0.47&0.53&0.49&0.45&0.49&0.50&0.53&0.50&0.49&0.56\\

1297: $M_{135}$&64&71&66&61&67&67&71&67&65&76&0.47&0.53&0.49&0.45&0.50&0.50&0.53&0.50&0.48&0.56\\

1298: $M_{136}$&64&72&66&62&68&67&72&67&65&77&0.47&0.53&0.49&0.46&0.50&0.49&0.53&0.49&0.48&0.57\\

1299: $M_{137}$&65&72&66&63&68&67&73&68&66&77&0.47&0.53&0.48&0.46&0.50&0.49&0.53&0.50&0.48&0.56\\

1300: $M_{138}$&65&72&67&64&68&67&74&69&66&78&0.47&0.52&0.49&0.46&0.49&0.49&0.54&0.50&0.48&0.57\\

1301: $M_{139}$&65&72&67&64&69&67&75&70&67&79&0.47&0.52&0.48&0.46&0.50&0.48&0.54&0.50&0.48&0.57\\

1302: $M_{140}$&66&73&67&64&70&67&76&70&68&79&0.47&0.52&0.48&0.46&0.50&0.48&0.54&0.50&0.49&0.56\\

1303: $M_{141}$&67&73&67&65&70&67&77&70&69&80&0.48&0.52&0.48&0.46&0.50&0.48&0.55&0.50&0.49&0.57\\

1304: $M_{142}$&68&73&68&65&71&67&77&71&70&80&0.48&0.51&0.48&0.46&0.50&0.47&0.54&0.50&0.49&0.56\\

1305: $M_{143}$&68&73&69&66&72&68&77&72&70&80&0.48&0.51&0.48&0.46&0.50&0.48&0.54&0.50&0.49&0.56\\

1306: $M_{144}$&69&73&70&67&73&68&77&73&70&80&0.48&0.51&0.49&0.47&0.51&0.47&0.53&0.51&0.49&0.56\\

1307: $M_{145}$&70&74&71&67&73&68&77&73&71&81&0.48&0.51&0.49&0.46&0.50&0.47&0.53&0.50&0.49&0.56\\

1308: $M_{146}$&71&75&71&68&73&68&78&73&71&82&0.49&0.51&0.49&0.47&0.50&0.47&0.53&0.50&0.49&0.56\\

1309: $M_{147}$&72&76&71&69&73&69&79&73&71&82&0.49&0.52&0.48&0.47&0.50&0.47&0.54&0.50&0.48&0.56\\

1310: $M_{148}$&72&77&72&70&73&69&79&74&71&83&0.49&0.52&0.49&0.47&0.49&0.47&0.53&0.50&0.48&0.56\\

1311: $M_{149}$&73&77&73&70&73&70&80&74&71&84&0.49&0.52&0.49&0.47&0.49&0.47&0.54&0.50&0.48&0.56\\

1312: $M_{150}$&73&77&73&71&74&70&81&75&72&84&0.49&0.51&0.49&0.47&0.49&0.47&0.54&0.50&0.48&0.56\\

1313: $M_{151}$&73&77&74&71&75&71&82&75&72&85&0.48&0.51&0.49&0.47&0.50&0.47&0.54&0.50&0.48&0.56\\

1314: $M_{152}$&74&77&74&72&75&72&82&76&73&85&0.49&0.51&0.49&0.47&0.49&0.47&0.54&0.50&0.48&0.56\\

1315: $M_{153}$&75&77&75&73&75&73&82&76&73&86&0.49&0.50&0.49&0.48&0.49&0.48&0.54&0.50&0.48&0.56\\

1316: $M_{154}$&76&78&76&74&76&73&82&76&73&86&0.49&0.51&0.49&0.48&0.49&0.47&0.53&0.49&0.47&0.56\\

1317: $M_{155}$&77&79&76&75&77&74&82&76&73&86&0.50&0.51&0.49&0.48&0.50&0.48&0.53&0.49&0.47&0.55\\

1318: $M_{156}$&78&79&77&76&78&74&82&76&74&86&0.50&0.51&0.49&0.49&0.50&0.47&0.53&0.49&0.47&0.55\\

1319: $M_{157}$&79&79&77&77&79&75&82&77&74&86&0.50&0.50&0.49&0.49&0.50&0.48&0.52&0.49&0.47&0.55\\

1320: $M_{158}$&80&79&78&78&79&76&82&78&74&86&0.51&0.50&0.49&0.49&0.50&0.48&0.52&0.49&0.47&0.54\\

1321: $M_{159}$&81&80&79&79&79&77&82&78&74&86&0.51&0.50&0.50&0.50&0.50&0.48&0.52&0.49&0.47&0.54\\

1322: $M_{160}$&82&81&79&80&79&77&82&79&75&86&0.51&0.51&0.49&0.50&0.49&0.48&0.51&0.49&0.47&0.54\\

1323: $M_{161}$&82&82&79&80&80&78&83&80&75&86&0.51&0.51&0.49&0.50&0.50&0.48&0.52&0.50&0.47&0.53\\

1324: $M_{162}$&82&82&80&81&81&79&83&80&76&86&0.51&0.51&0.49&0.50&0.50&0.49&0.51&0.49&0.47&0.53\\

1325: $M_{163}$&82&82&80&81&81&80&84&81&77&87&0.50&0.50&0.49&0.50&0.50&0.49&0.52&0.50&0.47&0.53\\

1326: $M_{164}$&82&82&80&82&82&81&85&82&77&87&0.50&0.50&0.49&0.50&0.50&0.49&0.52&0.50&0.47&0.53\\

1327: $M_{165}$&83&83&81&82&83&81&85&82&77&88&0.50&0.50&0.49&0.50&0.50&0.49&0.52&0.50&0.47&0.53\\

1328: $M_{166}$&84&83&81&83&84&81&85&82&78&89&0.51&0.50&0.49&0.50&0.51&0.49&0.51&0.49&0.47&0.54\\

1329: $M_{167}$&85&83&82&84&84&81&85&82&79&90&0.51&0.50&0.49&0.50&0.50&0.49&0.51&0.49&0.47&0.54\\

1330: $M_{168}$&85&84&83&85&84&81&85&83&80&90&0.51&0.50&0.49&0.51&0.50&0.48&0.51&0.49&0.48&0.54\\

1331: \hline

1332: \end{tabular}

1333: }

1334: \end{table}

1335: 

1336: \clearpage

1337: \begin{table}[htbp]

1338: \center\parbox{5in}{(Cont'd) Ratio of coverage of each point $q$ by

1339: members of $M_t$ as $M_t$ expands.}

1340: {\tiny

1341: \begin{tabular}{||l|cccccccccc|cccccccccc||}\hline

1342:       & \multicolumn{10}{c|}{$N(M_t,q)$}

1343:       & \multicolumn{10}{c||}{$Y(M_t,q)$} \\ \hline

1344: $M_t$ & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1345:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$

1346:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1347:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1348: $M_{169}$&86&84&84&85&85&81&86&83&81&90&0.51&0.50&0.50&0.50&0.50&0.48&0.51&0.49&0.48&0.53\\

1349: $M_{170}$&86&84&84&86&85&82&87&84&82&90&0.51&0.49&0.49&0.51&0.50&0.48&0.51&0.49&0.48&0.53\\

1350: $M_{171}$&87&84&84&87&85&83&87&84&83&91&0.51&0.49&0.49&0.51&0.50&0.49&0.51&0.49&0.49&0.53\\

1351: $M_{172}$&87&84&84&88&86&83&88&85&83&92&0.51&0.49&0.49&0.51&0.50&0.48&0.51&0.49&0.48&0.53\\

1352: $M_{173}$&87&85&85&89&87&83&89&85&83&92&0.50&0.49&0.49&0.51&0.50&0.48&0.51&0.49&0.48&0.53\\

1353: $M_{174}$&87&86&86&89&88&84&89&85&84&92&0.50&0.49&0.49&0.51&0.51&0.48&0.51&0.49&0.48&0.53\\

1354: $M_{175}$&87&86&86&90&88&85&89&86&85&93&0.50&0.49&0.49&0.51&0.50&0.49&0.51&0.49&0.49&0.53\\

1355: $M_{176}$&88&86&87&91&88&86&89&86&86&93&0.50&0.49&0.49&0.52&0.50&0.49&0.51&0.49&0.49&0.53\\

1356: $M_{177}$&88&86&87&92&88&87&90&87&86&94&0.50&0.49&0.49&0.52&0.50&0.49&0.51&0.49&0.49&0.53\\

1357: $M_{178}$&88&87&87&93&89&88&90&87&87&94&0.49&0.49&0.49&0.52&0.50&0.49&0.51&0.49&0.49&0.53\\

1358: $M_{179}$&89&88&87&93&90&89&90&87&87&95&0.50&0.49&0.49&0.52&0.50&0.50&0.50&0.49&0.49&0.53\\

1359: $M_{180}$&90&88&87&94&91&89&90&88&87&96&0.50&0.49&0.48&0.52&0.51&0.49&0.50&0.49&0.48&0.53\\

1360: $M_{181}$&90&89&87&94&92&89&90&89&88&97&0.50&0.49&0.48&0.52&0.51&0.49&0.50&0.49&0.49&0.54\\

1361: $M_{182}$&90&89&88&95&92&89&91&90&89&97&0.49&0.49&0.48&0.52&0.51&0.49&0.50&0.49&0.49&0.53\\

1362: $M_{183}$&91&89&88&96&93&90&92&90&89&97&0.50&0.49&0.48&0.52&0.51&0.49&0.50&0.49&0.49&0.53\\

1363: $M_{184}$&91&90&88&97&94&91&93&90&89&97&0.49&0.49&0.48&0.53&0.51&0.49&0.51&0.49&0.48&0.53\\

1364: $M_{185}$&92&91&88&97&94&92&94&90&90&97&0.50&0.49&0.48&0.52&0.51&0.50&0.51&0.49&0.49&0.52\\

1365: $M_{186}$&93&92&88&97&94&93&94&91&91&97&0.50&0.49&0.47&0.52&0.51&0.50&0.51&0.49&0.49&0.52\\

1366: $M_{187}$&94&93&88&97&94&93&95&92&92&97&0.50&0.50&0.47&0.52&0.50&0.50&0.51&0.49&0.49&0.52\\

1367: $M_{188}$&94&94&89&98&94&93&96&93&92&97&0.50&0.50&0.47&0.52&0.50&0.49&0.51&0.49&0.49&0.52\\

1368: $M_{189}$&94&95&90&99&95&94&96&93&92&97&0.50&0.50&0.48&0.52&0.50&0.50&0.51&0.49&0.49&0.51\\

1369: $M_{190}$&94&95&91&99&95&95&97&94&92&98&0.49&0.50&0.48&0.52&0.50&0.50&0.51&0.49&0.48&0.52\\

1370: $M_{191}$&95&95&92&100&95&95&98&95&92&98&0.50&0.50&0.48&0.52&0.50&0.50&0.51&0.50&0.48&0.51\\

1371: $M_{192}$&96&96&93&100&95&96&99&95&92&98&0.50&0.50&0.48&0.52&0.49&0.50&0.52&0.49&0.48&0.51\\

1372: $M_{193}$&96&97&93&101&95&96&100&96&92&99&0.50&0.50&0.48&0.52&0.49&0.50&0.52&0.50&0.48&0.51\\

1373: $M_{194}$&97&97&93&101&96&96&101&96&93&100&0.50&0.50&0.48&0.52&0.49&0.49&0.52&0.49&0.48&0.52\\

1374: $M_{195}$&98&97&93&101&97&97&102&96&94&100&0.50&0.50&0.48&0.52&0.50&0.50&0.52&0.49&0.48&0.51\\

1375: $M_{196}$&98&98&94&101&97&98&102&97&95&100&0.50&0.50&0.48&0.52&0.49&0.50&0.52&0.49&0.48&0.51\\

1376: $M_{197}$&98&99&95&101&98&98&103&97&96&100&0.50&0.50&0.48&0.51&0.50&0.50&0.52&0.49&0.49&0.51\\

1377: $M_{198}$&99&99&95&102&99&98&104&97&97&100&0.50&0.50&0.48&0.52&0.50&0.49&0.53&0.49&0.49&0.51\\

1378: $M_{199}$&99&99&95&103&100&99&105&97&97&101&0.50&0.50&0.48&0.52&0.50&0.50&0.53&0.49&0.49&0.51\\

1379: $M_{200}$&99&100&96&104&100&99&106&97&97&102&0.49&0.50&0.48&0.52&0.50&0.49&0.53&0.48&0.48&0.51\\

1380: $M_{201}$&99&101&96&105&101&99&106&97&98&103&0.49&0.50&0.48&0.52&0.50&0.49&0.53&0.48&0.49&0.51\\

1381: $M_{202}$&99&102&97&105&101&100&107&98&98&103&0.49&0.50&0.48&0.52&0.50&0.50&0.53&0.49&0.49&0.51\\

1382: $M_{203}$&100&102&98&105&102&100&107&98&99&104&0.49&0.50&0.48&0.52&0.50&0.49&0.53&0.48&0.49&0.51\\

1383: $M_{204}$&101&102&99&105&102&100&108&99&100&104&0.50&0.50&0.49&0.51&0.50&0.49&0.53&0.49&0.49&0.51\\

1384: $M_{205}$&101&103&99&106&102&101&109&100&100&104&0.49&0.50&0.48&0.52&0.50&0.49&0.53&0.49&0.49&0.51\\

1385: $M_{206}$&102&104&99&107&102&102&109&101&100&104&0.50&0.50&0.48&0.52&0.50&0.50&0.53&0.49&0.49&0.50\\

1386: $M_{207}$&103&105&100&107&102&102&109&102&101&104&0.50&0.51&0.48&0.52&0.49&0.49&0.53&0.49&0.49&0.50\\

1387: $M_{208}$&104&105&101&107&102&103&109&103&102&104&0.50&0.50&0.49&0.51&0.49&0.50&0.52&0.50&0.49&0.50\\

1388: $M_{209}$&104&106&102&108&103&103&109&103&103&104&0.50&0.51&0.49&0.52&0.49&0.49&0.52&0.49&0.49&0.50\\

1389: $M_{210}$&105&107&103&108&103&103&109&104&103&105&0.50&0.51&0.49&0.51&0.49&0.49&0.52&0.50&0.49&0.50\\

1390: $M_{211}$&106&107&104&108&104&104&109&104&104&105&0.50&0.51&0.49&0.51&0.49&0.49&0.52&0.49&0.49&0.50\\

1391: $M_{212}$&106&108&104&109&105&105&109&105&104&105&0.50&0.51&0.49&0.51&0.50&0.50&0.51&0.50&0.49&0.50\\

1392: $M_{213}$&106&108&105&109&106&105&110&105&105&106&0.50&0.51&0.49&0.51&0.50&0.49&0.52&0.49&0.49&0.50\\

1393: $M_{214}$&107&108&105&110&107&105&110&106&106&106&0.50&0.50&0.49&0.51&0.50&0.49&0.51&0.50&0.50&0.50\\

1394: $M_{215}$&107&108&106&111&107&106&110&106&107&107&0.50&0.50&0.49&0.52&0.50&0.49&0.51&0.49&0.50&0.50\\

1395: $M_{216}$&107&108&107&111&108&106&111&107&107&108&0.50&0.50&0.50&0.51&0.50&0.49&0.51&0.50&0.50&0.50\\

1396: $M_{217}$&108&108&108&111&109&107&112&107&107&108&0.50&0.50&0.50&0.51&0.50&0.49&0.52&0.49&0.49&0.50\\

1397: $M_{218}$&109&108&108&111&109&108&113&107&108&109&0.50&0.50&0.50&0.51&0.50&0.50&0.52&0.49&0.50&0.50\\

1398: $M_{219}$&109&109&109&111&109&108&113&108&109&110&0.50&0.50&0.50&0.51&0.50&0.49&0.52&0.49&0.50&0.50\\

1399: $M_{220}$&110&109&110&112&110&108&114&108&109&110&0.50&0.50&0.50&0.51&0.50&0.49&0.52&0.49&0.50&0.50\\

1400: $M_{221}$&110&109&111&113&111&108&114&108&110&111&0.50&0.49&0.50&0.51&0.50&0.49&0.52&0.49&0.50&0.50\\

1401: $M_{222}$&110&109&112&114&112&108&115&109&110&111&0.50&0.49&0.50&0.51&0.50&0.49&0.52&0.49&0.50&0.50\\

1402: $M_{223}$&110&110&113&114&113&108&115&109&111&112&0.49&0.49&0.51&0.51&0.51&0.48&0.52&0.49&0.50&0.50\\

1403: $M_{224}$&110&111&113&114&114&109&115&109&112&113&0.49&0.50&0.50&0.51&0.51&0.49&0.51&0.49&0.50&0.50\\

1404: $M_{225}$&110&111&114&114&114&110&116&110&113&113&0.49&0.49&0.51&0.51&0.51&0.49&0.52&0.49&0.50&0.50\\

1405: $M_{226}$&110&112&115&114&114&111&116&111&113&114&0.49&0.50&0.51&0.50&0.50&0.49&0.51&0.49&0.50&0.50\\

1406: $M_{227}$&111&112&115&115&115&112&116&111&114&114&0.49&0.49&0.51&0.51&0.51&0.49&0.51&0.49&0.50&0.50\\

1407: $M_{228}$&112&113&115&115&115&113&117&111&114&115&0.49&0.50&0.50&0.50&0.50&0.50&0.51&0.49&0.50&0.50\\

1408: $M_{229}$&112&113&115&115&116&114&117&112&115&116&0.49&0.49&0.50&0.50&0.51&0.50&0.51&0.49&0.50&0.51\\

1409: $M_{230}$&112&114&116&116&116&115&117&112&116&116&0.49&0.50&0.50&0.50&0.50&0.50&0.51&0.49&0.50&0.50\\

1410: $M_{231}$&113&114&117&116&116&116&117&113&116&117&0.49&0.49&0.51&0.50&0.50&0.50&0.51&0.49&0.50&0.51\\

1411: $M_{232}$&114&115&117&116&117&117&117&114&116&117&0.49&0.50&0.50&0.50&0.50&0.50&0.50&0.49&0.50&0.50\\

1412: $M_{233}$&115&115&117&116&117&118&117&115&117&118&0.49&0.49&0.50&0.50&0.50&0.51&0.50&0.49&0.50&0.51\\

1413: $M_{234}$&116&116&118&116&118&118&117&116&117&118&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1414: $M_{235}$&117&116&118&117&119&118&118&117&117&118&0.50&0.49&0.50&0.50&0.51&0.50&0.50&0.50&0.50&0.50\\

1415: $M_{236}$&117&117&119&118&119&119&118&117&117&119&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1416: $M_{237}$&118&117&120&118&119&120&119&118&117&119&0.50&0.49&0.51&0.50&0.50&0.51&0.50&0.50&0.49&0.50\\

1417: $M_{238}$&118&118&121&119&119&121&120&118&117&119&0.50&0.50&0.51&0.50&0.50&0.51&0.50&0.50&0.49&0.50\\

1418: $M_{239}$&119&118&122&119&120&121&121&118&117&120&0.50&0.49&0.51&0.50&0.50&0.51&0.51&0.49&0.49&0.50\\

1419: $M_{240}$&119&119&122&120&121&121&122&118&118&120&0.50&0.50&0.51&0.50&0.50&0.50&0.51&0.49&0.49&0.50\\

1420: $M_{241}$&120&119&123&120&122&121&122&119&118&121&0.50&0.49&0.51&0.50&0.51&0.50&0.51&0.49&0.49&0.50\\

1421: $M_{242}$&120&119&123&121&122&121&123&120&119&122&0.50&0.49&0.51&0.50&0.50&0.50&0.51&0.50&0.49&0.50\\

1422: $M_{243}$&120&120&123&122&122&122&124&120&120&122&0.49&0.49&0.51&0.50&0.50&0.50&0.51&0.49&0.49&0.50\\

1423: $M_{244}$&121&120&124&122&123&122&125&121&120&122&0.50&0.49&0.51&0.50&0.50&0.50&0.51&0.50&0.49&0.50\\

1424: $M_{245}$&122&121&124&122&123&123&125&121&121&123&0.50&0.49&0.51&0.50&0.50&0.50&0.51&0.49&0.49&0.50\\

1425: $M_{246}$&123&122&124&122&124&123&125&122&122&123&0.50&0.50&0.50&0.50&0.50&0.50&0.51&0.50&0.50&0.50\\

1426: $M_{247}$&123&123&124&122&124&124&125&123&123&124&0.50&0.50&0.50&0.49&0.50&0.50&0.51&0.50&0.50&0.50\\

1427: $M_{248}$&124&124&124&123&125&124&125&123&123&125&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1428: $M_{249}$&125&124&125&124&125&125&126&123&123&125&0.50&0.50&0.50&0.50&0.50&0.50&0.51&0.49&0.49&0.50\\

1429: $M_{250}$&125&125&125&124&126&126&126&124&124&125&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1430: $M_{251}$&125&126&125&125&126&126&126&125&125&126&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1431: $M_{252}$&126&126&126&126&126&126&126&126&126&126&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50&0.50\\

1432: \hline

1433: \end{tabular}

1434: }

1435: \end{table}

1436: 

1437: \begin{table}[htbp]

1438: \caption{Changes of $Y_{12}(q,M_t)$ as $M_t$ expands.  For each t, we show the

1439: ratings for each new member of $M_t$, the values $X_{12}$ for this new

1440: member, and $Y_{12}$ for the collection $M_t$ up to the inclusion of

1441: this new member.}

1442: \vspace{-0.15in}

1443: \center

1444: {\tiny

1445: \begin{tabular}{||c|c|c|c|c|c|c|cccccccccc||}\hline

1446:       &       &       &       &           & 

1447:         \multicolumn{2}{c|}{$X_{12}(q,m_t)$ if} &

1448:         \multicolumn{10}{c||}{$Y_{12}(q,M_t)$} \\ \cline{6-7}\cline{8-17}

1449: $M_t$ & $m_t$ & $r_1$ & $r_2$ & $r_1-r_2$ & $q\in m_t$ & $q \not\in m_t$ 

1450:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1451:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1452: $M_{1}$&$m_{1}$&0.20&0.80&-0.60&-0.33&1.33  &1.33&1.33&1.33&-0.33&1.33&-0.33&-0.33&1.33&-0.33&-0.33\\

1453: $M_{2}$&$m_{2}$&0.80&0.20&0.60&1.33&-0.33  &1.33&1.33&1.33&-0.33&0.50&-0.33&0.50&0.50&0.50&-0.33\\

1454: $M_{3}$&$m_{3}$&0.60&0.40&0.20&3.00&-2.00  &1.89&0.22&0.22&-0.89&1.33&-0.89&-0.33&1.33&1.33&0.78\\

1455: $M_{4}$&$m_{4}$&0.40&0.60&-0.20&-2.00&3.00  &2.17&0.92&-0.33&0.08&1.75&-1.17&-0.75&1.75&0.50&0.08\\

1456: $M_{5}$&$m_{5}$&0.60&0.40&0.20&3.00&-2.00  &2.33&0.33&0.33&-0.33&1.00&-1.33&0.00&2.00&0.00&0.67\\

1457: $M_{6}$&$m_{6}$&0.40&0.60&-0.20&-2.00&3.00  &2.44&0.78&0.78&-0.61&0.50&-1.44&0.50&1.33&-0.33&1.06\\

1458: $M_{7}$&$m_{7}$&0.20&0.80&-0.60&-0.33&1.33  &2.28&0.62&0.86&-0.57&0.38&-1.28&0.62&1.33&-0.10&0.86\\

1459: $M_{8}$&$m_{8}$&0.80&0.20&0.60&1.33&-0.33  &2.17&0.71&0.92&-0.33&0.29&-1.17&0.50&1.12&0.08&0.71\\

1460: $M_{9}$&$m_{9}$&0.60&0.40&0.20&3.00&-2.00  &1.70&0.96&1.15&0.04&0.59&-1.26&0.22&1.33&-0.15&0.41\\

1461: $M_{10}$&$m_{10}$&0.60&0.40&0.20&3.00&-2.00  &1.83&1.17&0.83&-0.17&0.33&-0.83&0.00&1.50&-0.33&0.67\\

1462: $M_{11}$&$m_{11}$&0.20&0.80&-0.60&-0.33&1.33  &1.79&1.18&0.88&-0.18&0.27&-0.79&0.12&1.48&-0.33&0.58\\

1463: $M_{12}$&$m_{12}$&0.20&0.80&-0.60&-0.33&1.33  &1.61&1.19&0.92&-0.19&0.22&-0.75&0.22&1.47&-0.19&0.50\\

1464: $M_{13}$&$m_{13}$&0.20&0.80&-0.60&-0.33&1.33  &1.59&1.20&0.82&-0.20&0.18&-0.72&0.31&1.46&-0.08&0.44\\

1465: $M_{14}$&$m_{14}$&0.60&0.40&0.20&3.00&-2.00  &1.69&0.97&0.97&-0.33&0.38&-0.45&0.14&1.57&-0.21&0.26\\

1466: $M_{15}$&$m_{15}$&0.60&0.40&0.20&3.00&-2.00  &1.78&0.78&1.11&-0.11&0.22&-0.55&0.33&1.33&0.00&0.11\\

1467: $M_{16}$&$m_{16}$&0.60&0.40&0.20&3.00&-2.00  &1.85&0.60&1.23&-0.23&0.08&-0.64&0.50&1.12&0.19&0.29\\

1468: $M_{17}$&$m_{17}$&0.60&0.40&0.20&3.00&-2.00  &1.92&0.74&1.04&-0.04&-0.04&-0.72&0.65&0.94&0.35&0.16\\

1469: $M_{18}$&$m_{18}$&0.40&0.60&-0.20&-2.00&3.00  &1.98&0.59&1.15&-0.15&0.13&-0.80&0.78&1.05&0.22&0.04\\

1470: $M_{19}$&$m_{19}$&0.40&0.60&-0.20&-2.00&3.00  &2.03&0.46&1.24&0.02&0.02&-0.86&0.89&0.89&0.37&-0.07\\

1471: $M_{20}$&$m_{20}$&0.40&0.60&-0.20&-2.00&3.00  &2.08&0.58&1.08&-0.08&-0.08&-0.67&0.75&1.00&0.25&0.08\\

1472: $M_{21}$&$m_{21}$&0.60&0.40&0.20&3.00&-2.00  &1.89&0.46&1.17&-0.17&-0.17&-0.73&0.86&1.09&0.38&0.22\\

1473: $M_{22}$&$m_{22}$&0.60&0.40&0.20&3.00&-2.00  &1.71&0.57&1.03&-0.26&-0.26&-0.56&0.95&1.18&0.50&0.12\\

1474: $M_{23}$&$m_{23}$&0.60&0.40&0.20&3.00&-2.00  &1.77&0.46&0.90&-0.33&-0.12&-0.40&0.83&1.26&0.61&0.03\\

1475: $M_{24}$&$m_{24}$&0.40&0.60&-0.20&-2.00&3.00  &1.61&0.57&0.99&-0.19&-0.19&-0.26&0.71&1.12&0.71&-0.05\\

1476: $M_{25}$&$m_{25}$&0.40&0.60&-0.20&-2.00&3.00  &1.47&0.67&0.87&-0.07&-0.27&-0.33&0.80&1.20&0.80&-0.13\\

1477: $M_{26}$&$m_{26}$&0.40&0.60&-0.20&-2.00&3.00  &1.52&0.56&0.76&0.05&-0.14&-0.40&0.69&1.27&0.88&-0.20\\

1478: $M_{27}$&$m_{27}$&0.60&0.40&0.20&3.00&-2.00  &1.58&0.65&0.84&-0.02&-0.21&-0.46&0.78&1.15&0.78&-0.09\\

1479: $M_{28}$&$m_{28}$&0.60&0.40&0.20&3.00&-2.00  &1.63&0.56&0.74&-0.09&-0.27&-0.51&0.86&1.21&0.86&0.02\\

1480: $M_{29}$&$m_{29}$&0.60&0.40&0.20&3.00&-2.00  &1.68&0.64&0.64&-0.16&-0.33&-0.56&0.93&1.10&0.93&0.13\\

1481: $M_{30}$&$m_{30}$&0.80&0.20&0.60&1.33&-0.33  &1.67&0.67&0.67&-0.17&-0.28&-0.55&0.89&1.06&0.94&0.11\\

1482: $M_{31}$&$m_{31}$&0.40&0.60&-0.20&-2.00&3.00  &1.71&0.58&0.58&-0.06&-0.33&-0.60&0.80&1.12&1.01&0.21\\

1483: $M_{32}$&$m_{32}$&0.40&0.60&-0.20&-2.00&3.00  &1.75&0.50&0.66&-0.12&-0.23&-0.65&0.86&1.02&1.07&0.14\\

1484: $M_{33}$&$m_{33}$&0.20&0.80&-0.60&-0.33&1.33  &1.74&0.52&0.68&-0.13&-0.23&-0.59&0.83&1.03&1.03&0.12\\

1485: $M_{34}$&$m_{34}$&0.60&0.40&0.20&3.00&-2.00  &1.63&0.60&0.74&-0.19&-0.28&-0.63&0.89&1.09&0.94&0.21\\

1486: $M_{35}$&$m_{35}$&0.60&0.40&0.20&3.00&-2.00  &1.52&0.67&0.81&-0.24&-0.33&-0.52&0.95&1.00&1.00&0.14\\

1487: $M_{36}$&$m_{36}$&0.20&0.80&-0.60&-0.33&1.33  &1.52&0.68&0.82&-0.24&-0.33&-0.52&0.96&0.96&1.01&0.13\\

1488: $M_{37}$&$m_{37}$&0.60&0.40&0.20&3.00&-2.00  &1.56&0.75&0.75&-0.15&-0.38&-0.56&0.88&0.88&1.06&0.21\\

1489: $M_{38}$&$m_{38}$&0.20&0.80&-0.60&-0.33&1.33  &1.55&0.76&0.72&-0.16&-0.38&-0.51&0.85&0.89&1.07&0.19\\

1490: $M_{39}$&$m_{39}$&0.40&0.60&-0.20&-2.00&3.00  &1.59&0.82&0.65&-0.08&-0.42&-0.55&0.91&0.82&1.12&0.14\\

1491: $M_{40}$&$m_{40}$&0.60&0.40&0.20&3.00&-2.00  &1.62&0.75&0.71&-0.12&-0.46&-0.46&0.83&0.75&1.17&0.21\\

1492: $M_{41}$&$m_{41}$&0.60&0.40&0.20&3.00&-2.00  &1.66&0.80&0.64&-0.17&-0.50&-0.37&0.89&0.80&1.09&0.16\\

1493: $M_{42}$&$m_{42}$&0.60&0.40&0.20&3.00&-2.00  &1.57&0.86&0.58&-0.09&-0.41&-0.41&0.82&0.86&1.13&0.10\\

1494: $M_{43}$&$m_{43}$&0.60&0.40&0.20&3.00&-2.00  &1.49&0.91&0.64&-0.14&-0.45&-0.45&0.87&0.79&1.18&0.17\\

1495: $M_{44}$&$m_{44}$&0.20&0.80&-0.60&-0.33&1.33  &1.45&0.92&0.65&-0.11&-0.45&-0.45&0.84&0.80&1.18&0.16\\

1496: $M_{45}$&$m_{45}$&0.60&0.40&0.20&3.00&-2.00  &1.48&0.96&0.70&-0.15&-0.37&-0.37&0.78&0.74&1.11&0.11\\

1497: $M_{46}$&$m_{46}$&0.60&0.40&0.20&3.00&-2.00  &1.51&1.01&0.64&-0.19&-0.30&-0.30&0.72&0.68&1.15&0.07\\

1498: $M_{47}$&$m_{47}$&0.40&0.60&-0.20&-2.00&3.00  &1.55&0.94&0.69&-0.12&-0.23&-0.33&0.66&0.62&1.19&0.02\\

1499: $M_{48}$&$m_{48}$&0.60&0.40&0.20&3.00&-2.00  &1.47&0.99&0.74&-0.16&-0.16&-0.26&0.60&0.67&1.12&-0.02\\

1500: $M_{49}$&$m_{49}$&0.60&0.40&0.20&3.00&-2.00  &1.50&0.92&0.79&-0.09&-0.20&-0.30&0.55&0.72&1.06&0.04\\

1501: $M_{50}$&$m_{50}$&0.60&0.40&0.20&3.00&-2.00  &1.53&0.87&0.83&-0.13&-0.23&-0.23&0.60&0.67&1.10&0.00\\

1502: $M_{51}$&$m_{51}$&0.60&0.40&0.20&3.00&-2.00  &1.46&0.91&0.88&-0.07&-0.27&-0.17&0.55&0.71&1.04&-0.04\\

1503: $M_{52}$&$m_{52}$&0.60&0.40&0.20&3.00&-2.00  &1.40&0.95&0.82&-0.11&-0.20&-0.20&0.60&0.76&1.08&-0.08\\

1504: $M_{53}$&$m_{53}$&0.80&0.20&0.60&1.33&-0.33  &1.36&0.96&0.83&-0.11&-0.21&-0.21&0.61&0.77&1.08&-0.08\\

1505: $M_{54}$&$m_{54}$&0.40&0.60&-0.20&-2.00&3.00  &1.39&0.99&0.78&-0.15&-0.15&-0.24&0.56&0.72&1.12&-0.02\\

1506: $M_{55}$&$m_{55}$&0.80&0.20&0.60&1.33&-0.33  &1.39&0.97&0.79&-0.15&-0.15&-0.24&0.55&0.73&1.12&0.00\\

1507: $M_{56}$&$m_{56}$&0.40&0.60&-0.20&-2.00&3.00  &1.42&1.01&0.74&-0.09&-0.18&-0.27&0.50&0.68&1.15&0.05\\

1508: $M_{57}$&$m_{57}$&0.20&0.80&-0.60&-0.33&1.33  &1.42&0.98&0.75&-0.10&-0.16&-0.27&0.49&0.69&1.16&0.05\\

1509: $M_{58}$&$m_{58}$&0.60&0.40&0.20&3.00&-2.00  &1.45&1.02&0.79&-0.13&-0.19&-0.22&0.44&0.64&1.10&0.10\\

1510: $M_{59}$&$m_{59}$&0.40&0.60&-0.20&-2.00&3.00  &1.47&1.05&0.74&-0.16&-0.22&-0.16&0.49&0.60&1.14&0.06\\

1511: $M_{60}$&$m_{60}$&0.40&0.60&-0.20&-2.00&3.00  &1.42&1.08&0.78&-0.19&-0.17&-0.19&0.53&0.56&1.17&0.03\\

1512: $M_{61}$&$m_{61}$&0.60&0.40&0.20&3.00&-2.00  &1.36&1.11&0.81&-0.14&-0.20&-0.22&0.57&0.51&1.20&0.00\\

1513: $M_{62}$&$m_{62}$&0.60&0.40&0.20&3.00&-2.00  &1.31&1.06&0.85&-0.09&-0.23&-0.17&0.53&0.55&1.23&-0.04\\

1514: $M_{63}$&$m_{63}$&0.40&0.60&-0.20&-2.00&3.00  &1.25&1.09&0.80&-0.12&-0.25&-0.20&0.57&0.59&1.25&0.01\\

1515: $M_{64}$&$m_{64}$&0.60&0.40&0.20&3.00&-2.00  &1.28&1.12&0.76&-0.15&-0.20&-0.23&0.53&0.63&1.20&0.06\\

1516: $M_{65}$&$m_{65}$&0.20&0.80&-0.60&-0.33&1.33  &1.26&1.13&0.77&-0.15&-0.18&-0.23&0.51&0.64&1.20&0.05\\

1517: $M_{66}$&$m_{66}$&0.40&0.60&-0.20&-2.00&3.00  &1.21&1.08&0.80&-0.18&-0.21&-0.18&0.47&0.68&1.23&0.10\\

1518: $M_{67}$&$m_{67}$&0.40&0.60&-0.20&-2.00&3.00  &1.23&1.11&0.76&-0.13&-0.23&-0.21&0.44&0.71&1.18&0.14\\

1519: $M_{68}$&$m_{68}$&0.40&0.60&-0.20&-2.00&3.00  &1.19&1.06&0.79&-0.16&-0.19&-0.23&0.48&0.74&1.21&0.11\\

1520: $M_{69}$&$m_{69}$&0.40&0.60&-0.20&-2.00&3.00  &1.21&1.02&0.75&-0.12&-0.21&-0.26&0.51&0.78&1.24&0.08\\

1521: $M_{70}$&$m_{70}$&0.60&0.40&0.20&3.00&-2.00  &1.24&1.05&0.79&-0.07&-0.24&-0.29&0.48&0.74&1.19&0.12\\

1522: $M_{71}$&$m_{71}$&0.60&0.40&0.20&3.00&-2.00  &1.19&1.00&0.82&-0.10&-0.19&-0.31&0.51&0.77&1.22&0.09\\

1523: $M_{72}$&$m_{72}$&0.60&0.40&0.20&3.00&-2.00  &1.22&1.03&0.78&-0.05&-0.15&-0.33&0.48&0.80&1.17&0.06\\

1524: $M_{73}$&$m_{73}$&0.60&0.40&0.20&3.00&-2.00  &1.24&1.06&0.74&-0.08&-0.10&-0.36&0.51&0.83&1.13&0.03\\

1525: $M_{74}$&$m_{74}$&0.60&0.40&0.20&3.00&-2.00  &1.27&1.02&0.70&-0.11&-0.06&-0.38&0.55&0.86&1.15&0.01\\

1526: $M_{75}$&$m_{75}$&0.60&0.40&0.20&3.00&-2.00  &1.22&1.04&0.73&-0.13&-0.09&-0.33&0.51&0.82&1.18&0.05\\

1527: $M_{76}$&$m_{76}$&0.60&0.40&0.20&3.00&-2.00  &1.25&1.07&0.70&-0.09&-0.05&-0.35&0.48&0.78&1.20&0.02\\

1528: $M_{77}$&$m_{77}$&0.20&0.80&-0.60&-0.33&1.33  &1.25&1.05&0.71&-0.07&-0.05&-0.35&0.47&0.79&1.20&0.01\\

1529: $M_{78}$&$m_{78}$&0.80&0.20&0.60&1.33&-0.33  &1.25&1.06&0.69&-0.08&-0.06&-0.35&0.46&0.80&1.20&0.03\\

1530: $M_{79}$&$m_{79}$&0.60&0.40&0.20&3.00&-2.00  &1.27&1.08&0.66&-0.04&-0.08&-0.37&0.49&0.83&1.16&0.00\\

1531: $M_{80}$&$m_{80}$&0.80&0.20&0.60&1.33&-0.33  &1.25&1.08&0.67&-0.04&-0.06&-0.37&0.48&0.83&1.17&0.00\\

1532: $M_{81}$&$m_{81}$&0.60&0.40&0.20&3.00&-2.00  &1.21&1.04&0.69&-0.07&-0.09&-0.33&0.45&0.86&1.19&0.04\\

1533: $M_{82}$&$m_{82}$&0.60&0.40&0.20&3.00&-2.00  &1.23&1.07&0.66&-0.09&-0.05&-0.35&0.42&0.82&1.21&0.07\\

1534: $M_{83}$&$m_{83}$&0.40&0.60&-0.20&-2.00&3.00  &1.19&1.09&0.69&-0.11&-0.01&-0.37&0.39&0.79&1.23&0.11\\

1535: $M_{84}$&$m_{84}$&0.40&0.60&-0.20&-2.00&3.00  &1.21&1.06&0.66&-0.13&-0.04&-0.33&0.42&0.82&1.25&0.08\\

1536: \hline

1537: \end{tabular}

1538: }

1539: \end{table}

1540: 

1541: \clearpage

1542: \begin{table}[htbp]

1543: \center\parbox{5in}{(Cont'd) Changes of $Y_{12}(q,M_t)$ as $M_t$

1544: expands.  For each t, we show the ratings for each new member of

1545: $M_t$, the values $X_{12}$ for this new member, and $Y_{12}$ for the

1546: collection $M_t$ up to the inclusion of this new member.}

1547: \center

1548: {\tiny

1549: \begin{tabular}{||c|c|c|c|c|c|c|cccccccccc||}\hline

1550:       &       &       &       &           & 

1551:         \multicolumn{2}{c|}{$X_{12}(q,m_t)$ if} &

1552:         \multicolumn{10}{c|}{$Y_{12}(q,M_t)$} \\ \cline{6-7}\cline{8-17}

1553: $M_t$ & $m_t$ & $r_1$ & $r_2$ & $r_1-r_2$ & $q\in m_t$ & $q \not\in m_t$ 

1554:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1555:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1556: $M_{85}$&$m_{85}$&0.60&0.40&0.20&3.00&-2.00  &1.18&1.02&0.69&-0.16&0.00&-0.29&0.39&0.84&1.27&0.06\\

1557: $M_{86}$&$m_{86}$&0.40&0.60&-0.20&-2.00&3.00  &1.20&1.04&0.65&-0.18&0.04&-0.31&0.36&0.87&1.24&0.09\\

1558: $M_{87}$&$m_{87}$&0.80&0.20&0.60&1.33&-0.33  &1.20&1.05&0.66&-0.18&0.03&-0.31&0.38&0.87&1.22&0.09\\

1559: $M_{88}$&$m_{88}$&0.80&0.20&0.60&1.33&-0.33  &1.20&1.05&0.67&-0.18&0.03&-0.29&0.37&0.88&1.20&0.08\\

1560: $M_{89}$&$m_{89}$&0.40&0.60&-0.20&-2.00&3.00  &1.16&1.07&0.70&-0.15&0.06&-0.31&0.34&0.85&1.22&0.06\\

1561: $M_{90}$&$m_{90}$&0.40&0.60&-0.20&-2.00&3.00  &1.18&1.09&0.67&-0.11&0.04&-0.33&0.37&0.87&1.18&0.04\\

1562: $M_{91}$&$m_{91}$&0.40&0.60&-0.20&-2.00&3.00  &1.15&1.11&0.69&-0.08&0.02&-0.35&0.40&0.89&1.15&0.02\\

1563: $M_{92}$&$m_{92}$&0.60&0.40&0.20&3.00&-2.00  &1.12&1.13&0.72&-0.10&0.05&-0.37&0.43&0.92&1.12&-0.01\\

1564: $M_{93}$&$m_{93}$&0.60&0.40&0.20&3.00&-2.00  &1.08&1.15&0.69&-0.06&0.03&-0.33&0.40&0.94&1.14&-0.03\\

1565: $M_{94}$&$m_{94}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.17&0.66&-0.08&0.06&-0.30&0.38&0.96&1.16&-0.05\\

1566: $M_{95}$&$m_{95}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.14&0.63&-0.05&0.04&-0.26&0.35&0.98&1.18&-0.07\\

1567: $M_{96}$&$m_{96}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.16&0.66&-0.02&0.01&-0.28&0.33&0.95&1.19&-0.04\\

1568: $M_{97}$&$m_{97}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.13&0.68&0.01&-0.01&-0.25&0.30&0.92&1.21&-0.06\\

1569: $M_{98}$&$m_{98}$&0.40&0.60&-0.20&-2.00&3.00  &1.08&1.09&0.70&-0.01&-0.03&-0.21&0.28&0.89&1.23&-0.03\\

1570: $M_{99}$&$m_{99}$&0.40&0.60&-0.20&-2.00&3.00  &1.10&1.11&0.73&0.02&-0.05&-0.23&0.26&0.86&1.20&0.00\\

1571: $M_{100}$&$m_{100}$&0.20&0.80&-0.60&-0.33&1.33  &1.08&1.12&0.73&0.02&-0.05&-0.22&0.25&0.87&1.20&0.00\\

1572: $M_{101}$&$m_{101}$&0.40&0.60&-0.20&-2.00&3.00  &1.10&1.13&0.76&0.00&-0.07&-0.18&0.28&0.84&1.17&-0.02\\

1573: $M_{102}$&$m_{102}$&0.20&0.80&-0.60&-0.33&1.33  &1.10&1.14&0.76&0.01&-0.07&-0.19&0.27&0.83&1.17&-0.02\\

1574: $M_{103}$&$m_{103}$&0.60&0.40&0.20&3.00&-2.00  &1.12&1.15&0.73&0.04&-0.09&-0.15&0.25&0.80&1.19&-0.04\\

1575: $M_{104}$&$m_{104}$&0.60&0.40&0.20&3.00&-2.00  &1.14&1.17&0.71&0.07&-0.11&-0.17&0.23&0.82&1.16&-0.01\\

1576: $M_{105}$&$m_{105}$&0.60&0.40&0.20&3.00&-2.00  &1.16&1.19&0.73&0.10&-0.13&-0.19&0.25&0.79&1.13&-0.03\\

1577: $M_{106}$&$m_{106}$&0.60&0.40&0.20&3.00&-2.00  &1.18&1.21&0.70&0.08&-0.14&-0.21&0.28&0.81&1.10&0.00\\

1578: $M_{107}$&$m_{107}$&0.40&0.60&-0.20&-2.00&3.00  &1.19&1.18&0.73&0.06&-0.11&-0.18&0.26&0.83&1.07&-0.02\\

1579: $M_{108}$&$m_{108}$&0.60&0.40&0.20&3.00&-2.00  &1.16&1.19&0.75&0.04&-0.09&-0.19&0.24&0.85&1.04&0.01\\

1580: $M_{109}$&$m_{109}$&0.40&0.60&-0.20&-2.00&3.00  &1.18&1.16&0.77&0.06&-0.10&-0.21&0.22&0.87&1.01&0.03\\

1581: $M_{110}$&$m_{110}$&0.40&0.60&-0.20&-2.00&3.00  &1.20&1.14&0.79&0.09&-0.08&-0.23&0.20&0.89&0.98&0.02\\

1582: $M_{111}$&$m_{111}$&0.80&0.20&0.60&1.33&-0.33  &1.20&1.14&0.79&0.09&-0.08&-0.21&0.19&0.88&0.99&0.01\\

1583: $M_{112}$&$m_{112}$&0.60&0.40&0.20&3.00&-2.00  &1.17&1.15&0.81&0.11&-0.09&-0.23&0.17&0.86&1.01&0.04\\

1584: $M_{113}$&$m_{113}$&0.40&0.60&-0.20&-2.00&3.00  &1.14&1.17&0.83&0.09&-0.07&-0.24&0.15&0.88&0.98&0.07\\

1585: $M_{114}$&$m_{114}$&0.40&0.60&-0.20&-2.00&3.00  &1.16&1.19&0.81&0.08&-0.04&-0.22&0.13&0.89&0.95&0.05\\

1586: $M_{115}$&$m_{115}$&0.60&0.40&0.20&3.00&-2.00  &1.13&1.16&0.83&0.10&-0.01&-0.23&0.12&0.91&0.97&0.03\\

1587: $M_{116}$&$m_{116}$&0.20&0.80&-0.60&-0.33&1.33  &1.13&1.16&0.83&0.10&-0.02&-0.23&0.11&0.92&0.96&0.04\\

1588: $M_{117}$&$m_{117}$&0.20&0.80&-0.60&-0.33&1.33  &1.13&1.16&0.82&0.09&-0.01&-0.23&0.11&0.92&0.96&0.04\\

1589: $M_{118}$&$m_{118}$&0.40&0.60&-0.20&-2.00&3.00  &1.15&1.14&0.84&0.12&-0.02&-0.21&0.09&0.94&0.94&0.02\\

1590: $M_{119}$&$m_{119}$&0.60&0.40&0.20&3.00&-2.00  &1.12&1.11&0.86&0.14&-0.04&-0.22&0.07&0.95&0.95&0.05\\

1591: $M_{120}$&$m_{120}$&0.60&0.40&0.20&3.00&-2.00  &1.14&1.12&0.87&0.13&-0.01&-0.24&0.10&0.93&0.93&0.03\\

1592: $M_{121}$&$m_{121}$&0.40&0.60&-0.20&-2.00&3.00  &1.15&1.14&0.85&0.11&0.01&-0.25&0.12&0.91&0.95&0.01\\

1593: $M_{122}$&$m_{122}$&0.40&0.60&-0.20&-2.00&3.00  &1.13&1.11&0.87&0.13&-0.01&-0.26&0.10&0.92&0.96&0.04\\

1594: $M_{123}$&$m_{123}$&0.20&0.80&-0.60&-0.33&1.33  &1.13&1.12&0.86&0.13&-0.01&-0.27&0.10&0.93&0.97&0.05\\

1595: $M_{124}$&$m_{124}$&0.60&0.40&0.20&3.00&-2.00  &1.14&1.09&0.84&0.15&-0.02&-0.28&0.08&0.94&0.98&0.07\\

1596: $M_{125}$&$m_{125}$&0.60&0.40&0.20&3.00&-2.00  &1.16&1.07&0.81&0.13&-0.04&-0.25&0.11&0.96&1.00&0.05\\

1597: $M_{126}$&$m_{126}$&0.60&0.40&0.20&3.00&-2.00  &1.13&1.08&0.79&0.16&-0.06&-0.27&0.13&0.98&1.02&0.04\\

1598: $M_{127}$&$m_{127}$&0.40&0.60&-0.20&-2.00&3.00  &1.11&1.06&0.81&0.18&-0.07&-0.24&0.11&0.99&1.03&0.02\\

1599: $M_{128}$&$m_{128}$&0.40&0.60&-0.20&-2.00&3.00  &1.09&1.07&0.83&0.16&-0.05&-0.22&0.10&0.97&1.05&0.01\\

1600: $M_{129}$&$m_{129}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.09&0.84&0.18&-0.06&-0.23&0.12&0.95&1.06&-0.01\\

1601: $M_{130}$&$m_{130}$&0.80&0.20&0.60&1.33&-0.33  &1.06&1.09&0.85&0.19&-0.06&-0.23&0.12&0.95&1.05&-0.01\\

1602: $M_{131}$&$m_{131}$&0.60&0.40&0.20&3.00&-2.00  &1.04&1.07&0.86&0.18&-0.04&-0.24&0.10&0.96&1.07&0.01\\

1603: $M_{132}$&$m_{132}$&0.20&0.80&-0.60&-0.33&1.33  &1.04&1.07&0.87&0.18&-0.04&-0.24&0.10&0.97&1.06&0.01\\

1604: $M_{133}$&$m_{133}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.08&0.84&0.17&-0.06&-0.26&0.12&0.98&1.07&0.03\\

1605: $M_{134}$&$m_{134}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.06&0.86&0.15&-0.07&-0.23&0.14&0.96&1.08&0.02\\

1606: $M_{135}$&$m_{135}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.07&0.84&0.14&-0.09&-0.21&0.16&0.97&1.10&0.00\\

1607: $M_{136}$&$m_{136}$&0.20&0.80&-0.60&-0.33&1.33  &1.01&1.06&0.84&0.13&-0.09&-0.20&0.16&0.98&1.10&0.00\\

1608: $M_{137}$&$m_{137}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.04&0.82&0.15&-0.10&-0.21&0.18&0.99&1.11&-0.02\\

1609: $M_{138}$&$m_{138}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.06&0.80&0.14&-0.08&-0.19&0.16&0.97&1.13&-0.03\\

1610: $M_{139}$&$m_{139}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.07&0.82&0.16&-0.09&-0.17&0.15&0.95&1.11&-0.05\\

1611: $M_{140}$&$m_{140}$&0.60&0.40&0.20&3.00&-2.00  &1.07&1.08&0.80&0.14&-0.07&-0.18&0.17&0.93&1.12&-0.06\\

1612: $M_{141}$&$m_{141}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.10&0.81&0.13&-0.05&-0.16&0.15&0.94&1.10&-0.07\\

1613: $M_{142}$&$m_{142}$&0.80&0.20&0.60&1.33&-0.33  &1.05&1.09&0.82&0.12&-0.04&-0.16&0.15&0.95&1.10&-0.07\\

1614: $M_{143}$&$m_{143}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.10&0.80&0.11&-0.05&-0.17&0.17&0.92&1.11&-0.05\\

1615: $M_{144}$&$m_{144}$&0.60&0.40&0.20&3.00&-2.00  &1.08&1.08&0.81&0.13&-0.03&-0.18&0.15&0.94&1.09&-0.07\\

1616: $M_{145}$&$m_{145}$&0.80&0.20&0.60&1.33&-0.33  &1.08&1.08&0.82&0.13&-0.03&-0.18&0.15&0.93&1.09&-0.06\\

1617: $M_{146}$&$m_{146}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.06&0.83&0.11&-0.01&-0.16&0.14&0.94&1.10&-0.07\\

1618: $M_{147}$&$m_{147}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.04&0.85&0.10&0.01&-0.17&0.12&0.96&1.12&-0.05\\

1619: $M_{148}$&$m_{148}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.05&0.86&0.12&-0.01&-0.19&0.11&0.97&1.10&-0.03\\

1620: $M_{149}$&$m_{149}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.06&0.84&0.14&0.01&-0.20&0.09&0.99&1.11&-0.04\\

1621: $M_{150}$&$m_{150}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.08&0.86&0.12&0.00&-0.18&0.08&0.97&1.09&-0.02\\

1622: $M_{151}$&$m_{151}$&0.20&0.80&-0.60&-0.33&1.33  &1.01&1.08&0.85&0.13&0.00&-0.18&0.08&0.97&1.09&-0.02\\

1623: $M_{152}$&$m_{152}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.06&0.83&0.15&-0.01&-0.16&0.06&0.98&1.10&-0.04\\

1624: $M_{153}$&$m_{153}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.07&0.81&0.14&0.00&-0.17&0.08&1.00&1.12&-0.05\\

1625: $M_{154}$&$m_{154}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.08&0.82&0.15&0.02&-0.18&0.07&0.98&1.09&-0.06\\

1626: $M_{155}$&$m_{155}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.06&0.84&0.14&0.01&-0.19&0.09&0.99&1.11&-0.04\\

1627: $M_{156}$&$m_{156}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.04&0.85&0.16&0.03&-0.20&0.07&0.97&1.12&-0.06\\

1628: $M_{157}$&$m_{157}$&0.40&0.60&-0.20&-2.00&3.00  &0.99&1.06&0.87&0.14&0.02&-0.22&0.09&0.95&1.13&-0.04\\

1629: $M_{158}$&$m_{158}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.04&0.88&0.16&0.00&-0.20&0.08&0.96&1.11&-0.05\\

1630: $M_{159}$&$m_{159}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.05&0.89&0.18&-0.01&-0.18&0.07&0.95&1.09&-0.06\\

1631: $M_{160}$&$m_{160}$&0.80&0.20&0.60&1.33&-0.33  &1.02&1.05&0.89&0.19&-0.01&-0.18&0.06&0.95&1.09&-0.06\\

1632: $M_{161}$&$m_{161}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.03&0.90&0.21&-0.02&-0.19&0.05&0.93&1.11&-0.04\\

1633: $M_{162}$&$m_{162}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.04&0.88&0.19&-0.03&-0.20&0.07&0.94&1.09&-0.02\\

1634: $M_{163}$&$m_{163}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.06&0.89&0.21&-0.02&-0.21&0.06&0.92&1.07&-0.04\\

1635: $M_{164}$&$m_{164}$&0.20&0.80&-0.60&-0.33&1.33  &1.06&1.06&0.90&0.21&-0.02&-0.21&0.05&0.92&1.07&-0.03\\

1636: $M_{165}$&$m_{165}$&0.60&0.40&0.20&3.00&-2.00  &1.07&1.07&0.91&0.19&0.00&-0.22&0.04&0.90&1.05&-0.01\\

1637: $M_{166}$&$m_{166}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.08&0.92&0.18&-0.01&-0.20&0.06&0.91&1.03&-0.02\\

1638: $M_{167}$&$m_{167}$&0.60&0.40&0.20&3.00&-2.00  &1.06&1.06&0.93&0.20&-0.02&-0.21&0.05&0.89&1.04&0.00\\

1639: $M_{168}$&$m_{168}$&0.80&0.20&0.60&1.33&-0.33  &1.06&1.07&0.94&0.20&-0.03&-0.21&0.04&0.90&1.05&-0.01\\

1640: \hline

1641: \end{tabular}

1642: }

1643: \end{table}

1644: 

1645: \clearpage

1646: \begin{table}[htbp]

1647: \center\parbox{5in}{(Cont'd) Changes of $Y_{12}(q,M_t)$ as $M_t$

1648: expands.  For each t, we show the ratings for each new member of

1649: $M_t$, the values $X_{12}$ for this new member, and $Y_{12}$ for the

1650: collection $M_t$ up to the inclusion of this new member.}

1651: \center

1652: {\tiny

1653: \begin{tabular}{||c|c|c|c|c|c|c|cccccccccc||}\hline

1654:       &       &       &       &           & 

1655:         \multicolumn{2}{c|}{$X_{12}(q,m_t)$ if} &

1656:         \multicolumn{10}{c|}{$Y_{12}(q,M_t)$} \\ \cline{6-7}\cline{8-17}

1657: $M_t$ & $m_t$ & $r_1$ & $r_2$ & $r_1-r_2$ & $q\in m_t$ & $q \not\in m_t$ 

1658:       & $q_0$ & $q_1$ & $q_2$ & $q_3$ & $q_4$

1659:       & $q_5$ & $q_6$ & $q_7$ & $q_8$ & $q_9$ \\ \hline

1660: $M_{169}$&$m_{169}$&0.60&0.40&0.20&3.00&-2.00  &1.07&1.05&0.95&0.19&-0.01&-0.22&0.06&0.88&1.06&-0.02\\

1661: $M_{170}$&$m_{170}$&0.40&0.60&-0.20&-2.00&3.00  &1.08&1.06&0.96&0.18&0.01&-0.23&0.05&0.86&1.04&0.00\\

1662: $M_{171}$&$m_{171}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.07&0.97&0.16&0.03&-0.25&0.07&0.87&1.02&-0.01\\

1663: $M_{172}$&$m_{172}$&0.20&0.80&-0.60&-0.33&1.33  &1.06&1.07&0.97&0.16&0.03&-0.24&0.06&0.87&1.02&-0.01\\

1664: $M_{173}$&$m_{173}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.05&0.96&0.15&0.01&-0.22&0.05&0.88&1.03&0.00\\

1665: $M_{174}$&$m_{174}$&0.60&0.40&0.20&3.00&-2.00  &1.06&1.06&0.97&0.14&0.03&-0.20&0.04&0.86&1.05&-0.01\\

1666: $M_{175}$&$m_{175}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.08&0.98&0.12&0.05&-0.21&0.06&0.85&1.03&-0.02\\

1667: $M_{176}$&$m_{176}$&0.60&0.40&0.20&3.00&-2.00  &1.08&1.06&0.99&0.14&0.04&-0.19&0.05&0.83&1.04&-0.03\\

1668: $M_{177}$&$m_{177}$&0.20&0.80&-0.60&-0.33&1.33  &1.08&1.06&0.99&0.14&0.04&-0.19&0.04&0.82&1.04&-0.03\\

1669: $M_{178}$&$m_{178}$&0.40&0.60&-0.20&-2.00&3.00  &1.09&1.04&1.01&0.13&0.03&-0.20&0.06&0.84&1.02&-0.01\\

1670: $M_{179}$&$m_{179}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.03&1.02&0.14&0.02&-0.21&0.08&0.85&1.03&-0.03\\

1671: $M_{180}$&$m_{180}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.04&1.03&0.13&0.01&-0.19&0.09&0.83&1.05&-0.04\\

1672: $M_{181}$&$m_{181}$&0.60&0.40&0.20&3.00&-2.00  &1.04&1.05&1.01&0.12&0.03&-0.20&0.08&0.84&1.06&-0.02\\

1673: $M_{182}$&$m_{182}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.03&1.02&0.13&0.02&-0.21&0.10&0.86&1.07&-0.03\\

1674: $M_{183}$&$m_{183}$&0.20&0.80&-0.60&-0.33&1.33  &1.01&1.03&1.02&0.13&0.01&-0.21&0.10&0.86&1.07&-0.02\\

1675: $M_{184}$&$m_{184}$&0.20&0.80&-0.60&-0.33&1.33  &1.02&1.02&1.02&0.13&0.01&-0.22&0.09&0.86&1.07&-0.02\\

1676: $M_{185}$&$m_{185}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.04&1.01&0.12&0.00&-0.20&0.11&0.85&1.08&-0.03\\

1677: $M_{186}$&$m_{186}$&0.80&0.20&0.60&1.33&-0.33  &1.03&1.04&1.00&0.12&0.00&-0.19&0.11&0.85&1.08&-0.03\\

1678: $M_{187}$&$m_{187}$&0.80&0.20&0.60&1.33&-0.33  &1.03&1.04&0.99&0.11&0.00&-0.19&0.11&0.85&1.08&-0.03\\

1679: $M_{188}$&$m_{188}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.05&1.00&0.13&-0.01&-0.20&0.13&0.86&1.07&-0.04\\

1680: $M_{189}$&$m_{189}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&1.03&0.99&0.12&-0.02&-0.21&0.14&0.87&1.08&-0.02\\

1681: $M_{190}$&$m_{190}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.04&0.97&0.13&-0.01&-0.22&0.13&0.86&1.09&-0.03\\

1682: $M_{191}$&$m_{191}$&0.60&0.40&0.20&3.00&-2.00  &1.04&1.03&0.98&0.15&-0.02&-0.23&0.15&0.87&1.07&-0.04\\

1683: $M_{192}$&$m_{192}$&0.60&0.40&0.20&3.00&-2.00  &1.06&1.04&0.99&0.14&-0.03&-0.21&0.16&0.86&1.06&-0.06\\

1684: $M_{193}$&$m_{193}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.02&1.00&0.12&-0.01&-0.19&0.15&0.84&1.07&-0.07\\

1685: $M_{194}$&$m_{194}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.03&1.02&0.14&-0.02&-0.18&0.14&0.85&1.05&-0.08\\

1686: $M_{195}$&$m_{195}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.04&1.03&0.15&-0.03&-0.19&0.13&0.86&1.03&-0.06\\

1687: $M_{196}$&$m_{196}$&0.80&0.20&0.60&1.33&-0.33  &1.03&1.04&1.03&0.15&-0.04&-0.18&0.13&0.87&1.04&-0.06\\

1688: $M_{197}$&$m_{197}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.05&1.04&0.14&-0.02&-0.19&0.14&0.85&1.05&-0.07\\

1689: $M_{198}$&$m_{198}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.06&1.05&0.13&-0.03&-0.17&0.13&0.86&1.03&-0.06\\

1690: $M_{199}$&$m_{199}$&0.00&1.00&-1.00&0.00&1.00  &1.00&1.06&1.05&0.13&-0.03&-0.17&0.13&0.86&1.03&-0.05\\

1691: $M_{200}$&$m_{200}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.05&1.03&0.12&-0.01&-0.16&0.12&0.87&1.04&-0.06\\

1692: $M_{201}$&$m_{201}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&1.03&1.04&0.11&-0.02&-0.14&0.13&0.88&1.02&-0.07\\

1693: $M_{202}$&$m_{202}$&0.60&0.40&0.20&3.00&-2.00  &1.00&1.04&1.05&0.10&-0.03&-0.12&0.15&0.89&1.01&-0.08\\

1694: $M_{203}$&$m_{203}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.03&1.06&0.09&-0.02&-0.13&0.14&0.88&1.02&-0.07\\

1695: $M_{204}$&$m_{204}$&0.80&0.20&0.60&1.33&-0.33  &1.01&1.02&1.06&0.09&-0.02&-0.14&0.14&0.88&1.02&-0.07\\

1696: $M_{205}$&$m_{205}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&1.01&1.07&0.08&-0.01&-0.14&0.13&0.87&1.03&-0.05\\

1697: $M_{206}$&$m_{206}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.02&1.06&0.09&-0.02&-0.13&0.12&0.88&1.02&-0.06\\

1698: $M_{207}$&$m_{207}$&1.00&0.00&1.00&1.00&0.00  &1.03&1.02&1.06&0.09&-0.02&-0.13&0.12&0.88&1.02&-0.06\\

1699: $M_{208}$&$m_{208}$&0.80&0.20&0.60&1.33&-0.33  &1.03&1.01&1.06&0.09&-0.02&-0.12&0.12&0.88&1.02&-0.07\\

1700: $M_{209}$&$m_{209}$&0.60&0.40&0.20&3.00&-2.00  &1.02&1.02&1.07&0.10&0.00&-0.13&0.11&0.87&1.03&-0.07\\

1701: $M_{210}$&$m_{210}$&0.80&0.20&0.60&1.33&-0.33  &1.02&1.02&1.07&0.10&0.00&-0.13&0.11&0.87&1.02&-0.07\\

1702: $M_{211}$&$m_{211}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.01&1.08&0.09&0.01&-0.12&0.10&0.86&1.03&-0.08\\

1703: $M_{212}$&$m_{212}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&0.99&1.09&0.08&0.00&-0.13&0.11&0.84&1.04&-0.06\\

1704: $M_{213}$&$m_{213}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.00&1.07&0.09&-0.01&-0.11&0.10&0.85&1.02&-0.07\\

1705: $M_{214}$&$m_{214}$&0.60&0.40&0.20&3.00&-2.00  &1.06&0.99&1.06&0.11&0.01&-0.12&0.09&0.86&1.03&-0.08\\

1706: $M_{215}$&$m_{215}$&0.40&0.60&-0.20&-2.00&3.00  &1.07&1.00&1.04&0.10&0.02&-0.13&0.10&0.87&1.02&-0.09\\

1707: $M_{216}$&$m_{216}$&0.40&0.60&-0.20&-2.00&3.00  &1.08&1.01&1.03&0.11&0.01&-0.11&0.09&0.86&1.03&-0.10\\

1708: $M_{217}$&$m_{217}$&0.40&0.60&-0.20&-2.00&3.00  &1.06&1.01&1.01&0.12&0.00&-0.12&0.08&0.87&1.04&-0.08\\

1709: $M_{218}$&$m_{218}$&0.40&0.60&-0.20&-2.00&3.00  &1.05&1.02&1.02&0.14&0.01&-0.13&0.08&0.88&1.02&-0.09\\

1710: $M_{219}$&$m_{219}$&0.80&0.20&0.60&1.33&-0.33  &1.04&1.03&1.03&0.13&0.01&-0.13&0.07&0.88&1.03&-0.09\\

1711: $M_{220}$&$m_{220}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.03&1.01&0.12&0.00&-0.12&0.06&0.89&1.03&-0.07\\

1712: $M_{221}$&$m_{221}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.04&1.00&0.12&-0.01&-0.10&0.08&0.90&1.02&-0.08\\

1713: $M_{222}$&$m_{222}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.05&0.98&0.11&-0.01&-0.09&0.07&0.89&1.03&-0.07\\

1714: $M_{223}$&$m_{223}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.06&0.99&0.10&0.00&-0.10&0.06&0.87&1.04&-0.05\\

1715: $M_{224}$&$m_{224}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&1.05&1.00&0.11&-0.01&-0.11&0.07&0.88&1.02&-0.06\\

1716: $M_{225}$&$m_{225}$&0.60&0.40&0.20&3.00&-2.00  &1.03&1.03&1.01&0.10&-0.02&-0.09&0.08&0.89&1.03&-0.07\\

1717: $M_{226}$&$m_{226}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.04&1.02&0.09&-0.03&-0.08&0.08&0.90&1.02&-0.06\\

1718: $M_{227}$&$m_{227}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.05&1.03&0.08&-0.04&-0.09&0.09&0.91&1.01&-0.04\\

1719: $M_{228}$&$m_{228}$&0.40&0.60&-0.20&-2.00&3.00  &0.99&1.04&1.04&0.09&-0.02&-0.10&0.08&0.92&1.02&-0.05\\

1720: $M_{229}$&$m_{229}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.05&1.05&0.11&-0.03&-0.10&0.09&0.91&1.00&-0.06\\

1721: $M_{230}$&$m_{230}$&0.60&0.40&0.20&3.00&-2.00  &0.98&1.05&1.05&0.12&-0.04&-0.09&0.08&0.90&1.01&-0.07\\

1722: $M_{231}$&$m_{231}$&0.60&0.40&0.20&3.00&-2.00  &0.99&1.04&1.06&0.11&-0.05&-0.08&0.07&0.90&1.00&-0.06\\

1723: $M_{232}$&$m_{232}$&0.60&0.40&0.20&3.00&-2.00  &1.00&1.05&1.05&0.10&-0.04&-0.06&0.07&0.91&0.99&-0.06\\

1724: $M_{233}$&$m_{233}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.04&1.04&0.09&-0.04&-0.05&0.06&0.92&0.99&-0.05\\

1725: $M_{234}$&$m_{234}$&0.80&0.20&0.60&1.33&-0.33  &1.01&1.04&1.04&0.09&-0.04&-0.05&0.05&0.92&0.99&-0.05\\

1726: $M_{235}$&$m_{235}$&0.40&0.60&-0.20&-2.00&3.00  &1.00&1.05&1.05&0.08&-0.05&-0.04&0.05&0.91&1.00&-0.04\\

1727: $M_{236}$&$m_{236}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.03&1.03&0.07&-0.03&-0.05&0.06&0.92&1.01&-0.05\\

1728: $M_{237}$&$m_{237}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.02&1.04&0.06&-0.04&-0.03&0.07&0.93&0.99&-0.06\\

1729: $M_{238}$&$m_{238}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&1.01&1.03&0.06&-0.03&-0.04&0.06&0.94&1.00&-0.04\\

1730: $M_{239}$&$m_{239}$&0.40&0.60&-0.20&-2.00&3.00  &1.01&1.02&1.02&0.07&-0.04&-0.03&0.05&0.95&1.01&-0.05\\

1731: $M_{240}$&$m_{240}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&1.00&1.02&0.06&-0.05&-0.02&0.04&0.96&1.00&-0.04\\

1732: $M_{241}$&$m_{241}$&0.60&0.40&0.20&3.00&-2.00  &1.03&0.99&1.03&0.05&-0.03&-0.03&0.04&0.96&0.98&-0.03\\

1733: $M_{242}$&$m_{242}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&1.00&1.04&0.04&-0.02&-0.01&0.03&0.95&0.97&-0.03\\

1734: $M_{243}$&$m_{243}$&0.40&0.60&-0.20&-2.00&3.00  &1.04&0.99&1.05&0.03&-0.01&-0.02&0.02&0.96&0.96&-0.02\\

1735: $M_{244}$&$m_{244}$&0.60&0.40&0.20&3.00&-2.00  &1.05&0.98&1.06&0.03&0.00&-0.03&0.03&0.97&0.95&-0.03\\

1736: $M_{245}$&$m_{245}$&0.60&0.40&0.20&3.00&-2.00  &1.06&0.98&1.04&0.02&0.00&-0.02&0.02&0.96&0.96&-0.02\\

1737: $M_{246}$&$m_{246}$&0.80&0.20&0.60&1.33&-0.33  &1.06&0.98&1.04&0.02&0.00&-0.02&0.02&0.96&0.96&-0.02\\

1738: $M_{247}$&$m_{247}$&0.60&0.40&0.20&3.00&-2.00  &1.05&0.99&1.03&0.01&-0.01&-0.01&0.01&0.97&0.97&-0.01\\

1739: $M_{248}$&$m_{248}$&0.40&0.60&-0.20&-2.00&3.00  &1.03&0.98&1.03&0.00&-0.01&0.01&0.03&0.97&0.97&-0.01\\

1740: $M_{249}$&$m_{249}$&0.40&0.60&-0.20&-2.00&3.00  &1.02&0.99&1.02&-0.01&0.00&0.00&0.02&0.98&0.98&0.00\\

1741: $M_{250}$&$m_{250}$&0.60&0.40&0.20&3.00&-2.00  &1.01&1.00&1.01&-0.02&0.01&0.01&0.01&0.99&0.99&-0.01\\

1742: $M_{251}$&$m_{251}$&0.60&0.40&0.20&3.00&-2.00  &1.00&1.00&1.00&0.00&0.00&0.00&0.00&1.00&1.00&0.00\\

1743: $M_{252}$&$m_{252}$&0.80&0.20&0.60&1.33&-0.33  &1.00&1.00&1.00&0.00&0.00&0.00&0.00&1.00&1.00&0.00\\

1744: \hline

1745: \end{tabular}

1746: }

1747: \end{table}

1748: 

1749: \end{document}

1750: