1: \documentclass[twocolumn,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
2:
3: \usepackage{graphicx}% Include figure files
4: \usepackage{dcolumn}% Align table columns on decimal point
5: \usepackage{bm}% bold math
6:
7: %\nofiles
8:
9: \begin{document}
10:
11: \title{On Optimality Condition of Complex
12: Systems: Computational Evidence\\ }% Force line breaks with \\
13:
14: \author{Victor Korotkikh}
15: \email{v.korotkikh@cqu.edu.au}
16: \author{Galina Korotkikh}
17: \email{g.korotkikh@cqu.edu.au} \affiliation{Central Queensland
18: University, Mackay, Queensland 4740, Australia}
19:
20: \author{Darryl Bond}
21: \email{dbond@nrggos.com.au} \affiliation{ Central Queensland
22: University, Gladstone, Queensland 4680, Australia}
23:
24: \begin{abstract}
25:
26: A general condition determining the optimal performance of a
27: complex system has not yet been found and the possibility of its
28: existence is unknown. To contribute in this direction, an
29: optimization algorithm as a complex system is presented. The
30: performance of the algorithm for any problem is controlled as a
31: convex function with a single optimum. To characterize the
32: performance optimums, certain quantities of the algorithm and the
33: problem are suggested and interpreted as their complexities. An
34: optimality condition of the algorithm is computationally found: if
35: the algorithm shows its best performance for a problem, then the
36: complexity of the algorithm is in a linear relationship with the
37: complexity of the problem. The optimality condition provides a new
38: perspective to the subject by recognizing that the relationship
39: between certain quantities of the complex system and the problem
40: may determine the optimal performance.
41:
42: \end{abstract}
43:
44: \pacs{89.75.-k, 89.75.Fb}
45:
46: \maketitle
47:
48: The efficient management of complex systems is increasingly
49: important. However, despite significant progress and interest in
50: complex systems \cite{Holland_1}, there is a limited understanding
51: of the problem. In particular, because the existence of principles
52: governing the non-equilibrium situation has not yet been
53: established \cite{Ball_1}, the possibility of a general condition
54: determining the optimal performance of a complex system is still
55: unknown. We aim to contribute in this direction and present
56: results of computational experiments in order to discuss the
57: possibility of an optimality condition of complex systems. The
58: results provide a new perspective to the subject by recognizing
59: that the relationship between certain quantities of the complex
60: system and the problem may determine the optimal performance.
61:
62: In this Letter, we consider an optimization algorithm as a complex
63: system by using the benchmark traveling salesman problems
64: \cite{Reinelt_1}. For any problem tested, we can control the
65: performance of the algorithm as a convex function with a single
66: optimum. Consequently, we take this opportunity to investigate
67: whether the performance optimums may be characterized in terms of
68: an optimality condition of the algorithm. Namely, we describe the
69: algorithm by a trace of the variance-covariance matrix derived
70: from the dynamics of the algorithm as a complex system. We also
71: specify a problem by a trace of the distance matrix. Remarkably,
72: the computational analysis of the performance optimums reveals a
73: relationship between the quantities of the algorithm and the
74: problem by approximating it well enough with a linear function.
75:
76: To recognize the potential importance of the result, we interpret
77: the quantities as the complexities of the algorithm and the
78: problem, and formulate the following optimality condition: {\it if
79: the algorithm shows its best performance for a problem, then the
80: complexity of the algorithm is in the linear relationship with the
81: complexity of the problem}. Further experimental facts suggest how
82: the optimality condition may be extended into a wider context.
83:
84: Let us consider an optimization algorithm ${\cal A}$ as a complex
85: system of $N$ computational agents that minimize the average
86: distance in the traveling salesman problem. All agents start in
87: the same city and at each step an agent visits the next city by
88: using one of two strategies: a random strategy, i.e., visit the
89: next city at random, or the greedy strategy, i.e, visit the next
90: closest city. We define that all agents start with the random
91: strategy. The state of the agents visiting $n$ cities of the
92: problem can be described at step $j = 1,...,n-1$ by a binary
93: sequence $s_{j} = s_{1j}...s_{Nj}$, where $s_{ij} = +1$, if agent
94: $i = 1,...,N$ uses the random strategy and $s_{ij} = -1$, if the
95: agent uses the greedy strategy to visit the next city. The
96: dynamics of the agents is realized through their choice of
97: strategies and can be encoded by an $N \times (n-1)$ binary
98: strategy matrix $S = \{ s_{ij}, \ i = 1,...,N, \ j = 1,...,n-1 \}
99: $.
100:
101: We introduce a variable parameter $v$ that controls the dynamics
102: of the agents in a specific manner. Let $D_{ij}$ be the distance
103: travelled by agent $i=1,...,N$ after $j=1,...,n-1$ steps and
104: $$
105: D_{j}^{-}=\min_{i=1,...,N}D_{ij}, \
106: D_{j}^{+}=\max_{i=1,...,N}D_{ij}.
107: $$
108: All distances travelled by the agents after $j = 1,...,n-1$ steps
109: belong to the interval $[D_{j}^{-},D_{j}^{+}]$. The parameter $v$
110: specifies a threshold point
111: $$
112: D_{j}(v)=D_{j}^{+}-v(D_{j}^{+}-D_{j}^{-}), \ \ \ 0 \leq v \leq
113: 1+\gamma, \ \ \ \gamma = 0.1
114: $$
115: dividing the interval into two parts, i.e., successful
116: $[D_{j}^{-}, D_{j}(v)]$ and unsuccessful $(D_{j}(v),D_{j}^{+}]$.
117: If the distance $D_{ij}$ travelled by agent $i = 1,...,N$ after
118: $j=1,...,n-1$ steps belongs to the successful interval, then we
119: regard the agent's last strategy as successful. If the distance
120: $D_{ij}$ belongs to the unsuccessful interval, then the agent's
121: last strategy is unsuccessful.
122:
123: In order to choose the next strategy, each agent uses an optimal
124: rule \cite{Korotkikh_1} that relies on the Prouhet-Thue-Morse
125: (PTM) sequence and has the following description:
126:
127: 1. {\it If your last strategy is successful, continue with the
128: same strategy.
129:
130: 2. If your last strategy is unsuccessful, consult PTM generator
131: which strategy to use next}.
132:
133: The PTM sequence $+1-1-1+1-1+1+1-1 \ ... \ $ gives a symbolic
134: description of chaos resulting from period-doubling
135: \cite{Feigenbaum_1} in complex systems \cite{Allouche_1}. It can
136: be also associated with formation processes of integer relations
137: progressing well through the levels of a hierarchical structure
138: \cite{Korotkikh_1}. The formation processes produce a measure of
139: complexity \cite{Korotkikh_1} that we use as a guide in defining
140: the complexities of the algorithm and the problem.
141:
142: Each agent has its own PTM generator and a pointer attached to it.
143: The pointer starts with the first bit of the PTM sequence and
144: after each consultation moves one step further, so that the next
145: bit of the PTM sequence can be used, if the strategy is
146: unsuccessful. The control is realized by changing the parameter
147: $v$ from $0$ to $1 + \gamma$. For the limiting values of the
148: parameter $v$, the algorithm ${\cal A}$ produces the following
149: dynamics of the agents.
150:
151: When $v = 0$, then $D_{j}(v) = D_{j}^{+}$. Therefore, the
152: successful interval $[D_{j}^{-},D_{j}(v)]$ coincides with the
153: whole interval $[D_{j}^{-},D_{j}^{+}]$ and the last strategy of
154: each agent is successful at each step $j = 1,...,n-1$. This means
155: that each agent always uses the random strategy and the strategy
156: matrix becomes
157: $$
158: S = \left(\begin{array}{ccccc}
159: +1 & +1 & ... & +1 & +1\\
160: +1 & +1 & ... & +1 & +1\\
161: . & . & ... & . & .\\
162: +1 & +1 & ... & +1 & +1\\
163: +1 & +1 & ... & +1 & +1\\
164: \end{array} \right)
165: $$
166:
167: In the opposite limit, when $v = 1 +\gamma$, then
168: $D_{j}(v)<D_{j}^{-}$. Thus, the unsuccessful interval
169: $(D_{j}(v),D_{j}^{+}]$ covers the whole interval
170: $[D_{j}^{-},D_{j}^{+}]$. This means that the last strategy of each
171: agent is always unsuccessful and, according to the rule, at each
172: step $j = 1,..., n-1$ an agent asks the PTM generator which
173: strategy to use next. As a result, the binary sequence of each
174: agent becomes the initial segment of length $(n-1)$ of the PTM
175: sequence and the strategy matrix is turned to
176: $$
177: S = \left(\begin{array}{ccccc}
178: +1 & -1 & ... & - 1 & + 1\\
179: +1 & -1 & ... & - 1 & + 1\\
180: . & . & ... & . & .\\
181: +1 & -1 & ... & - 1 & + 1\\
182: +1 & -1 & ... & - 1 & + 1\\
183: \end{array} \right)
184: $$
185:
186: The role of the parameter $v$ may be seen in terms of
187: correlations. When $v = 0$, the agents are independent, because
188: the behavior of an agent does not depend on the others. However,
189: as the parameter $v$ gets larger, the successful interval gets
190: smaller, and the behaviors of the agents become more correlated
191: through the PTM sequence. To stay independent, an agent has to
192: show a result approaching the current minimum. Consequently, the
193: rule urges an agent to follow the PTM sequence more strongly and
194: as a result, the presence of the PTM sequence in the strategy
195: matrix becomes more evident. When $v = 1 + \gamma$, the whole
196: strategy matrix consists of the PTM sequences.
197:
198: By extensive computational experiments, we investigate how the
199: performance of the algorithm ${\cal A}$ may depend on the
200: parameter $v$. We tested the problems $P$ = $\{$eil76, eil101,
201: st70, rat195, lin105, kroC100, kroB100, kroA100, kroD100, d198,
202: kroA150, pr107, u159, pr144, pr144, pr152, pr226, pr136, pr76,
203: ts225$\}$ belonging to the benchmark traveling salesman problems
204: \cite{Reinelt_1}. Let $D_{i}(p,v)$ be the distance travelled by
205: agent $i = 1,...,N$ for a problem $p \in P$ and a value $v$ of the
206: parameter. The performance of the algorithm ${\cal A}$ is
207: characterized by the average distance travelled by $N$ agents
208: $D(p,v) = \sum_{i = 1}^{N} D_{i}(p,v)/N$, which is sought to be
209: minimized.
210:
211: In the experiments, the algorithm ${\cal A}$ is applied to each
212: problem $p \in P$ with the number of agents $N = 2000$ and values
213: $v_{i} = i\triangle v, \ i = 0, 1,...,20, \ v = 1.1$ of the
214: parameter, where $\triangle v = 0.05$. To eliminate randomness,
215: the computations are repeated in a series of $10$ tests and the
216: performance functions $\{D_{k}(p,v), \ k = 1,...,10 \}$ are
217: averaged into $\bar D(p,v)$. Remarkably, for each problem $p \in
218: P$, it is found that the performance $\bar D(p,v)$ of the
219: optimization algorithm ${\cal A}$ behaves as a {\it convex
220: function} of the parameter $v$ with the only optimum at $v^{*}(p)$
221: (Fig. 1).
222:
223: \begin{figure}
224: \includegraphics[width=.46\textwidth]{eil101.eps}
225: \caption{\label{fig:zero} The performance of the algorithm ${\cal
226: A}$ as a convex function of the parameter $v$ for problem ei101.}
227: \end{figure}
228:
229: We characterize the performance optimums $\{v^{*}(p), \ p \in P
230: \}$ by certain quantities of the algorithm and the problem and
231: interpret the quantities as their complexities. The algorithm
232: ${\cal A}$ is described by the quadratic trace of the
233: variance-covariance matrix derived from the strategy matrix
234: \cite{Korotkikh_2}. Namely, for a problem $p \in P$, a set of
235: strategy matrices $ S_{k}(v^{*}(p)) = \{ s_{ij}(k), i = 1,...,N; \
236: j = 1,...,n-1 \}, k = 1,...,10 $ is obtained as a result of ten
237: runs for the value $v^{*}(p)$ of the parameter. For each strategy
238: matrix $S_{k}(v^{*}(p)), k = 1,...,10$ the variance-covariance
239: matrix $ V(S_{k}(v^{*}(p))) = \{ V_{ij}(k), \ i, j = 1,...,N \}, i
240: = 1,...,10, $ and its quadratic trace
241: $$
242: tr(V^{2}(S_{k}(v^{*}(p)))) = \sum_{i=1}^{N} \lambda_{ik}^{2}
243: $$
244: are computed, where $V_{ij}(k)$ is the linear correlation
245: coefficient between agents $i = 1,...,N$ and $j = 1,...,N$ and
246: $\lambda_{ik}, i = 1,...,N, \ k = 1,...,10$ are the eigenvalues of
247: the variance-covariance matrix $V(S_{k}(v^{*}(p)))$.
248:
249: The average $tr(V^{2}(S(v^{*}(p))))$ of the traces is used to
250: describe the complexity
251: $$
252: C({\cal A}(p)) = \frac{1}{N^{2}}tr(V^{2}(S(v^{*}(p))))
253: $$
254: of the algorithm ${\cal A}$. There is a connection between
255: $N^{2}/tr(V^{2}(S))$ and the number of KLD (Karhunen-Loeve
256: decomposition) modes $D_{KLD}$ \cite{Korotkikh_3}. The quantity
257: $D_{KLD}$ measures the complexity of spatiotemporal data
258: \cite{Sirovich_1} and a correlation length $\xi_{KLD}$, based on
259: $D_{KLD}$, can characterize high-dimensional inhomogeneous
260: spatiotemporal chaos \cite{Zoldi_1}. In our case, it turns more
261: appropriate to consider the complexity of the algorithm ${\cal A}$
262: in terms of $tr(V^{2}(S))/N^{2}$, although the complexity is
263: greater, the smaller $tr(V^{2}(S))/N^{2}$.
264:
265: We describe the complexity $C(p)$ of the problem $p \in P$ by the
266: quadratic trace
267: $$
268: C(p) = \frac{1}{n^{2}}tr(M^{2}(p)) = \frac{1}{n^{2}}\sum_{i=1}^{n}
269: \lambda_{i}^{2}
270: $$
271: of the distance matrix $ M(p) = \{ d_{ij}/d_{max}, i,j = 1,...,n
272: \}, $ where $\lambda_{i}, i = 1,...,n$ are the eigenvalues of the
273: distance matrix, $d_{ij}$ is the distance between cities $i =
274: 1,...,n$ and $j = 1,...,n$ and $d_{max}$ is the maximum of the
275: distances.
276:
277: An optimality condition of the algorithm ${\cal A}$ is sought
278: through a possible relationship between the complexity $C({\cal
279: A}(p))$ of the algorithm ${\cal A}$ and the complexity $C(p)$ of
280: the problem $p$. For this purpose, we consider the points with
281: coordinates $\{x = C(p), y = C({\cal A}(p)), \ p \in P \}$. The
282: result of the analysis shown in Fig. 2 suggests a possible linear
283: relationship between the complexities. The regression line is
284: calculated
285: $$
286: y = \alpha x + \beta = 0.67x + 0.33,
287: $$
288: \begin{equation}
289: \label{OC1} \alpha= 0.67 \pm 0.01, \ \ \ \beta = 0.33 \pm 0.01,
290: \end{equation}
291: where the standard error of estimate is $0.09$ and the absolute
292: value of the maximal individual error is $0.05$. The coefficient
293: of determination of $0.71$ tells that $71$ percent of the
294: variation in the complexity of the algorithm ${\cal A}$ is
295: explained by the regression line.
296:
297: \begin{figure}
298: \includegraphics[width=.44\textwidth]{ptm.eps}
299: \caption{\label{fig:zero} The regression line of the relationship
300: between the complexity of the algorithm ${\cal A}$ and the
301: complexity of the problem with the bounds including all points.}
302: \end{figure}
303:
304: The experiment is repeated a number of times and each time the
305: results confirm the consistency of the regression line
306: (\ref{OC1}). Therefore, within the accuracy of the linear
307: regression, we are able to formulate an optimality condition: {\it
308: if the algorithm ${\cal A}$ shows its best performance for a
309: problem $p$, then the complexity $C({\cal A}(p))$ of the algorithm
310: ${\cal A}$ is in the linear relationship with the complexity
311: $C(p)$ of the problem $p$}
312: \begin{equation}
313: \label{OC2} C({\cal A}(p)) = 0.67 \times C(p) + 0.33.
314: \end{equation}
315:
316: Computational investigations on the role of the PTM generator for
317: the algorithm ${\cal A}$ suggest how the optimality condition
318: (\ref{OC2}) may be extended. For this reason we use a different
319: algorithm ${\cal B}$, which works exactly in the same manner as
320: the algorithm ${\cal A}$, except it consults a random generator
321: instead of the PTM generator. To find a possible connection
322: between the best performances and the relationship between the
323: complexities, we compare the algorithms ${\cal A}$ and ${\cal B}$.
324:
325: First, for each problem $p \in {\cal P}$ the best performance of
326: the algorithm ${\cal A}$ is compared with the best performance of
327: the algorithm ${\cal B}$. Fig. 3 shows that the algorithm ${\cal
328: A}$ demonstrates significantly better results than the algorithm
329: ${\cal B}$ for seventeen problems and results for the other three
330: are close.
331:
332: \begin{figure}
333: \includegraphics[width=.46\textwidth]{ptm-rnd.eps}
334: \caption{\label{fig:zero} The best performances of the algorithm
335: ${\cal A}$ using PTM generator are compared with the best
336: performances of the algorithm ${\cal B}$ using random generator.
337: The problems are ordered as in the description of the class $P$.}
338: \end{figure}
339:
340: Second, we examine the relationship between the complexities of
341: the algorithm ${\cal B}$ and the problem (Fig. 4) in the same
342: manner as we have done for the algorithm ${\cal A}$ (Fig. 2). The
343: situation deteriorates and the relationship is less consistent for
344: the algorithm ${\cal B}$ in comparison with the algorithm ${\cal
345: A}$. Formally, the calculated regression line
346: $$
347: y = \alpha'x + \beta' = 0.48x + 0.23,
348: $$
349: \begin{equation}
350: \label{OC3} \alpha' = 0.48 \pm 0.02, \ \ \ \beta' = 0.23 \pm 0.01
351: \end{equation}
352: is a less accurate estimator of the relationship between the
353: complexities of the algorithm ${\cal B}$ and the problem. The
354: standard error of estimate of $0.18$ and the absolute value of the
355: maximal individual error of $0.1$ of the regression line
356: (\ref{OC3}) are doubled in comparison with the regression line
357: (\ref{OC1}). The coefficient of determination is only $0.26$ in
358: comparison with $0.71$ for the algorithm ${\cal A}$.
359:
360: Therefore, computationally, we reveal a connection between the
361: best performances and the relationship between the complexities.
362: In particular, the better result of the algorithm ${\cal A}$ (Fig.
363: 3) corresponds to the fact that the relationship between the
364: complexities of the algorithm ${\cal A}$ and the problem appears
365: more consistent than it is in the case of the algorithm ${\cal
366: B}$. In order to suggest potential implications of the optimality
367: condition (\ref{OC2}) to a wider context, we would like to use
368: this proposition.
369:
370: The observed connection allows us to assume a possible sequence of
371: algorithms whose best performances converge to the optimal
372: solutions of the problems as the relationships between the
373: complexities converge to a certain function $g$. Notice, that the
374: algorithms ${\cal A}$ and ${\cal B}$ may be well among the
375: elements of the sequence with the algorithm ${\cal B}$ followed by
376: the algorithm ${\cal A}$. According to the experiments, it seems
377: likely that the function $g$ may be a linear function.
378:
379: \begin{figure}
380: \includegraphics[width=.44\textwidth]{rnd.eps}
381: \caption{\label{fig:zero} The regression line of the relationship
382: between the complexity of the algorithm ${\cal B}$ and the
383: complexity of the problem.}
384: \end{figure}
385:
386: The possible convergence of the sequence would promise an
387: algorithm ${\cal G}$ to link the optimal performance and the
388: relationship between the complexities. As a result, the best
389: performance of the algorithm ${\cal G}$ for any problem could
390: actually provide the optimal solution. An optimality condition of
391: the algorithm ${\cal G}$ may extend the optimality condition of
392: the algorithm ${\cal A}$ and explain the optimal performance in
393: terms of the relationship between the complexities: if the
394: algorithm ${\cal G}$ shows its best performance for a problem $p$,
395: and thus finds the optimum solution, then the complexity $C({\cal
396: G}(p))$ of the algorithm is in the relationship $C({\cal G}(p))=
397: g(C(p))$ with the complexity $C(p)$ of the problem.
398:
399: The optimality condition of the algorithm ${\cal G}$ may suggest a
400: new approach to optimization that would be primarily concerned
401: with the efficient control of the algorithm's complexity in
402: matching the complexity of the problem. For example, if the
403: algorithm ${\cal G}$ could potentially find the optimal solution
404: to a problem $p$ of complexity $C(p)$ under the condition $C({\cal
405: G}(p)) = g(C(p))$, would it then be possible to obtain the optimal
406: solution by a different algorithm working with the same complexity
407: for the problem?
408:
409: The ability to evaluate the complexity of the problem before the
410: actual computation would be beneficial. In this case, the finding
411: of the optimal solution could be connected with the tuning of the
412: algorithm's complexity in order to match the already known
413: complexity of the problem. Experimentally, it is observed that for
414: any problem the performance of the algorithm ${\cal A}$ is a
415: convex function of the parameter. As a result, the best
416: performance of the algorithm ${\cal A}$ for the problem can be
417: efficiently obtained by the minimization of this convex function.
418: Therefore, it would be important to understand whether the
419: performance of the algorithm ${\cal G}$, as the function of its
420: complexity, may behave in a similar way to provide us with
421: efficient means for finding optimal solutions.
422:
423: In conclusion, we have presented an optimization algorithm as a
424: complex system. For any problem tested, the performance of the
425: algorithm has been controlled as a convex function with a single
426: optimum. By the characterization of the performance optimums, an
427: optimality condition of the algorithm has been proposed. The
428: optimality condition has revealed that the relationship between
429: certain quantities of the complex system and the problem may
430: determine the optimal performance. The result provides a
431: computational evidence to the possibility of an optimality
432: condition of complex systems.
433:
434: This work was supported by CQU Research Advancement Awards Scheme
435: grant no. IN9022 and Faculty of Informatics and Communication
436: Research Grant Scheme no. FRG00406.
437:
438: \begin{thebibliography}{00}
439:
440: \bibitem{Holland_1}
441: J. H. Holland {\it Emergence: From Chaos to Order}
442: (Addison-Wesley, Reading, Massachusetts, 1998); Y. Bar-Yam {\it
443: Dynamics of Complex Systems} (Westview Press, 1997); S. Kauffman
444: {\it At Home in the Universe} (Oxford University Press, New York,
445: 1995).
446: \bibitem{Ball_1} P. Ball, Nature {\bf 402}, c73 (1999),
447: and references therein.
448: \bibitem{Reinelt_1} G. Reinelt {\it TSPLIB Version 1.2
449: $\lbrack$ online $\rbrack$}
450: (ftp://ftp.wiwi.unifrankfurt.de/pub/TSPLIB 1.2) $\lbrack$ Accessed
451: 28/11/2000$\rbrack$.
452: \bibitem{Korotkikh_1} V. Korotkikh {\it A Mathematical Structure
453: for Emergent Computation} (Kluwer, Dordrecht, 1999).
454: \bibitem{Feigenbaum_1} M. Feigenbaum, Los Alamos Sci. {\bf 1},
455: 4 (1980).
456: \bibitem{Allouche_1} J. Allouche and M. Cosnard, in {\it Dynamical Systems
457: and Cellular Automata} (Academic Press, 1985).
458: \bibitem{Korotkikh_2} G. Korotkikh and V. Korotkikh, in
459: {\it Optimization and Industry: New Frontiers}, edited by P.
460: Pardalos and V. Korotkikh (Kluwer, Dordrecht, 2003).
461: \bibitem{Korotkikh_3} V. Korotkikh and G. Korotkikh, in
462: {\it Quantitative Neuroscience}, edited by P. Pardalos, C.
463: Sackellares, P. Carney and L. Iasemidis (Kluwer, Dordrecht, 2004).
464: \bibitem{Sirovich_1} L. Sirovich and A.E. Deane, J. Fluid Mech.
465: {\bf 222}, 251 (1991); S. Ciliberto and B. Nikolaenko, Europhys.
466: Lett. {\bf 14}, 303 (1991); R. Vautard and M. Ghil, Physica
467: (Amsterdam) {\bf 35D}, 395 (1989).
468: \bibitem{Zoldi_1} S.M. Zoldi and H.S. Greenside, Phys. Rev. Lett.
469: {\bf 78}, 1687 (1997).
470:
471: \end{thebibliography}
472:
473: \end{document}
474: