1: \section{Benchmarking Data}
2:
3: \label{data}
4:
5: The timing data used to compute the
6: performance profiles in Sections \ref{case1} and \ref{case2}
7: are generated with the \cops\ test set,
8: which currently consists of seventeen different applications,
9: all models in the AMPL \cite{AMPL} modeling
10: language. The choice of the test problem set $ \cP $ is always a source
11: of disagreement because there is no consensus on how
12: to choose problems.
13: The \cops\ problems are selected to be interesting and difficult,
14: but these criteria are subjective.
15: % We include problems with multiple minima, for which we treat
16: % convergence to any local minimum equally, and problems with a large
17: % number of degrees of freedom.
18: For each of the applications in the \cops\ set we use
19: four instances of the application obtained by varying
20: a parameter in the application, for example, the number
21: of grid points in a discretization. Application descriptions
22: and complete absolute timing results for the full test set
23: are given in \cite{EDD00}.
24:
25: Section \ref{case1} focuses on only the subset of the eleven optimal control and
26: parameter estimation applications
27: in the \cops\ set, while the discussion in Section \ref{case2} covers the
28: complete performance results. Accordingly, we provide here information
29: specific to this subset of the \cops\ problems as
30: well as an analysis of the test set as a whole.
31: Table~\ref{table:cops-stats} gives the quartiles
32: for three problem parameters: the number of
33: variables $n$, the number of constraints, and the
34: ratio $ (n -n_e)/n $, where $ n_e $ is the number
35: of equality constraints. In the optimization literature,
36: $ n - n_e $ is often called the degrees of freedom of
37: the problem, since it is an upper bound on the number
38: of variables that are free at the solution.
39:
40: The data in Table
41: \ref{table:cops-stats}
42: is fairly representative of the distribution of these
43: parameters throughout the test set and shows
44: that at least three-fourths of the problems have
45: the number of variables $n$ in the interval $ [400, 5000 ] $.
46: Our aim was to avoid problems where $n$ was
47: in the range $ [ 1 , 50 ] $ because other benchmarking
48: problem sets tend to
49: have a preponderance of problems with $ n $ in this range.
50: The main difference between the full \cops\ set and the
51: \cops\ subset is that the \cops\ subset is more
52: constrained with $ n_e \ge n/2 $ for all the problems.
53: Another feature of the \cops\ subset is that the equality
54: constraints are the result of either difference or collocation
55: approximations to differential equations.
56:
57:
58: We ran our final complete runs with the
59: same options for all models. The options involve setting the
60: output level for each solver so that we can gather the data
61: we need, increasing the iteration limits as much as allowed, and
62: increasing the super-basics limits for MINOS and SNOPT to 5000.
63: None of the failures we record in the final trials include any
64: solver error messages about having violated these limits.
65: While we relieved restrictions unnecessary for our testing, all
66: other parameters were left at the solvers' default settings.
67:
68: \begin{table}
69: \small
70: \caption{Problem data for \cops\ test set}
71: \begin{tabular}{@{}l c c c c c|c c c c c@{}}
72: \vspace{-7pt}
73: & \multicolumn{5}{c|}{Full \cops} & \multicolumn{5}{c@{}}{\cops\ subset} \\
74: \vspace{-3pt}
75: & \multicolumn{5}{r@{}|}{\hrulefill} & \multicolumn{5}{@{}l@{}}{\hrulefill} \\
76: & $\min$ & $q_1$ & $q_2$ & $q_3$ & $\max$
77: & $\min$ & $q_1$ & $q_2$ & $q_3$ & $\max$ \\ \hline
78: \vspace{1pt}
79: Num. variables & 48 & 400 & 1000 & 2402 & 5000 & 100 & 449 & 899 & 2000 & 4815 \\
80: Num. constraints & 0 & 150 & 498 & 1598 & 5048 & 51 & 400 & 800 & 1601 & 4797 \\
81: Degrees freedom & 0 & 23 & 148 & 401 & 5000 & 0 & 5 & 99 & 201 & 1198 \\
82: Deg. freedom (\%) & 0.0 & 1.0 & 33.2 & 100.0 & 100.0 &
83: 0.0 & 0.4 & 19.8 & 33.1 & 49.9\\
84: \hline
85: \end{tabular}
86: \label{table:cops-stats}
87: \end{table}
88:
89: The script for generating the timing data
90: sends a problem to each solver successively,
91: so as to minimize the effect of
92: fluctuation in the machine load. The script tracks the wall-clock
93: time from the start of the solve, killing any process that runs 3,600
94: seconds, which we declare unsuccessful, and beginning the next solve.
95: We cycle through all the problems, recording the
96: wall-clock time as well as the combination of AMPL system time (to
97: interpret the model and compute varying amounts of derivative
98: information required by each solver) and AMPL solver time for each
99: model variation.
100: We repeat the cycle for any model for
101: which one of the solvers' AMPL time and the wall-clock time differ
102: by more than 10 percent. To further ensure consistency, we have
103: verified that the
104: AMPL time results we present could be reproduced to within 10 percent
105: accuracy. All computations were done on a SparcULTRA2
106: running Solaris~7.
107:
108: We have ignored the effects of the stopping criteria and the memory
109: requirements of the solvers.
110: Ideally we would have used the same stopping criteria, but this
111: is not possible in the AMPL environment. In any case, differences in
112: computing time due to the stopping criteria are not likely to change
113: computing times by more than a factor of two.
114: Memory requirements can also play an important role.
115: In particular, solvers that use direct linear equation solvers are often more
116: efficient in terms of computing time provided there is enough memory.
117:
118: The solvers that we benchmark have
119: different requirements. MINOS and SNOPT use only first-order information,
120: while LANCELOT and LOQO need second-order information.
121: The use of second-order information can reduce the number of
122: iterations, but the cost per iteration usually increases. In
123: addition, obtaining second-order information is more costly and
124: may not even be possible.
125: MINOS and SNOPT are specifically designed for problems with a modest
126: number of degrees of freedom, while this is not the case for LANCELOT
127: and LOQO.
128: As an example of comparing solvers with similar requirements,
129: Section \ref{lp} shows the performance of linear programming solvers.
130:
131:
132:
133: