1: \section{Case Study: The Full COPS}
2:
3: \label{case2}
4: \begin{figure*}
5: \centerline {\includegraphics[height=9.3cm]{copsalllog.eps}}
6: \caption {Performance profile for full COPS set}
7: \label {figure:copsprof}
8: \end{figure*}
9: We now expand our analysis of the data to include all
10: the problems in version 2.0 of the \cops\ \cite{cops-home}
11: test set. We present in
12: Figure \ref{figure:copsprof}
13: a log$_2$ scaled view of the performance profiles for the
14: solvers on that test set.
15:
16: Figure \ref{figure:copsprof} gives a clear indication of the
17: relative performance of each solver.
18: As in the performance profiles
19: in Section \ref{case1}, this figure shows that performance profiles
20: eliminate the undue influence of a small number of problems
21: on the benchmarking process and
22: the sensitivity of the results associated
23: with the ranking of solvers. In addition,
24: performance profiles provide an estimate of the
25: expected performance difference between solvers.
26:
27: The most significant aspect of Figure \ref{figure:copsprof},
28: as compared with Figure \ref{figure:log2},
29: is that on this test set LOQO dominates all other solvers:
30: the performance profile for LOQO lies above
31: all others for all performance ratios.
32: The interpretation of the results in Figure~\ref{figure:copsprof}
33: is important. In particular, these results
34: do not imply that LOQO is faster on every problem.
35: They indicate only that, for any $ \tau \ge 1 $,
36: LOQO solves more problems within a factor of $ \tau $ of any other
37: solver time. Moreover, by examining $\rho_s (1)$ and $\rho_s(r_M)$,
38: we can also say that LOQO is the fastest solver on approximately 58\% of
39: the problems, and that LOQO solves the most problems (about 87\%)
40: to optimality.
41:
42: The difference between the results in Section \ref{case1}
43: and these results is due to a number of factors.
44: First of all, as can be seen in Table \ref{table:cops-stats},
45: the degrees of freedom for
46: the full \cops\ test set is much larger than for the
47: restricted subset of optimal control and parameter estimation problems.
48: Since, as noted in Section \ref{data},
49: MINOS and SNOPT are designed for problems with a modest
50: number of degrees of freedom, we should expect the
51: performance of MINOS and SNOPT to deteriorate on the
52: full \cops\ set.
53: This deterioration can be seen by comparing
54: Figure \ref{figure:copsprof} with Figure \ref{figure:log2}
55: and noting that the performance profiles of MINOS and SNOPT
56: are similar but generally lower in Figure~\ref{figure:copsprof}.
57:
58: Another reason for the
59: difference between the results in Section \ref{case1}
60: and these results is that
61: MINOS and SNOPT use only first-order information,
62: while LOQO uses second-order information.
63: The benefit of using second-order information usually
64: increases as the number of variables increases,
65: so this is another factor that benefits LOQO.
66:
67: The results in this section underscore our observation
68: that performance profiles provide a convenient tool
69: for comparing and evaluating the performance of optimization solvers,
70: but, like all tools, performance profiles must be used with care.
71: A performance profile
72: reflects the performance only on the data being used, and thus it
73: is important to understand the test set and the solvers
74: used in the benchmark.
75: