1: \documentclass[letterpaper, twocolumn, superscriptaddress, floatfix]{revtex4}
2: \pdfoutput=1
3: %\documentclass[letterpaper, endfloats*, preprint, superscriptaddress]{revtex4}
4: \bibliographystyle{plos}
5:
6: % To merge into nice file with figure at end.
7: %
8: % 1) Use the preprint options above
9: % 2) Use the makeatletter command below
10: % 3) Use the \clearpage \listoffigures commands at the end of the document
11: % 4) dvips with -pp option to only include necessary pages
12: % 5) Put figures in other file (figure_file.tex). Use \setcounter{page} to
13: % get proper numbering
14: % 6) Edit PLoS.lof to give figures proper page numbers in main text and latex
15: % again
16: % 7) Merge them with <gs -q -dBATCH -dNOPAUSE -dSAFER -sOutputFile=Merged.pdf -sDEVICE=pdfwrite PLoS.ps figure_file.ps>
17:
18: %\makeatletter
19: %\def\@dotsep{4.5}
20: %\makeatother
21:
22: \usepackage{xspace}
23: \usepackage{amsmath}
24: \usepackage{graphicx}
25:
26: \usepackage[usenames]{color}
27: \newcommand{\comment}[1]{\textcolor{red}{XXX #1 XXX}}
28:
29: \newcommand{\ChiSq}{\ensuremath{\chi^2}\xspace}
30: \newcommand{\HchiSq}{\ensuremath{H^{\chi^2}}\xspace}
31:
32: \newcommand{\BrownEtAl}{Brown \emph{et~al}.\xspace}
33:
34: \newcommand{\citecoll}{\cite{bib:Tyson1991, bib:Zwolak2005a, bib:Goldbeter1991, bib:Vilar2002, bib:Edelstein1996, bib:Kholodenko2000, bib:Lee2003, bib:Leloup1999, bib:Brown2004, bib:Dassow2000, bib:Ueda2001, bib:Locke2005, bib:Zak2003, bib:Curto1998, bib:Chassagnole2002, bib:Chen2004, bib:Sasagawa2005}\xspace}
35:
36: \begin{document}
37: %\vskip\abovecaptionskip
38: %\hbox to \hsize{\hfil #1\hfil}%
39: %\vskip\belowcaptionskip}
40:
41:
42: \date{\today}
43:
44: \pagestyle{myheadings}
45: \markboth{}{Sloppy Systems Biology}
46:
47: % Needs to be 75 characters or less
48: \title{Universally Sloppy Parameter Sensitivities in Systems Biology Models}
49:
50: \author{Ryan N. Gutenkunst}
51: \email{rng7@cornell.edu}
52: \affiliation{Laboratory of Atomic and Solid State Physics, Cornell University, Ithaca, NY, USA}
53: \author{Joshua J. Waterfall}
54: \affiliation{Department of Molecular Biology and Genetics, Cornell University, Ithaca, NY, USA}
55: \author{Fergal P. Casey}
56: \affiliation{UCD Conway Institute of Biomolecular \& Biomedical Research, University College Dublin, Belfield, Ireland}
57: \author{Kevin S. Brown}
58: \affiliation{Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA}
59: \author{Christopher R. Myers}
60: \affiliation{Cornell Theory Center, Cornell University, Ithaca, NY, USA}
61: \author{James P. Sethna}
62: \affiliation{Laboratory of Atomic and Solid State Physics, Cornell University, Ithaca, NY, USA}
63:
64: % 250 - 300 words
65: \begin{abstract}
66: Quantitative computational models play an increasingly important role in modern biology.
67: Such models typically involve many free parameters, and assigning their values is often a substantial obstacle to model development.
68: Directly measuring \emph{in vivo} biochemical parameters is difficult, and collectively fitting them to other experimental data often yields large parameter uncertainties.
69: Nevertheless, in earlier work we showed in a growth-factor-signaling model that collective fitting could yield well-constrained predictions, even when it left individual parameters very poorly constrained.
70: We also showed that the model had a `sloppy' spectrum of parameter sensitivities, with eigenvalues roughly evenly distributed over many decades.
71: Here we use a collection of models from the literature to test whether such sloppy spectra are common in systems biology.
72: Strikingly, we find that every model we examine has a sloppy spectrum of sensitivities.
73: We also test several consequences of this sloppiness for building predictive models.
74: In particular, sloppiness suggests that collective fits to even large amounts of ideal time-series data will often leave many parameters poorly constrained. Tests over our model collection are consistent with this suggestion.
75: This difficulty with collective fits may seem to argue for direct parameter measurements, but sloppiness also implies that such measurements must be formidably precise and complete to usefully constrain many model predictions.
76: We confirm this implication in our growth-factor-signaling model.
77: Our results suggest that sloppy sensitivity spectra are universal in systems biology models.
78: The prevalence of sloppiness highlights the power of collective fits and suggests that modelers should focus on predictions rather than on parameters.
79: \end{abstract}
80:
81: \maketitle
82:
83: \textbf{Non-technical Summary:}
84: Dynamic systems biology models typically involve many kinetic parameters, the quantitative determination of which has been a serious obstacle to using these models.
85: Previous work showed for a particular model that useful predictions could be extracted from a fit long before the experimental data constrained the parameters, even to within orders of magnitude.
86: This was attributed to a `sloppy' pattern in the model's parameter sensitivities; the sensitivity eigenvalues were roughly evenly spaced over many decades.
87: Consequently, the model behavior depended effectively on only a few `stiff' parameter combinations.
88: Here we study the converse problem, showing that direct parameter measurements are very inefficient at constraining the model's behavior.
89: To yield effective predictions such measurements must be very precise and complete; even a single imprecise parameter often destroys predictivity.
90: We also show here that the characteristic sloppy eigenvalue pattern is reproduced in sixteen other diverse models from the systems biology literature.
91: The apparent universality of sloppiness suggests that predictions from most models will be very fragile to single uncertain parameters and that collective parameters fits can often yield tight predictions with loose parameters.
92: Together these results argue that focusing on parameter values may be a very inefficient route to useful models.
93:
94: \noindent\hrulefill
95:
96: Dynamic computational models are powerful tools for developing and testing
97: hypotheses about complex biological systems~\cite{bib:Kitano2002,
98: bib:Locke2005, bib:Voit2006}. It has even been suggested that such models will
99: soon replace databases as the primary means for exchanging biological
100: knowledge~\cite{bib:Aldridge2006}. A major challenge with such models,
101: however, is that they often possess tens or even hundreds of free parameters
102: whose values can significantly affect model
103: behavior~\cite{bib:Ingram2006,bib:Mayo2006}. While high-throughput methods for
104: discovering interactions are well-developed~\cite{bib:Sachs2005},
105: high-throughput methods for measuring biochemical parameters remain
106: limited~\cite{bib:Maerkl2007}. Furthermore, using values measured \emph{in vitro} in an \emph{in vivo} application may introduce substantial
107: inaccuracies~\cite{bib:Minton2001, bib:Teusink2000}. On the other hand,
108: collectively fitting parameters~\cite{bib:Mendes1998,bib:Jaqaman2006} by
109: optimizing the agreement between the model and available data often yields
110: large parameter uncertainties~\cite{bib:Brodersen1987,bib:Cho2003,bib:Rodriguez-Fernandez2006}.
111: In approaches typically more focused on steady-state distributions of
112: fluxes in metabolic networks, metabolic control analysis has been used
113: to quantify the sensitivity of model behavior with respect to
114: parameter variation~\cite{bib:Fell1997}, and flux-balance analysis and related techniques have probed the robustness of metabolic networks~\cite{bib:Wiback2004, bib:Famili2005}.
115:
116: One way to cope with the dearth of reliable parameter values is to focus on
117: predictions that are manifestly parameter-independent~\cite{bib:Bailey2001},
118: but these are mostly qualitative. An alternative is not to forsake
119: quantitative predictions, but to accompany them with well-founded
120: uncertainty estimates based on an ensemble of parameter sets statistically drawn from all sets
121: consistent with the available data~\cite{bib:Brown2003a}.
122: (Uncertainties in the model structure itself may be important in some cases.
123: Here we focus on parameter uncertainties, as they are often important on their
124: own.)
125:
126: Brown \emph{et~al}.\ took this approach in developing a computational model of the well-studied growth-factor-signaling network in PC12
127: cells~\cite{bib:Brown2004}. They collectively fit their model's 48 parameters to 68 data points from 14 cell-biology experiments (mostly Western blots). After the fit, all 48
128: parameters had large uncertainties; their 95\% confidence intervals each
129: spanned more than a factor of 50. Surprisingly, while fitting this modest
130: amount of data did not tightly constrain any single parameter value, it did
131: enable usefully tight quantitative predictions of behavior under interventions,
132: some of which were verified experimentally.
133:
134: In calculating their uncertainties, \BrownEtAl found that the quantitative
135: behavior of their model was much more sensitive to changes in certain
136: combinations of parameters than others. Moreover, the sensitivity eigenvalues were
137: approximately equally spaced in their logarithm, a pattern deemed `sloppy'.
138: Such sloppy sensitivities were subsequently seen in other multi-parameter
139: fitting problems, from interatomic potentials~\cite{bib:Frederiksen2004} to
140: sums of exponentials~\cite{bib:Waterfall2006}. The fact that sloppiness arises in such disparate contexts suggests that it
141: may be a universal property of nonlinear multi-parameter models. (Here the term `universal' has a technical meaning from statistical physics, denoting a shared common property with a deep underlying cause; see~\cite{bib:Waterfall2006}.
142: Universality in this sense does not imply that all models must necessarily share the property.)
143: %For example, a system whose experiments are concocted to constrain each parameter separately would not be sloppy.
144:
145: In this work, we begin by empirically testing seventeen systems biology
146: models from the literature, examining the sensitivity of their behavior to
147: parameter changes. Strikingly, we find that Brown \emph{et~al}'s model is not
148: unique in its sloppiness; every model we examine exhibits a sloppy parameter
149: sensitivity spectrum. (Thus, in the models we've examined sloppiness is also
150: universal in the common English sense of ubiquity.) We
151: then study the implications of sloppiness for
152: constraining parameters and predictions. We argue that obtaining precise
153: parameter values from collective fits will remain difficult even with extensive
154: time-series data, because the behavior of a sloppy model is very insensitive
155: to many parameter combinations. We also argue that, to usefully constrain model
156: predictions, direct parameter measurements must be both very precise and
157: complete, because sloppy models are also conversely very sensitive to some parameter
158: combinations. Tests over our collection of models support the first
159: prediction, and detailed analysis of the model of \BrownEtAl supports the second contention.
160:
161: Sloppiness, while not unique to biology, is particularly relevant to biology,
162: because the collective behavior of most biological systems is much
163: easier to measure \emph{in vivo} than the values of individual parameters.
164: Much work has focused on optimizing experimental design to best constrain model parameters with collective fits~\cite{bib:Faller2003,bib:Zak2003,bib:Gadkar2005}.
165: % or large programs to measure such parameters directly~\cite{XXX}.
166: We argue against this focus on parameter values,
167: particularly when our understanding of a system is tentative and incomplete. Concrete predictions can be extracted from models long before their parameters are even roughly known~\cite{bib:Brown2004}, and
168: when a system is not already well-understood, it can be more profitable to design experiments to directly improve predictions of interesting system behavior~\cite{bib:Casey2006} rather than to improve estimates of parameters.
169:
170: \section{Results}
171:
172: \subsection{Systems Biology Models have Sloppy Sensitivity Spectra}
173: \label{sec:SloppySpectra}
174:
175: Our collection of 17 systems biology models~\citecoll was drawn primarily from
176: the BioModels database~\cite{bib:BioModels}, an online repository of models
177: encoded in the Systems Biology Markup Language (SBML)~\cite{bib:Hucka2003}.
178: The collected models encompass a diverse range of biological systems, including
179: circadian rhythm, metabolism, and signaling. All the models are formulated as
180: systems of ordinary differential equations, and they range from having about ten to
181: more than two hundred parameters. In most cases, these parameters were not
182: systematically fit or measured in the original publication.
183:
184: We quantified the change in model behavior as parameters $\theta$ varied from
185: their published values $\theta^*$ by the average squared change in molecular
186: species time courses:
187: \begin{equation} \ChiSq(\theta) \equiv
188: \frac{1}{2\,N_c\,N_s} \sum_{s, c}\frac{1}{T_c} \int^{T_c}_0
189: \left[\frac{y_{s,c}(\theta, t) - y_{s,c}(\theta^*,
190: t)}{\sigma_s}\right]^2\,\mathrm{d}t,\label{eqn:ChiSq}
191: \end{equation}
192: a kind of continuous least-squares fit of parameters $\theta$ to `data'
193: simulated from published parameters $\theta^*$.
194: Here $y_{s,c}(\theta, t)$ is the time course of molecular species $s$ given
195: parameters $\theta$ in condition $c$, and $T_c$ is the `measurement' time for
196: that condition. We took the species normalization $\sigma_s$ to be equal to
197: the maximum value of species $s$ across the conditions considered; other consistent normalizations yield the same qualitative conclusions.
198:
199: For each model, the
200: sum in Equation~\ref{eqn:ChiSq} runs over all molecular species in the model
201: and (except where infeasible) over all experimental conditions considered in
202: the corresponding paper---an attempt to neutrally measure system behavior under
203: conditions deemed significant by the original authors. (The total number
204: of conditions and species are denoted by $N_c$ and $N_s$, respectively.)
205: SBML files and SloppyCell~\cite{bib:SloppyCell}
206: scripts for all models and conditions are available in Dataset S1.
207:
208: \begin{figure*}
209: \begin{center}
210: \includegraphics{sloppiness_lines}
211: \caption{Parameter sensitivity spectra\\
212: Subfigure A illustrates the quantities we calculate from \HchiSq, while subfigures B and C show that all the models we examined have sloppy sensitivity spectra.\\
213: A: Analyzing \HchiSq corresponds to approximating the surfaces of constant
214: model behavior change (constant \ChiSq)
215: as ellipsoids. The width of each principal axis is proportional to one over the
216: square root of the corresponding eigenvalue. The inner ellipsoid's projection
217: onto and intersection with the $\log \theta_1$ axis are denoted $P_1$
218: and $I_1$, respectively.\\
219: B: Plotted are the eigenvalue spectra of \HchiSq for our collection of systems biology models. The many decades generally spanned indicate the ellipses have very large aspect ratio. (The spectra have each been normalized by their largest eigenvalue. Not all values are visible for all models.)\\
220: C: Plotted is the spectrum of $I/P$ for each parameter
221: in each model in our collection. Generally very few parameters have $I/P \approx 1$, suggesting that the ellipses are skewed from the bare parameter axes. (Not all values are visible for all models.)\\
222: The models are ordered by increasing number of free parameters and are:
223: (a) eukaryotic cell cycle~\protect\cite{bib:Tyson1991},
224: (b) Xenopus egg cell cycle~\protect\cite{bib:Zwolak2005a},
225: (c) eukaryotic mitosis~\protect\cite{bib:Goldbeter1991},
226: (d) generic circadian rhythm~\protect\cite{bib:Vilar2002},
227: (e) nicotinic acetylcholine intra-receptor dynamics~\protect\cite{bib:Edelstein1996},
228: (f) generic kinase cascade~\protect\cite{bib:Kholodenko2000},
229: (g) Xenopus Wnt signaling~\protect\cite{bib:Lee2003},
230: (h) Drosophila circadian rhythm~\protect\cite{bib:Leloup1999},
231: (i) rat growth-factor signaling~\protect\cite{bib:Brown2004},
232: (j) Drosophila segment polarity~\protect\cite{bib:Dassow2000},
233: (k) Drosophila circadian rhythm~\protect\cite{bib:Ueda2001},
234: (l) Arabidopsis circadian rhythm~\protect\cite{bib:Locke2005},
235: (m) \emph{in silico} regulatory network~\protect\cite{bib:Zak2003},
236: (n) human purine metabolism~\protect\cite{bib:Curto1998},
237: (o) E. coli carbon metabolism~\protect\cite{bib:Chassagnole2002},
238: (p) budding yeast cell cycle~\protect\cite{bib:Chen2004},
239: (q) rat growth-factor signaling~\protect\cite{bib:Sasagawa2005}.}
240: \label{fig:sloppiness} \end{center}
241: \end{figure*}
242:
243: To analyze each model's sensitivity to parameter variation, we considered the
244: Hessian matrix corresponding to \ChiSq:
245: \begin{equation}
246: \HchiSq_{j,k} \equiv
247: \frac{d^2 \chi^2(\theta)}{d \log \theta_j\,d \log
248: \theta_k}.\label{eqn:HchiSq}
249: \end{equation}
250: We took derivatives with respect to
251: $\log \theta$ to consider \emph{relative} changes in parameter values, because
252: biochemical parameters can have different units and widely varying scales.
253: Analyzing \HchiSq corresponds to approximating the surfaces of constant model
254: behavior deviation (as quantified by \ChiSq) to be $N_p$-dimensional
255: ellipsoids, where $N_p$ is the number of parameters in the model.
256: Figure~\ref{fig:sloppiness}A schematically illustrates these ellipsoids and
257: some of their characteristics. (Details of calculating \HchiSq and related
258: quantities are found in Methods. Dataset S1 includes \HchiSq for each model.)
259:
260: The principal axes of the ellipsoids are the eigenvectors of \HchiSq, and the
261: width of the ellipsoids along each principal axis is proportional to one over
262: the square root of the corresponding eigenvalue. The narrowest axes are called
263: `stiff', and the broadest axes `sloppy'~\cite{bib:Brown2003a}. The eigenvalue
264: spectra for the models in our collection are shown in
265: Figure~\ref{fig:sloppiness}B (each normalized by its largest eigenvalue). In
266: every case, the eigenvalues span many decades. All but one span more than $10^6$,
267: indicating that the sloppiest axes of the ellipsoids illustrated in
268: Figure~\ref{fig:sloppiness}A are generally more than one thousand times as long as the
269: stiffest axes. In each spectrum the eigenvalues are also approximately evenly
270: spaced in their logarithm; there is no well-defined cutoff between `important'
271: and `unimportant' parameter combinations.
272:
273: The Hessian matrix is a local quadratic approximation to the generally nonlinear \ChiSq function. Principal component analysis of extensive Monte Carlo runs in the \BrownEtAl model, however, indicates that the sloppiness revealed by \HchiSq is indicative of full nonlinear \ChiSq function~\cite{bib:Brown2003a}.
274:
275: Along with their relative widths, the degree to which the principal axes of the
276: ellipsoids are aligned to the bare parameter axes is also important. We
277: estimated this by comparing the ellipsoids' intersections $I_i$ with
278: and projections $P_i$ onto each bare parameter axis $i$. If
279: $I_i/P_i = 1$ then one of the principal axes of the
280: ellipsoids lies along bare parameter direction $i$.
281: Figure~\ref{fig:sloppiness}C plots the $I/P$ spectrum for each
282: model. In general, very few axes have $I/P \approx 1$; the
283: ellipses are skewed from single parameter directions.
284:
285: Naively, one might expect the stiff eigenvectors to embody the most
286: important parameters and the sloppy directions to embody parameter correlations
287: that might suggest removable degrees of freedom, simplifying the model.
288: Empirically, we have found that the eigenvectors often tend to involve significant components of many different parameters; plots of the five stiffest eigenvectors for each model are in Supporting Text S1.
289: This is understandable theoretically; the nearly-degenerate sloppy eigenvectors should mix, and the stiff eigenvectors can include arbitrary admixtures of unimportant directions to a given important parameter combination.
290: (Indeed, in analogous random-matrix theories the eigenvectors are known to be uncorrelated random vectors~\cite{bib:Mehta2004}.)
291: While the relatively random eigenvectors studied here may not be useful
292: in guiding model reduction, more direct explorations of parameter
293: correlations have yielded interesting correlated parameter
294: clusters~\cite{bib:WaterfallPhD}.
295: %Here we focus on model behavior; other approaches to model reduction are being pursued elsewhere~\cite{bib:WaterfallPhD}.
296:
297: These characteristic parameter sensitivities that evenly span many decades and are skewed from bare parameter axes
298: define a `sloppy' model~\cite{bib:Brown2003a}.
299: Figures~\ref{fig:sloppiness}B and~\ref{fig:sloppiness}C show that every model we have examined has a sloppy sensitivity spectrum. Next we discuss some broad questions about the relation between model predictions, collective fits, and parameter measurements and see how the sloppy properties of these models may suggest answers.
300:
301: \subsection{Consequences of Sloppiness}
302:
303: \begin{figure} \begin{center}
304: \includegraphics{explanation}
305: \caption{Sloppiness and uncertainties\\
306: As in Figure~\ref{fig:sloppiness}A, the contours
307: represent surfaces of constant model behavior deviation. The clouds of points
308: represent parameter set ensembles.\\
309: A: Collective fitting of model parameters
310: naturally constrains the parameter set ensemble along stiff directions and
311: allows it to expand along sloppy directions. The resulting ensemble may be very
312: large, yet encompass little variation in model behavior, yielding small
313: prediction uncertainties despite large parameter uncertainties. ($\Sigma_1$
314: denotes the 95\% confidence for the value of $\theta_1$.)\\
315: B: If all parameters
316: are directly measured to the same precision, the parameter set ensemble is
317: spherical. The measurement precision required for well-constrained predictions
318: is set by the stiffest direction.\\
319: C: If one parameter (here $\theta_2$) is
320: known less precisely than the rest, the cloud is ellipsoidal. If not aligned
321: with a sloppy direction, the cloud will admit many model behaviors and yield
322: large prediction uncertainties. (Note that the aspect ratio of the real
323: contours can be greater than 1000.) }
324: \label{fig:explanation}
325: \end{center}
326: \end{figure}
327:
328: The difficulty of extracting precise parameter values from collective fits in systems biology modeling is well-known~\cite{bib:Gadkar2005}.
329: %It is well known that collective fits of parameters in systems biology models often yield large parameter uncertainties~\cite{XXX}.
330: Sloppiness offers an explanation for this
331: and predicts that it will
332: be true even for fitting to complete data that the model can fit perfectly. In a collective fit, the parameter set
333: ensemble samples from all sets of parameters for which the model behavior is
334: consistent with the data. Because sloppy models are very insensitive to
335: parameter combinations that lie along sloppy directions, the parameter set
336: ensemble can extend very far in those directions, as illustrated schematically
337: in Figure~\ref{fig:explanation}A. As a result, individual parameters can be
338: very poorly determined (\emph{e.g.}, confidence interval indicated by $\Sigma_1$ in
339: Figure~\ref{fig:explanation}A). Below we discuss a test of this prediction over all the models
340: in our collection.
341:
342: Unless one has direct interest in the kinetic constants for the underlying
343: reactions, uncertainties in model predictions are generally more
344: important than uncertainties in model parameters. The parameter set ensemble illustrated in
345: Figure~\ref{fig:explanation}A yields large uncertainties on individual
346: parameters, but can yield small uncertainties on predictions. While the
347: fitting process allows the ensemble to expand along sloppy directions, the
348: fit naturally constrains the ensemble along stiff directions, so that model behavior varies
349: little within the ensemble, and predictions can be consequently tight.
350:
351: Direct parameter measurements, on the other hand, will have uncertainties that
352: are uncorrelated with the model's underlying stiff and sloppy directions. For
353: example, if all parameter measurements are of the same precision, the parameter
354: set ensemble is spherical, as illustrated in Figure~\ref{fig:explanation}B.
355: For tight predictions, this ensemble must not cross many contours, so the
356: required precision is set by the stiffest direction of the model.
357: Consequently, high precision parameter measurements are required to yield tight
358: predictions. Moreover, these measurements must be complete. If one parameter
359: is known less precisely, the parameter set ensemble expands along that
360: parameter axis, as illustrated in Figure~\ref{fig:explanation}C. If that axis
361: is not aligned with a sloppy direction, model behavior will vary dramatically
362: across the parameter set ensemble and predictions will have correspondingly
363: large uncertainties. Below we discuss tests of both these notions, exploring the effects of
364: direct parameter measurement uncertainty on predictions of a particular model.
365:
366: \subsubsection{Parameter Values from Collective Fits}\label{sec:params_from_fits}
367:
368: \begin{figure}
369: \begin{center}
370: \includegraphics{fitting_params}
371: \caption{Fitting parameters to idealized data\\
372: Shown are histograms of the relative confidence interval size $\Sigma$ for each
373: parameter in each model of our collection, after fitting 100 times as many
374: time-series data points (each with 10\% uncertainty) as parameters. In most
375: cases a large number of parameters are left with greater than 100\%
376: uncertainty. (A parameter constrained with 95\% probability to lie between 1
377: and 100 would have $\Sigma \approx 100$.)\\
378: Labels are as in Figure~\ref{fig:sloppiness}.}\label{fig:fitting_params}
379: \end{center}
380: \end{figure}
381:
382: Does the sloppiness of these models really prevent one from extracting
383: parameters from collective fits? Here we discuss a test of this prediction using an idealized fitting procedure.
384:
385: Our \ChiSq measure of model behavior change (Equation~\ref{eqn:ChiSq})
386: corresponds to the cost function for fitting model parameters to
387: continuous time-series data that the model fits perfectly at parameters
388: $\theta^*$; \HchiSq is the corresponding Fisher information matrix
389: (Equation~\ref{eqn:HchiSq}). We used this idealized situation to test the
390: prediction that collective fits will often poorly constrain individual
391: parameters in our collection of sloppy models.
392:
393: We defined the relative 95\% confidence interval size $\Sigma_i$ as the ratio
394: between parameter $i$ at the upper and lower extremes of the interval, minus
395: one. (A parameter value constrained after the fit to lie between 10 and 1000
396: would have $\Sigma \approx 100$, while one constrained between 1.0 and 1.5 would have
397: $\Sigma = 0.5$.) We assumed 100 times as many data points (each with 10\% uncertainty) as the number of parameters in each model. Figure~\ref{fig:fitting_params} shows histograms of the
398: quadratic approximation to $\Sigma$ for each parameter in each model after
399: fitting such data. (See Methods.) For most of the models, the figure indicates that such fitting leaves many parameters with
400: greater than 100\% uncertainty ($\Sigma > 1$).
401: Indeed, even fitting this large amount of
402: ideal data can leave many parameter values very poorly determined, as
403: expected from the sloppiness of these models and our discussion of
404: Figure~\ref{fig:explanation}A.
405:
406: The fact that nonlinear multiparameter models often allow a wide range of correlated parameters to fit data has long been appreciated. As one example, a 1987 paper by Brodersen \emph{et~al}.\ on ligand binding to hemoglobin and albumin empirically found many sets of parameters that acceptably fit experimental data, with individual parameter values spanning huge ranges~\cite{bib:Brodersen1987}.
407: Our sloppy model perspective (\cite{bib:Brown2003a,bib:Brown2004,bib:Waterfall2006}, Figure 1) shows that there is a deep underlying universal pattern in such least-squares fitting.
408: Indeed, an analysis of the acceptable binding parameter sets from the 1987 study shows the same characteristic sloppy eigenvalue spectrum we observed in Figure~\ref{fig:sloppiness}B (Supporting Text~S5).
409:
410: \subsubsection{Predictions from Direct Parameter Measurements}\label{sec:pred_from_meas}
411:
412: \begin{figure}
413: \begin{center}
414: \includegraphics{uncerts_nobf}
415: \caption{Parameter and prediction uncertainties\\
416: A: Our example prediction is for ERK activity upon EGF stimulation given PI3K
417: inhibition in this 48-parameter model of growth-factor-signaling in PC12
418: cells~\protect\cite{bib:Brown2004}.\\
419: B: Shaded regions are 95\% confidence intervals calculated via exhaustive Monte Carlo for our example
420: prediction given various scenarios for constraining parameter values.\\
421: C: Plotted is the relative size $\Sigma$ of the 95\%
422: confidence interval for each parameter.\\
423: The scenarios represented are: (red,
424: squares) all model parameters individually measured to high precision, (blue,
425: triangles) all parameters precisely measured, except one estimated to low
426: precision, (yellow, circles) all parameters collectively fit to 14 real cell-biology
427: experiments. Precisely measured individual parameter values enable a tight
428: prediction (B: middle red band), but even one poorly known parameter can
429: destroy predictive power (B: wide blue band). In contrast, the collective fit
430: yields a tight prediction (B: tightest yellow band) but only very loose
431: parameter constraints (C: circles).
432: The large parameter uncertainties from the collective fit (C: circles) calculated here by Monte Carlo are qualitatively similar to those seen in the linearized fit to idealized data (Figure~\ref{fig:fitting_params}, model (i)). (For clarity, the dashed red lines trace the boundary of the red confidence interval.)
433: }\label{fig:uncerts}
434: \end{center}
435: \end{figure}
436:
437: Figures~\ref{fig:explanation}B and~\ref{fig:explanation}C suggests that direct parameter measurements must be both precise and complete to usefully constrain predictions in sloppy systems. Here we discuss a test of this notion in a specific model.
438:
439: We worked with the
440: 48-parameter growth-factor-signaling model of \BrownEtAl, shown schematically
441: in Figure~\ref{fig:uncerts}A~\cite{bib:Brown2004}. The parameters in this
442: model were originally collectively fit to 14 time-series cell-biology
443: experiments. We focused on this model because it is instructive to compare
444: our results concerning direct parameter measurements with prior results from
445: collective fitting. For our analysis, we assumed that hypothetical direct
446: parameter measurements would be centered about the original best-fit values.
447:
448: One important test of the model was a prediction of the time-course of ERK
449: activity upon EGF stimulation, given inhibition of the PI3K branch of the
450: pathway. The yellow shaded region in Figure~\ref{fig:uncerts}B shows the
451: uncertainty bound on this prediction from the original collective
452: fit, calculated by exhaustive Monte Carlo~\cite{bib:Brown2004}. The tightness of this prediction is remarkable
453: considering the huge uncertainties the collective fit left in the individual
454: parameter values (yellow circles in Figure~\ref{fig:uncerts}C). Not a single
455: parameter was constrained to better than a factor of 50.
456:
457: How precise would direct parameter measurements have to be to yield as tight a
458: prediction as the collective fit? For this prediction, the PI3K branch
459: (inhibited) and C3G branch (NGF-dependent) of the pathway are irrelevant in the
460: model; the remaining reactions involve 24 parameters. To achieve the red
461: prediction in Figure~\ref{fig:uncerts}B, all 24 involved parameters must be
462: measured to within a factor of plus or minus 25\% (Figure~\ref{fig:uncerts}C,
463: red squares). With current techniques, measuring even a single \emph{in vivo}
464: biochemical parameter to such precision would be a challenging experiment.
465: Such high precision is required because, as illustrated in
466: Figure~\ref{fig:explanation}B, the measurements need to constrain the stiffest
467: combination of model parameters.
468:
469: What if a single parameter is left unmeasured? For example, consider high
470: precision measurements of 23 of the 24 involved parameters, all but the rate
471: constant for the activation of Mek by Raf1. For this unmeasured parameter, we
472: assumed that an informed estimate could bound it at 95\% confidence to within
473: a total range of 1000 (\emph{e.g.}, between $1 s^{-1}$ and $1000 s^{-1}$). The resulting prediction
474: (blue in Figure~\ref{fig:uncerts}B) has very large uncertainty and would likely
475: be useless. Note that these hypothetical measurements constrain every
476: individual parameter value more tightly than the original collective fit (blue
477: triangles versus yellow circles in Figure~\ref{fig:uncerts}C), yet the
478: prediction is much less well-constrained. Neither this parameter nor this
479: prediction is unique. Uncertainty for this prediction is large if any one of
480: about 18 of the 24 involved parameters is unmeasured (Supporting Text S2). Furthermore,
481: other possible predictions in this model are similarly fragile to single
482: unmeasured parameters (Supporting Text S3).
483:
484: To usefully constrain Brown \emph{et~al}.'s model, direct parameter measurements would need to be both precise and complete. By contrast, collective parameter fitting yielded tight predictions with only a modest number of experiments. These results are expected given the model's sloppiness.
485:
486: \section{Discussion}
487: By examining seventeen models from the systems biology literature~\citecoll,
488: we showed that their parameter sensitivities all share striking common
489: features deemed `sloppiness'; the sensitivity eigenvalues span many
490: decades roughly evenly (Figure~\ref{fig:sloppiness}B), and tend not to
491: be aligned with single parameters (Figure~\ref{fig:sloppiness}C).
492: We argued that sloppy parameter sensitivities help explain the difficulty of extracting precise parameter estimates from collective fits, even from comprehensive data.
493: Additionally, we argued that direct parameter measurements should be inefficient at constraining predictions from sloppy models.
494: We then showed that collective parameter fits to complete time-series
495: data do indeed yield large parameter uncertainties in our model collection
496: (Figure~\ref{fig:fitting_params}).
497: Finally, we confirmed for the
498: 48-parameter signaling model of \BrownEtAl~\cite{bib:Brown2004} that
499: direct parameter measurements must be formidably precise and complete
500: to usefully constrain model predictions (Figure~\ref{fig:uncerts}).
501:
502: What causes sloppiness?
503: (1)~Fundamentally, sloppiness involves an extraordinarily singular coordinate transformation in parameter space between the bare parameters natural in biology (\emph{e.g.}, binding affinities and rate constants) and the eigenparameters controlling system behavior, as discussed in~\cite{bib:Waterfall2006}.
504: Both experimental interventions and biological evolution work in the bare parameter space, so this parameterization is fundamental to the system, not an artifact of the modeling process.
505: (2)~Sloppiness depends not just upon the model, but also on the data
506: it is fit to; exhaustive experiments designed to decouple the system and
507: separately measure each parameter will naturally not yield sloppy
508: parameter sensitivities.
509: (3)~In biological systems fit to time-series data, Brown and Sethna~\cite{bib:Brown2003a} note that sloppiness may arise due to
510: under-determined systems, proximity to bifurcations, and separation of time or concentration scales, but they doubt that these can explain all the sloppiness found in their model.
511: Our analysis includes complete data on all species, and hence is overdetermined.
512: Small eigenvalues near bifurcations are associated with
513: dynamic variables, and not the system parameters we investigate.
514: To study the effect of time and concentration scales we calculated \HchiSq for a version of the \BrownEtAl model in which all concentrations and rate constants were scaled to one~\cite{bib:SiggiaNote}. The resulting model remains sloppy, with eigenvalues roughly uniformly spanning five decades (Supporting Text S4).
515: %We are not certain of the underlying cause of the remaining sloppiness.
516: (4)~Motivated by simple example systems, we have argued elsewhere that sloppiness emerges from a redundancy between the effects of different parameter combinations~\cite{bib:Waterfall2006}.
517: We are presently investigating decompositions of parameter space
518: into sloppy subsystems~\cite{bib:WaterfallPhD} and the use of physically or biologically motivated nonlinear coordinate changes
519: to remove sloppiness or motivate simpler models.
520: These potential methods for model refinement, however, demand a
521: complete and sophisticated understanding of the system that is
522: unavailable for many biological systems of current interest.
523:
524: %Sloppiness may also provide a more quantitative understanding of robustness~\cite{bib:Wagner2005} in certain contexts;
525: %we too find that our system behavior is remarkably independent of parameter
526: %choice, depending on only three or four stiff directions rather than
527: %forty or fifty independent parameter constraints.
528:
529: Parameter estimation has been a serious obstacle in systems biology modeling.
530: With tens of unknown parameters, a typical modeling effort might draw some values
531: from the literature (possibly from \emph{in vitro} measurements or different cell lines)~\cite{bib:Kholodenko2000, bib:Curto1998},
532: set classes of constants to the same value (\emph{e.g.}, phosphorylation rates)~\cite{bib:Vilar2002, bib:Edelstein1996, bib:Sasagawa2005},
533: and adjust key parameters to qualitatively best fit the existing data~\cite{bib:Ueda2001,bib:Chen2004, bib:Locke2005}.
534: In retrospect, these approaches may be successful because the
535: models are sloppy---they can be tuned to reality by adjusting one key parameter per stiff direction, independently of how reliably the other parameters are estimated.
536:
537: Computational modeling is a potentially invaluable tool for extrapolating from current experiments and distinguishing between models.
538: But we cannot trust the predictions of these models without testing how much they depend on uncertainties in these estimated parameters.
539: Conversely, if we insist upon a careful uncertainty analysis, it would seem unnecessary to insist upon tight prior estimates of the parameters, since they do not significantly enhance model predictivity.
540: Because the behavior of a sloppy model is dominated by a few stiff directions that nonetheless involve almost all the parameters, direct parameter measurements constrain predictions much less efficiently than comparably difficult experiments probing collective system behavior.
541:
542: Our suggestion of making predictions from models with very poorly known parameters may appear dangerous.
543: A model with tens or hundreds of unmeasured parameters might seem completely untrustworthy; we certainly believe that any prediction derived solely from a best-fit set of parameters is of little value.
544: Uncertainty bounds derived from rigorous sensitivity analysis, however, distinguish those predictions that can be trusted from those that cannot.
545: Of course, successful fits and predictions may arise from models that are incorrect in significant ways; for example, one model pathway with adjusted parameters may account for two parallel pathways in the real system.
546: A model that is wrong in some details may nevertheless be useful in guiding and interpreting experiments.
547: For computational modeling to be useful in incompletely understood systems, we must focus not on building the final, perfect, model with all parameters precisely determined, but on building incomplete, tentative and falsifiable models in the most expressive and predictive fashion feasible.
548:
549: Given that direct parameters measurements do not efficiently constrain model
550: behavior, how do we suggest that experimentalists decide what experiment
551: to do next? If the goal is to test the assumptions underlying a model,
552: one should look for predictions with tight uncertainty estimates that can
553: be readily tested experimentally. If the goal is to reduce uncertainty in
554: crucial model predictions, one must invoke the statistical methods of
555: optimal experimental design, which we have studied
556: elsewhere~\cite{bib:Casey2006} and which may be conveniently implemented
557: in modeling environments that incorporate sensitivity analysis (such as
558: SloppyCell~\cite{bib:SloppyCell}).
559:
560: In our approach, the model and its parameters cannot be treated in isolation from the data that informed model development and parameter fitting.
561: This complicates the task of exchanging knowledge in the modeling community.
562: To support our approach, standards such as SBML~\cite{bib:Hucka2003} that facilitate automated model
563: exchange will need to be extended to facilitate automated data exchange.
564:
565: Every one of the 17 systems biology models we studied exhibits a sloppy
566: spectrum of parameter sensitivity eigenvalues; they all span many decades roughly
567: evenly and tend not be aligned with single parameters. This
568: striking and apparently universal feature has important consequences for the modeling process. It suggests that modelers would be wise to try collective parameter fits and to focus not on the quality of their parameter values but on the quality of their predictions.
569:
570: \section{Methods}\label{sec:methods}
571: \subsection{Hessian Computations}
572: \HchiSq can
573: be calculated as
574: \begin{equation} \HchiSq_{j,k} = \frac{1}{N_c\,N_s} \sum_{s,
575: c}\frac{1}{T_c\,\sigma_s^2} \int^{T_c}_0 {\frac{d \, y_{s,c}(\theta^*,
576: t)}{d \log \theta_j}} {\frac{d\, y_{s, c}(\theta^*, t)}{d \log \theta_k}} \bigg|_{\theta^*}\,\mathrm{d}t.\label{eqn:HchiSqEx}
577: \end{equation}
578: Second derivative terms $\left({d^2\, y_{s,c}(\theta^*,
579: t)}/{d \log \theta_i \, d \log \theta_j}\right)$ might be
580: expected, but they vanish because we evaluate \HchiSq at $\theta^*$.
581: Equation~\ref{eqn:HchiSqEx} is convenient because the first derivatives
582: ${d\, y_{s, c}(\theta^*, t)}/{d \log \theta_k}$ can be calculated by
583: integrating sensitivity equations. This avoids the use of finite-difference
584: derivatives, which are troublesome in sloppy systems.
585:
586: The projections $P_i$ of the ellipsoids shown in
587: Figure~\ref{fig:explanation}A onto bare parameter axis $i$ are proportional to
588: $\sqrt{\big(\operatorname{inv}\HchiSq\big)_{i,i}}$. The intersections
589: $I_i$ with axis $i$ are proportional to $\sqrt{1/\HchiSq_{i,i}}$,
590: with the same proportionality constant.
591:
592: \subsection{Parameter Uncertainties}\label{sec:methods_fit}
593: To rescale \HchiSq so that it corresponds
594: to fitting $N_d$ data points, each with uncertainty a fraction $f$ of the
595: species' maximal value, we multiply \HchiSq by $N_d/f^2$. In the quadratic
596: approximation, the one-standard-deviation uncertainty in the logarithm of
597: parameter $\theta_i$ after such a collective fit is given by $\sigma^2_{\log
598: \theta_i} = \left(f^2/N_d\right) \big(\operatorname{inv}\HchiSq\big)_{i,i}$. The relative
599: size of the 95\% confidence interval is then $\Sigma_{i} = \exp\left(4\sigma_{\log
600: \theta_i}\right) - 1$.
601:
602: \subsection{Prediction Uncertainties}\label{sec:methods_pred_uncerts}
603: The red and blue prediction uncertainties
604: shown in Figure~\ref{fig:uncerts}B were calculated by randomly generating 1000
605: parameter sets consistent with the stated parameter uncertainties. (For each
606: parameter $\theta_i$, the logarithm of its value is drawn from a normal
607: distribution with mean $\log \theta_i$ and standard deviation $\sigma_{\log
608: \theta_i}$ specified by desired $\Sigma$.) For each parameter set, the Erk time
609: course was calculated, and at each timepoint the shaded regions in the figure
610: contain the central 95\% of the time courses.
611:
612: \subsection{Software}
613: All computations were performed in the open-source
614: modeling environment SloppyCell, version 0.81~\cite{bib:SloppyCell}. SBML files
615: and SloppyCell scripts to reproduce all presented calculations are in Dataset S1.
616:
617: \section{Supporting Information}
618: Text S1: Stiffest Eigenvectors
619:
620: Text S2: Effect of Other Poorly Determined Parameters
621:
622: Text S3: Fragility of Other Predictions
623:
624: Text S4: Rescaled Model of \BrownEtAl
625:
626: Text S5: Eigenvalue Analysis of Brodersen \emph{et al.}\ Binding Studies
627:
628: Dataset S1: SBML Files, SloppyCell Scripts, and \ChiSq-Hessians
629:
630: \subsection{Accession Numbers}
631: Models discussed that are in the BioModels database~\cite{bib:BioModels} are:
632: (a)~BIOMD0000000005,
633: (c)~BIOMD0000000003,
634: (d)~BIOMD0000000035,
635: (e)~BIOMD0000000002,
636: (f)~BIOMD0000000010,
637: (h)~BIOMD0000000021,
638: (i)~BIOMD0000000033,
639: (k)~BIOMD0000000022,
640: (l)~BIOMD0000000055,
641: (n)~BIOMD0000000015,
642: (o)~BIOMD0000000051,
643: (p)~BIOMD0000000056,
644: (q)~BIOMD0000000049.
645:
646: \section{Acknowledgments}
647: We thank Eric Siggia for suggesting study of the rescaled model of \BrownEtAl
648: We also thank Rick Cerione and Jon Erickson for sharing their biological insights and John Guckenheimer, Eric Siggia, and Kelvin Lee for helpful discussions about dynamical systems.
649: Computing resources were kindly provided by the USDA-ARS plant pathogen systems biology group in Ithaca, NY.
650: Finally, we thank several anonymous reviewers whose comments strengthened the manuscript.
651:
652: \section{Funding}
653: RNG was supported by an NIH Molecular Biophysics Training Grant, T32-GM-08267.
654: JJW was supported by a DOE Computational Science Graduate Fellowship.
655: CRM acknowledges support from USDA-ARS project 1907-21000-017-05.
656: This work was supported by NSF grant DMR-0218475.
657:
658: \bibliography{sloppy}
659:
660: %\clearpage
661: %\listoffigures
662:
663: \end{document}
664:
665: %Theoretical work has defined a class of
666: %sloppy matrices, and showed that one way sloppiness can arise is from
667: %redundancy in the parameters naturally used to quantify a
668: %system~\cite{bib:Waterfall2006}.
669: