1: %%
2: %% Beginning of file 'sample.tex'
3: %%
4: %% Modified 2005 December 5
5: %%
6: %% This is a sample manuscript marked up using the
7: %% AASTeX v5.x LaTeX 2e macros.
8:
9: %% The first piece of markup in an AASTeX v5.x document
10: %% is the \documentclass command. LaTeX will ignore
11: %% any data that comes before this command.
12:
13: %% The command below calls the preprint style
14: %% which will produce a one-column, single-spaced document.
15: %% Examples of commands for other substyles follow. Use
16: %% whichever is most appropriate for your purposes.
17: %%
18: %%\documentclass[12pt,preprint]{aastex}
19:
20: %% manuscript produces a one-column, double-spaced document:
21:
22: %\documentclass[manuscript]{aastex}
23:
24: %% preprint2 produces a double-column, single-spaced document:
25:
26: \documentclass[preprint2]{aastex}
27:
28: %% Sometimes a paper's abstract is too long to fit on the
29: %% title page in preprint2 mode. When that is the case,
30: %% use the longabstract style option.
31:
32: %% \documentclass[preprint2,longabstract]{aastex}
33:
34: %% If you want to create your own macros, you can do so
35: %% using \newcommand. Your macros should appear before
36: %% the \begin{document} command.
37: %%
38: %% If you are submitting to a journal that translates manuscripts
39: %% into SGML, you need to follow certain guidelines when preparing
40: %% your macros. See the AASTeX v5.x Author Guide
41: %% for information.
42:
43: \newcommand{\vdag}{(v)^\dagger}
44: \newcommand{\myemailFF}{Fabio.Frescura@wits.ac.za}
45: \newcommand{\myemailCE}{chrise@uj.ac.za}
46:
47: %% You can insert a short comment on the title page using the command below.
48:
49: %\slugcomment{Not to appear in Nonlearned J., 45.}
50:
51: %% If you wish, you may supply running head information, although
52: %% this information may be modified by the editorial offices.
53: %% The left head contains a list of authors,
54: %% usually a maximum of three (otherwise use et al.). The right
55: %% head is a modified title of up to roughly 44 characters.
56: %% Running heads will not print in the manuscript style.
57:
58: \shorttitle{Significance of Periodogram Peaks} \shortauthors{Frescura, Engelbrecht
59: and Frank}
60:
61: %% This is the end of the preamble. Indicate the beginning of the
62: %% paper itself with \begin{document}.
63:
64: \begin{document}
65:
66: %% LaTeX will automatically break titles if they run longer than
67: %% one line. However, you may use \\ to force a line break if
68: %% you desire.
69:
70: \title{Significance Tests for Periodogram Peaks}
71:
72: %% Use \author, \affil, and the \and command to format
73: %% author and affiliation information.
74: %% Note that \email has replaced the old \authoremail command
75: %% from AASTeX v4.0. You can use \email to mark an email address
76: %% anywhere in the paper, not just in the front matter.
77: %% As in the title, use \\ to force line breaks.
78:
79: \author{F. A. M. Frescura}
80: \affil{Centre for Theoretical Physics, University of the Witwatersrand, Private Bag
81: 3, WITS 2050, South Africa} \email{\myemailFF}
82:
83: \author{C. A. Engelbrecht}
84: \affil{Department of Physics, University of Johannesburg, PO Box 524, AUCKLAND PARK
85: 2006, South Africa} \email{\myemailCE}
86:
87: \and
88:
89: \author{B. S. Frank}
90: \affil{School of Physics, University of the Witwatersrand, Private Bag 3, WITS 2050,
91: South Africa}
92:
93: %% Notice that each of these authors has alternate affiliations, which
94: %% are identified by the \altaffilmark after each name. Specify alternate
95: %% affiliation information with \altaffiltext, with one command per each
96: %% affiliation.
97:
98: % \altaffiltext{1}{Visiting Astronomer, Cerro Tololo Inter-American Observatory.
99: % CTIO is operated by AURA, Inc.\ under contract to the National Science
100: % Foundation.}
101: % \altaffiltext{2}{Society of Fellows, Harvard University.}
102:
103: %% Mark off your abstract in the ``abstract'' environment. In the manuscript
104: %% style, abstract will output a Received/Accepted line after the
105: %% title and affiliation information. No date will appear since the author
106: %% does not have this information. The dates will be filled in by the
107: %% editorial office after submission.
108:
109: \begin{abstract}
110: We discuss methods currently in use for determining the significance of peaks in the
111: periodograms of time series. We discuss some general methods for constructing
112: significance tests, false alarm probability functions, and the role played in these
113: by independent random variables and by empirical and theoretical cumulative
114: distribution functions. We also discuss the concept of ``independent frequencies" in
115: periodogram analysis. We propose a practical method for estimating the significance
116: of periodogram peaks, applicable to all time series irrespective of the spacing of
117: the data. This method, based on Monte Carlo simulations, produces significance tests
118: that are tailor-made for any given astronomical time series.
119: \end{abstract}
120:
121: %% Keywords should appear after the \end{abstract} command. The uncommented
122: %% example has been keyed in ApJ style. See the instructions to authors
123: %% for the journal to which you are submitting your paper to determine
124: %% what keyword punctuation is appropriate.
125:
126: \keywords{Methods: data analysis --- Methods: statistical --- Stars: oscillations }
127:
128: %\keywords{globular clusters: general --- globular clusters:
129: % individual(NGC 6397, NGC 6624, NGC 7078, Terzan 8}
130:
131:
132: %% From the front matter, we move on to the body of the paper.
133: %% In the first two sections, notice the use of the natbib \citep
134: %% and \citet commands to identify citations. The citations are
135: %% tied to the reference list via symbolic KEYs. The KEY corresponds
136: %% to the KEY in the \bibitem in the reference list below. We have
137: %% chosen the first three characters of the first author's name plus
138: %% the last two numeral of the year of publication as our KEY for
139: %% each reference.
140:
141:
142: %% Authors who wish to have the most important objects in their paper
143: %% linked in the electronic edition to a data center may do so by tagging
144: %% their objects with \objectname{} or \object{}. Each macro takes the
145: %% object name as its required argument. The optional, square-bracket
146: %% argument should be used in cases where the data center identification
147: %% differs from what is to be printed in the paper. The text appearing
148: %% in curly braces is what will appear in print in the published paper.
149: %% If the object name is recognized by the data centers, it will be linked
150: %% in the electronic edition to the object data available at the data centers
151: %%
152: %% Note that for sources with brackets in their names, e.g. [WEG2004] 14h-090,
153: %% the brackets must be escaped with backslashes when used in the first
154: %% square-bracket argument, for instance, \object[\[WEG2004\] 14h-090]{90}).
155: %% Otherwise, LaTeX will issue an error.
156:
157:
158: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
159: %
160: % TEXT OF PAPER
161: %
162: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
163:
164:
165: \section{INTRODUCTION}
166:
167: Periodogram analysis is a vital ingredient of asteroseismology. It is used to
168: identify periodicities of oscillations in the observed star. Typically, the data
169: analysed are noisy. The effect of noise in the data is to produce spurious peaks in
170: the periodogram which arise, not because of any periodicity in the observed system,
171: but because of the way that the noisy signal has been sampled. These spurious peaks
172: can be surprisingly large. It is essential therefore to have reliable tests by which
173: to determine the significance of periodogram peaks.
174:
175: This topic has already received attention in the literature. Key classical papers
176: include those of \citet{deeming75}, \citet{lomb76}, \citet{scargle82}, and
177: \citet{hb86}. We discuss pertinent aspects of these papers in the sections that
178: follow. More recently, criticisms of these papers have appeared in the work of
179: \citet{koen90} and Schwarzenberg-Czerny (1998), amongst others. Much of the
180: criticism has revolved around the appropriate means for attaching significance to
181: peaks that arise in a calculated periodogram.
182:
183: Significance tests for periodograms are hugely important to the asteroseismologist
184: who relies on periodograms to deliver precise values for purported eigenfrequencies
185: of pulsation. Comparison of the values of observationally determined
186: eigenfrequencies with the values predicted by the latest theoretical models should,
187: in principle, allow the identification of modes actually excited in real stars, and,
188: subsequently, allow for asteroseismological analysis of those stars.
189:
190: Asteroseismology appears to be on the threshold of a golden age, as extensive
191: surveys like ASAS \citep{pojmanski98}, and space missions in the mold of COROT
192: \citep{baglin02}, hugely increase the number of known pulsating stars, as well as
193: the time coverage available for their analysis. It is expected that periodogram
194: analysis will continue to play a prominent role in asteroseismology. Hence, accurate
195: interpretation of periodogram peaks is an issue of prime importance.
196:
197: In this paper, we consider methods currently used for assessing the relevance of
198: periodogram peaks, and propose a practical method applicable to all time series,
199: irrespective of the spacing of the data. This method produces significance tests
200: that are tailor-made for any given astronomical time-series.
201:
202: The structure of this paper is as follows. We first discuss the construction of
203: significance tests in general, and of Scargle's significance test in particular. We
204: consider the concept of ``independent frequencies" in periodogram analysis. We
205: comment on aspects of the work reported by \citet{scargle82}, \citet{hb86}, and
206: \citet{sc98}. We report our attempts at reproducing the results of \citet{hb86} by
207: Monte Carlo simulation, and discuss our failure to reproduce their results in
208: detail. The conclusions forced on us by the discrepancies between our results and
209: theirs lead us to the main points made in this paper. They also lead us to propose a
210: pragmatic method, applicable to all time series, for assessing the significance of
211: periodogram peaks. En route, we also discuss the problem of over-sampling the
212: periodogram.
213:
214: Definitions of the periodogram assumed in our discussion are given in Appendices
215: \ref{classicalP} and \ref{scargleP} of this paper. Detailed discussions of the
216: phenomena of aliasing and spectral leakage, to which we refer in the text, may be
217: found in \citet{deeming75} and \cite{scargle82}.
218:
219:
220: \section{SIGNIFICANCE TESTS}
221:
222: Noisy data produce noisy periodograms. Peaks in a periodogram may therefore not be
223: due to the presence of any real periodic phenomenon at all. They may simply be
224: random fluctuations in periodogram power caused by the presence of a noise component
225: in the data. Peaks arising in this way are spurious: they are not due to any real
226: periodicity in the observed phenomena, but are simply artifacts of chance events in
227: the accompanying noise.
228:
229: Simulations show that noise in a time series can produce surprisingly large spurious
230: peaks in the associated periodogram. It is important therefore to develop reliable
231: tests for determining whether a given periodogram peak reflects a real periodicity
232: in the data, or is simply an artifact of the noise. In this section, we consider the
233: theoretical basis for a class of general, model-independent, tests. These determine
234: the probability that the periodogram powers observed in a data set might have arisen
235: from pure noise alone, with no other form of signal present. For a definition or
236: pure noise, see Appendix \ref{noiseApp}.
237:
238: Note that, in this paper, we do not use the word ``power" in the formal statistical
239: sense, where it means the probability of rejection of the null hypothesis given that
240: the null hypothesis is false, but in its accepted physical sense. Thus ``periodogram
241: power" at frequency $\omega$ means $P_X(\omega)$, as defined in Appendices
242: \ref{classicalP} and \ref{scargleP}.
243:
244: The basis for this general class of tests is the cumulative distribution function
245: (CDF),
246: \begin{eqnarray}
247: F_Z(z) = {\rm Pr} [ Z \leq z ]
248: \label{st-1}
249: \end{eqnarray}
250: where, the random variable $Z=P_X(\omega)$ is the periodogram power at frequency
251: $\omega$ for the time series $X$, and $z$ is some selected power threshold. The
252: function $F_Z(z)$ gives the probability that, when the data $X$ are pure noise,
253: their periodogram power at the given frequency $\omega$ does not rise above
254: power-level $z$.
255:
256: Suppose a model of the observed system predicts an oscillation at frequency
257: $\omega$. Then, we expect $P_X(\omega)$ to be large at this frequency. However, pure
258: noise by itself might equally well produce a large value of $P_X(\omega)$. The
259: CDF in equation (\ref{st-1}) provides an objective criterion by which to determine
260: whether the observed large value of $P_X(\omega)$ is due to the presence of a {\em
261: bona fide} signal, or is nothing more than a spurious large fluctuation due only to
262: the presence of noise. Suppose the data $X$ are pure noise. Then the probability
263: that the normalised periodogram power at this frequency is less than a specified
264: value $z_0$ is
265: \begin{eqnarray}
266: p_0 = F_Z(z_0)
267: \end{eqnarray}
268: Inverting this function,
269: \begin{eqnarray}
270: z_0 = {F_Z}^{-1}(p_0)
271: \end{eqnarray}
272: we obtain, for given $p_0$, the threshold power-level $z_0$ for which a power value
273: $Z\leq z_0$ has a probability $p_0$ of being due to pure noise alone. Equivalently,
274: a power value at frequency $\omega$ that exceeds $z_0$ has probability $1-p_0$ of
275: being due to pure noise alone. This test is both primitive and negative. It does
276: {\em not} tell us that $p_0$ is the probability that our signal contains a periodic
277: component of frequency $\omega$, but only that $p_0$ is the probability that our
278: signal is {\em not pure noise}.
279:
280: In practice, one does not evaluate the periodogram power at a single frequency only,
281: but at a selected set $\{ \omega_\mu : \mu = 1, 2, ..., N \}$ of frequencies. The
282: procedure normally followed when looking for periodicities in data is this: the
283: periodogram is evaluated at the selected frequencies $\omega_\mu$, and the
284: periodogram power $P_X(\omega_\mu)$ is plotted against $\omega_\mu$; this plot is
285: then scanned for its highest peaks. The conclusion one would like to draw from the
286: plot is that peaks that rise substantially above all others indicate the presence of
287: genuine periodicities in the observed system. However, before we can have confidence
288: in this conclusion, we need first to rule out the possibility that the observed
289: periodogram plot could have been produced by pure noise alone. This is done by
290: calculating the probability that {\em the entire observed periodogram profile} could
291: be produced by pure noise alone. Suppose $X$ is pure noise. Consider the probability
292: that {\em all} of the periodogram powers $\{P_X(\omega_\mu) : \mu = 1,2,...,N\}$ at
293: the sampled frequencies $\{\omega_\mu\}$ fall below a specified power threshold $z$.
294: Define a new random variable,
295: \begin{eqnarray}
296: Z_{\rm max} = {\rm sup}\ \{ P_X(\omega_\mu) : \mu=1,2,...,N\}
297: \end{eqnarray}
298: Thus $Z_{\rm max}$ is the maximum periodogram power among the set of $N$ {\em
299: sampled} powers. Now, the power at {\em each} of the sampled values will fall below
300: some specified threshold $z$ if and only if $Z_{\rm max}\leq z$. We thus need to
301: calculate the CDF
302: \begin{eqnarray}
303: F_{Z_{\rm max}} (z) = {\rm Pr} [ Z_{\rm max} \leq z ]
304: \end{eqnarray}
305: The function $F_{Z_{\rm max}}(z)$ gives the probability that, when the data $X$ are
306: pure noise, the periodogram power $P_X(\omega_\mu)$ does not rise above the
307: threshold $z$ at {\em any} of the sampled frequencies $\{ \omega_\mu \}$. We
308: construct the second significance test as follows. Let $z_0$ be a specified power
309: threshold. The probability that pure noise alone will produce periodogram powers
310: $P_X(\omega_\mu)$ that do {\em not} exceed the threshold $z_0$ at {\em any} of the
311: sampled frequencies $\{ \omega_\mu \}$ is given by
312: \begin{eqnarray}
313: p_0 = F_{Z_{\rm max}} (z_0)
314: \end{eqnarray}
315: Inverting this function,
316: \begin{eqnarray}
317: z_0 = {F_{Z_{\rm max}}}^{-1}(p_0)
318: \end{eqnarray}
319: For given $p_0$, this inverse function defines a threshold power-level $z_0$ such
320: that, if the periodogram power at each of the frequencies $\{\omega_\mu \}$ has
321: value $Z\leq z_0$, then the observed periodogram profile has probability $p_0$ of
322: being due to pure noise alone. This test reduces the probability of spurious
323: detections.
324:
325: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
326:
327: \section{SCARGLE'S SIGNIFICANCE TEST}
328:
329: If the data are Gaussian pure noise, the periodogram power $Z=P_X(\omega)$ at any
330: given frequency $\omega$ of the sampled signal $X_k$ is exponentially distributed
331: with probability density function defined by (Scargle 1982, p 848),
332: \begin{eqnarray}
333: \lefteqn{ p_Z (z)\ dz = {\rm Pr} [ z < Z < z+dz ] } \nonumber \\
334: && = \frac{1}{\sigma_X^2} \ e^{-z/ \sigma_X^2}\ dz
335: \end{eqnarray}
336: The cumulative distribution function is thus given by
337: \begin{eqnarray}
338: \lefteqn{ P_Z (z) = {\rm Pr} [ Z < z ] } \nonumber \\
339: && = \int_{\zeta=0}^z \ p_Z(\zeta)\ d\zeta
340: = 1- e^{-z/ \sigma_X^2}
341: \end{eqnarray}
342: We are interested in the probability that the periodogram power at the given
343: frequency is greater than a specified threshold $z$. This is given by
344: \begin{eqnarray}
345: {\rm Pr} [ Z > z ] = 1- P_Z (z) = e^{-z/ \sigma_X^2}
346: \end{eqnarray}
347: As the observed power $z$ becomes larger, it becomes exponentially less likely that
348: so high a power level (or higher) could be produced by pure noise alone, and
349: correspondingly more likely that the observed power level is due to a genuine
350: deterministic (i.e., non-noise) feature in the measured signal. Of course, this does
351: not mean necessarily that the suspected deterministic signal is {\em harmonic} with
352: frequency $\omega$, but simply that it is unlikely that this high power is due to
353: the noise component alone.
354:
355: It is worth noting that the argument of the exponential in the cumulative
356: distribution function is not simply the observed power $z$, but the ratio
357: $z/\sigma^2_X$, which is the ratio of the periodogram power to the total variance of
358: the data (called total input signal power by some). This is an important point,
359: worth emphasising, as did \citet{hb86}. If the incorrect power ratio is used, then
360: the statistical tests considered by Scargle will necessarily fail. Thus,
361: normalisation of the periodogram power by the number $N_0$ of data points used to
362: calculate the periodogram (classical normalisation), or by the residual power after
363: a sine curve has been removed from the data, or by the variance of the observational
364: uncertainty, all lead to completely different statistical distributions for the
365: periodogram power and invalidate Scargle's analysis summarised in this paper. Of
366: course, this does not make alternative normalisations ``wrong". It does mean however
367: that they must be accompanied by alternative statistical analyses \citep{sc98}.
368:
369: In practice, we do not evaluate the periodogram power at a single frequency alone,
370: but at a set of conveniently chosen frequencies $\{\omega_\mu : \mu = 1,2,...,N \}$.
371: We shall return to the question of how to choose these frequencies in a later
372: section. For the moment, suppose that we have the values of $P_X$ not at one value
373: of the frequency alone, but over a set of frequencies. This enables us to devise a
374: stronger test in which we determine the probability that the observed periodogram
375: power {\em over the entire set} of sampled frequencies have been produced by pure
376: noise alone.
377:
378: To develop this new, stronger statistical test, we need to assume with Scargle that
379: we have evaluated the periodogram power at a set $\{\omega_\mu : \mu = 1,2,...,N_i
380: \}$ of frequencies chosen in such a way that the random variables $\{ Z_\mu =
381: P_X(\omega_\mu) : \mu = 1,2,...,N_i \}$ are {\em mutually independent}. \citet{hb86}
382: refer to a set of frequencies chosen in this way as ``independent frequencies". This
383: is an abuse of terminology, since it is not the frequencies that are ``independent",
384: but the random variables $Z_\mu$. However, this lack of precision leads to no
385: ambiguity and so is tolerable.
386:
387: A large body of theorems is available for use if the random variables under
388: consideration are independent. Abandoning the condition of independence creates
389: serious complications in both the reasoning and the proofs of the results.
390:
391: Suppose we observe a periodogram power at one of the $\omega_\mu$, that is higher
392: than a given threshold $z$. We ask, what is the probability that pure noise alone
393: could have produced a periodogram power of this level or higher among all of the
394: sampled independent periodogram frequencies? First, we calculate the probability
395: that {\em all} the sampled periodogram powers are less than the threshold power $z$.
396: Define
397: \begin{eqnarray*}
398: Z_{\rm max}= {\rm sup }\ \{ Z_1, Z_2, ..., Z_{N_i} \}
399: \end{eqnarray*}
400: The probability that any given power $Z_\mu$ in this set falls below the threshold
401: is
402: \begin{eqnarray*}
403: {\rm Pr\ }[ Z_\mu < z ] = 1 - e^{-z/\sigma_X^2}
404: \end{eqnarray*}
405: Since the $Z_\mu$ are independent, the probability that they all fall below the
406: threshold $z$ is given by
407: \begin{eqnarray*}
408: \lefteqn{ {\rm Pr\ }[ Z_1 < z \mbox{ and } Z_2 < z \mbox{ and }...
409: \mbox{ and } Z_{N_i} < z
410: ] } \hspace*{1cm} \\
411: &=& {\rm Pr\ }[ Z_1 < z ] \ {\rm Pr\ }[ Z_2 < z ] \ ...\ {\rm Pr\ }[ Z_{N_i} < z
412: ] \\
413: &=& \left[ 1 - e^{- z/\sigma_X^2} \right]^{N_i}
414: \end{eqnarray*}
415: The probability that {\em not all} the powers $Z_\mu$ are less than the threshold
416: $z$, that is, the probability that {\em at least one} of the powers $Z_\mu$ is above
417: the threshold $z$, is then,
418: \begin{eqnarray}
419: {\rm Pr\ }[ Z_{\rm max} > z ] &=& 1 - \left[ 1 - e^{- z/\sigma_X^2} \right]^{N_i}
420: \label{scargle-14}
421: \end{eqnarray}
422: This is the function that Scargle proposes as a false alarm probability. The idea is
423: that we choose a probability, say $p_A$, that we regard as an acceptable level of
424: risk for the false detection of real deterministic signals. We solve the above
425: formula for $z$, to get a reference power threshold level $z_A$ given by
426: \begin{eqnarray}
427: z_A &=& - \sigma_X^2 \ln \left[ 1- \left( 1 - p_A \right)^{1/N_i} \right]
428: \end{eqnarray}
429: Then, if we claim a detection whenever the power level at one of the frequencies
430: $\{\omega_\mu : \mu = 1,2,...,N_i \}$ exceeds the reference level $z_A$, the
431: probability that we will be {\em wrong} is given by $p_A$. That is, on average we
432: will be wrong only $p_A$ of the time, since pure noise can produce fluctuations
433: above this level at these frequencies only $p_A$ of the time.
434:
435: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
436:
437: \section{INDEPENDENT FREQUENCIES}
438:
439: Scargle's test is constructed on the assumption that we can identify a set of
440: frequencies at which the periodogram powers are independent random variables. In the
441: case where the time-domain data are evenly spaced, we are guaranteed the existence
442: of such a set. These are called the {\em natural frequencies} \citep{scargle82}, or
443: the {\em standard frequencies} \citep{priestley81}. These are given by
444: \begin{eqnarray}
445: \omega_k &=& \frac{2\pi k}{T}
446: \end{eqnarray}
447: where $T$ is the total time span of the data set, that is, $T=t_{N_0}-t_1$, and $k=
448: 0,...,[N_0/2]$, where $[N_0/2]$ signifies the integer part of $N_0/2$. The
449: statistics of $P_X(\omega_k)$ with $k=0$ are different from those with $k\neq 0$
450: \citep{priestley81}. If we omit $P_X(\omega_0)$, this leaves us with at most
451: $[N_0/2]$ independent frequencies. In practice, the omission of $\omega_0$ from the
452: set of independent frequencies is of no consequence. This frequency corresponds to a
453: DC component in the signal which is generally removed from the data before their
454: periodogram is calculated. Thus, in the case of evenly spaced data, we can easily
455: construct the Scargle false alarm probability function and apply it to determine the
456: significance of high periodogram-power levels at these ``independent frequencies".
457:
458: It is worth emphasising that, since the false alarm probability function assumes
459: independent powers at the examined frequencies, {\em we can only use it to put a
460: significance level on the values of the periodogram-power at the chosen independent
461: frequencies}. Peaks found at other frequency values by over-sampling the periodogram
462: {\em cannot} be assessed in this way.
463:
464: In the unevenly sampled case, the situation changes dramatically. The statistical
465: analysis of the classical periodogram becomes intractable. The results are
466: sampling-grid dependent, and no general analysis applicable to all cases has yet
467: been produced. To simplify the statistical analysis, Scargle proposed that the
468: definition of the periodogram be modified. His modified periodogram had already been
469: used by \citet{barning63}, \citet{vanicek69}, and \citet{lomb76}. These authors did
470: not view the modified periodogram as an attempt to estimate the Fourier power
471: spectrum from unevenly sampled data, but as a spectral method for searching for the
472: best-fit harmonic function to their data. The novelty of Scargle's approach was that
473: he generated the same spectral method as used by these authors by imposing simple
474: constraints on a generalised form of the Fourier transform. The constraints were
475: that the modified periodogram should mimic as closely as possible the statistical
476: properties of the classical periodogram, and that the resulting spectral function
477: should be insensitive to time translations of the data in the time domain.
478:
479: The demand that the modified periodogram should mimic as closely as possible the
480: statistical properties of the classical periodogram was only partially successful.
481: Forcing time translation invariance, and demanding that the statistics of the random
482: variable $P_X(\omega)$ at a single selected frequency remain unchanged, that is,
483: demanding that $P_X(\omega)$ be exponentially distributed, exhausts the free
484: parameters in Scargle's modified FT, giving Lomb's spectral formula. In this way, he
485: reproduced some properties of the periodogram for the evenly sampled case. However,
486: this is the best that he could do. Most other familiar properties are lost. The most
487: important loss is the existence of independent frequencies.
488:
489: All relevant information about correlation and mutual dependence of the random
490: variables $\{ P_X(\omega)\}$ is contained in the window function, $G(\omega)$. (For
491: a discussion of the window function, see \citet{scargle82}, Appendix D, p 850, and
492: also his discussion on p 840.) Thus, the coefficient of linear correlation between
493: $P_X(\omega)$ and $P_X(\omega')$ is given by $G(\omega'-\omega)$ \citep{lomb76}. For
494: independence of $P_X(\omega)$ and $P_X(\omega')$, it is necessary (but not
495: sufficient) that $G(\omega'-\omega)=0$. Furthermore, for mutual independence of a
496: set $\{ P_X(\omega_k): k=1,2,...,r\}$ of periodogram powers, it would also be
497: necessary (but not sufficient) to have the $\omega_k$ evenly spaced. These are very
498: difficult conditions to realise in practice. \citet{koen90} has searched numerically
499: for such mutually uncorrelated sets in a variety of sampling schemes and failed to
500: turn up more than two simultaneously uncorrelated frequencies.
501:
502: For Scargle, this loss of independent frequencies is not debilitating. He says (p
503: 840, column 1) that ``... if the frequency grid is well chosen, the degree of
504: dependence between the powers at the different frequencies is usually small", and (p
505: 840, column 2) that, ``With a wide variety of sampling schemes $G(\omega)$ does have
506: nulls, or relatively small minima, that are approximately evenly spaced... Such
507: nulls comprise a set of natural frequencies at which to evaluate the periodogram. At
508: these frequencies the $P(\omega)$ form a set of approximately independent random
509: variables - thus closely simulating the situation with evenly spaced data". The
510: implication, though not explicitly stated by Scargle, is that in spite of the loss
511: of independence of the random variables $P(\omega)$ at the natural frequencies, the
512: false alarm probability given by our equation (\ref{scargle-14}) (equation (14) in
513: \citet{scargle82}, p 839), still provides a reliable significance test in the wide
514: variety of sampling schemes that he considered.
515:
516: It seems that Scargle's recommendation for the case of unevenly spaced data is as
517: follows: evaluate the modified periodogram at the natural frequencies defined by the
518: given data span, and use the false alarm probability calculated for the evenly
519: spaced case to evaluate the significance of the periodogram peaks. He further
520: recommends that, to improve the detection efficiency, we decrease the number of
521: frequencies inspected (p 842). The effect of this reduction is that we reduce power
522: threshold for a given significance level of peak-heights.
523:
524: The value of $N_i$ is a critical ingredient in Scargle's false alarm probability
525: function. There has been some debate concerning its correct value, as well as its
526: meaning. \citet{hb86} appear to have been unsatisfied with the value $N_i=[N_0/2]$
527: and proposed to determine $N_i$ by a method which we describe in the following
528: section.
529:
530:
531: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
532:
533: \section{HORNE AND BALIUNAS DETERMINATION OF $N_{\lowercase{i}}$}
534:
535: \citet{hb86}, (HB in the remainder of this paper), determined $N_i$ by the following
536: procedure. They simulated a large number of data sets, each consisting of
537: pseudo-Gaussian noise. The periodogram of each data set was evaluated from
538: $\omega=2\pi/T$ to $\omega=\pi N_0/T$, where $T$ is the total time interval. They
539: then chose the highest peak in each periodogram, combined these, and fitted the
540: Scargle false alarm probability function to the peak distribution using $N_i$ as the
541: variable parameter.
542:
543: The HB simulations investigated three major types of spacing in the time coordinate.
544: In the first, the data were evenly spaced in time. In the second, each time followed
545: the previous one by a random number between 0 and 1. In the third, the data were
546: clumped in groups of three at each evenly spaced time interval.
547:
548: In the case where the data are evenly spaced in time, theoretical statistical
549: analysis provides us with a very clear, unambiguous picture of what to expect from
550: the simulations: the random variables $\{P_X(\omega_k) : k=1,..., [N_0/2] \}$, where
551: $\omega_k = 2\pi k /T$ and $T$ is the total time interval covered by the data, are
552: mutually independent; the window function, which contains all relevant information
553: about dependencies and correlations of the random variables $P_X(\omega)$, shows
554: that these are the only frequencies at which the periodogram powers are independent
555: (Scargle p 840 and 843); the listed frequencies $\omega_k$ contain maximal
556: information about the power distribution of the sampled signal. This is seen from
557: the fact that the discrete Fourier transform evaluated at these frequencies contains
558: exactly enough information to reconstruct completely the original data. So, from
559: theory, we expect the total number $N_i$ of independent frequencies in the case of
560: evenly spaced time series consisting of zero mean pure noise to be exactly
561: $[N_0/2]$. In practice, a simulated time series, generated from a zero mean
562: distribution, will not have precisely zero mean. We must therefore remove its mean
563: before finding its periodogram. Once this is done, the theory guarantees that our
564: simulated data set will have exactly $[N_0/2]$ independent frequencies. The point is
565: this: for the evenly spaced data sets that we have simulated, the number of
566: independent frequencies in the periodogram is at most $[N_0/2]$. In real data, this
567: number may need to be further reduced if we estimate other parameters.
568:
569: Surprisingly, the best fits obtained by HB consistently produced values of $N_i$
570: which were substantially higher than this expected upper limit (HB, Table 1, p 759).
571: In fact, their fitted values are consistently higher than $N_0$, with the exception
572: of their two smallest data sets (10 and 15 points respectively) where the fitted
573: value of $N_i$ is slightly less than $N_0$, but still about twice as large as
574: expected.
575:
576: These results are puzzling. Theory and simulations appear to be in conflict.
577: \citet{cumming99} note that Baliunas has indicated typographical errors in the
578: values listed in HB. \citet{koen90} and \citet{sc96} have also noted mistakes in HB.
579: We have repeated the HB simulations for the case of even sampling in the time
580: domain. We have also extended somewhat the scope of their investigations to consider
581: the alternative false alarm probability function proposed by \citet{sc98}, as well
582: as the effects of over-sampling the periodogram. The results are interesting, and we
583: report them in the corresponding sections below.
584:
585: In our first set of simulations, we attempted to reproduce the results reported by
586: HB in their Table 1, p 759, for the case of evenly spaced data. HB describe the
587: method they followed in their simulations as follows: ``The periodogram of each data
588: set was evaluated from $\omega=2\pi/T$ to $\omega=\pi N_0/T$ ... The highest peak
589: was then chosen in each periodogram." It was not clear to us whether they sampled
590: the periodogram values $P_X(\omega)$ {\em only} at the natural frequencies $\omega_k
591: = 2\pi k /T$, and then chose the highest periodogram power from this restricted
592: sampled set, as prescribed by Scargle; or whether they followed the practice of a
593: not insubstantial number of astronomers who search for the highest periodogram peak
594: in the given range by grossly over-sampling the periodogram, and then choose the
595: maximum value obtained irrespective of whether it occurs at one of the natural
596: frequencies $\omega_k$. Accordingly, we ran two sets of simulations implementing
597: both procedures. We fitted the Scargle false alarm function to our results by the
598: method of least squares. All our periodograms were normalised using the sample
599: variance of the simulated data, and not the variance of the distribution used to
600: generate the sample. We failed to reproduce the HB results in detail. Sampling the
601: periodogram at the natural frequencies only and choosing the highest value among
602: these yielded values of $N_i$ that were consistently lower than those obtained by
603: HB. In fact, we obtained values very close to $[N_0/2]$, as expected theoretically,
604: but in conflict with the results published by Horne and Baliunas. Searching for the
605: highest peak by over-sampling also yielded values that were consistently lower than
606: HB, but higher than sampling at the natural frequencies. More precisely, our results
607: agree closely with those of HB for the smaller data sets up to $170$ data points.
608: This leads us to suspect that the HB table was constructed by gross over-sampling.
609: However, our results strongly deviate from theirs for the larger data sets with $N_0
610: > 170$, with our values being substantially lower. Plotting $N_i$ vs. $N_0$ (Figure
611: \ref{figure0}),
612: %%%%%%%%%%%%%%%%%%%%%%%%%%
613: \begin{figure}
614: %\includegraphics*[bb = 0 0 2000 180]{NiVsN0.bmp}
615: %\includegraphics[width=80mm]{f1.ps}\\
616: %\vspace*{-7.9cm}\\
617: \includegraphics*[width=84mm,height=55mm]{f1.eps}\\
618: \caption{
619: Plots of $N_i$ vs. $N_0$ of the data published by \citet{hb86} in their Table 1, p
620: 759, for the case of evenly spaced data, and of our simulations, fitting the Scargle
621: and Schwarzenberg-Czerny false alarm functions to the empirical CDF's obtained by
622: sampling at the natural frequencies and by over-sampling. Solid dots: published
623: Horne and Baliunas values; asterisks: Scargle function fitted to empirical CDF's
624: obtained by sampling at the natural frequencies; circled crosses:
625: Schwarzenberg-Czerny function fitted to the same; stars: Scargle function fitted to
626: empirical CDF's obtained by over-sampling; triangles: Schwarzenberg-Czerny function
627: fitted to the same. The solid line is the theoretically expected relationship
628: $N_i=[N_0/2]$.
629: }
630: \label{figure0}
631: \end{figure}
632: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
633: we observe the following features. The values yielded by our simulations increase
634: linearly with $N_0$, as expected. In contrast, the results published by HB in their
635: Table 1 appear to lie, not on a quadratic (as claimed by them), but on two straight
636: lines of different slope, a sharp change in slope appearing for data sets with $N_0
637: > 170$. This seems to be indicative of a systematic error. Fitting a quadratic function to
638: these data points, as was done by HB, may therefore be misleading and renders
639: suspect its use in estimating the parameter $N_i$.
640:
641: Note however that, in the case of over-sampling, both our results and those of HB
642: consistently yield values of $N_i$ that are higher than the theoretically expected
643: value of $[N_0/2]$. These values are thus apparently in conflict with the theory.
644: The interpretation of $N_i$ as the number of independent frequencies is therefore
645: questionable in this context. The HB method for determining $N_i$ is eminently
646: practical and reasonable, but {\em it only yields correct values when the
647: periodogram is sampled at the natural frequencies}. This means that, in the context
648: of over-sampling, we cannot assign to the parameter $N_i$ the meaning that it had in
649: its original derivation, namely the number of independent frequencies in the
650: associated periodogram. Rather, we must treat $N_i$ as nothing more than a floating
651: parameter in a one-parameter family of candidate CDF functions which we are
652: attempting to fit to our data.
653:
654: Another problem with the HB method should be noted. Inspection of a plot of the
655: best-fit Scargle false alarm probability function shows it to be a very
656: uncomfortable fit to the experimentally obtained cumulative distributions of
657: periodogram peak heights (see Figure \ref{figure1}). This is true both in the case
658: of sampling at the natural frequencies and of over-sampling. Its general trend is
659: good: it is flat near value 1 at low peak heights, drops rapidly over the peak
660: height mid-range, and levels off to zero for larger peak heights. However, its
661: detailed behaviour simply does not match that of the experimental curve. It drops
662: too quickly, and levels off too soon. This mismatch is most pronounced for small
663: data sets, and becomes progressively less noticeable as the data sets increase in
664: size. But it never vanishes completely. The conclusion forced on us by our
665: simulations is that {\em the Scargle false alarm probability function fails to
666: reproduce the detailed behaviour of the simulated data sets}. This is both good news
667: and bad news: good news because it shows that the Scargle function {\em
668: underestimates} the significance of periodogram peaks; and bad news because it
669: leaves us without an useable false alarm probability function.
670:
671: In summary, we cannot in general regard $N_i$ as anything more than a fitting
672: parameter. Furthermore, {\em the Scargle probability function incorrectly describes
673: the statistical behaviour of the periodogram in these simulations}. We discuss a
674: possible reason for its failure in a later section. For the moment, we simply note
675: that it manifestly fails to provide a convincing fit to the empirical CDF produced
676: by our simulations. It displays the correct general characteristics of a CDF but,
677: notoriously, all CDF's tend to look alike, so simply displaying correct general
678: features is not a point in its favour. Our conclusion therefore is that {\em the HB
679: method is not in general a way to assess the number of independent frequencies in a
680: periodogram. Rather, it is a method for estimating the best-fit parameter $N_i$ in
681: an ill-fitting class of candidate CDF functions}.
682:
683: This does {\em not} make the Scargle probability function or the HB method for
684: estimating $N_i$ worthless. In those cases (large data sets) where the Scargle
685: function gives a reasonable fit to the empirical data, the HB method provides a
686: value of $N_i$ that makes the Scargle function a good estimate of the correct false
687: alarm probability and provides a formula in closed form that can be used as a
688: significance test. However, note that this formula consistently underestimates the
689: significance of periodogram peaks, this underestimation becoming increasingly severe
690: as the data sets become smaller.
691:
692: %%%%%%%%%%%%%%%%%%%%%%%%%%
693: \begin{figure}
694: %\includegraphics*[bb = 0 0 240 140]{Fig1N0=10exp.bmp}
695: %\includegraphics*[bb = 0 0 260 180]{Fig1N0=50exp.bmp}
696: %\includegraphics*[bb = 0 0 260 180]{Fig1N0=75exp.bmp}
697: %\includegraphics[width=84mm]{f2a.ps}\\
698: %\vspace*{-7.7cm}\\
699: %\includegraphics[width=84mm]{f2b.ps}\\
700: %\vspace*{-7.7cm}\\
701: %\includegraphics[width=84mm]{f2c.ps}\\
702: %\vspace*{-7.7cm}\\
703: \includegraphics*[width=80mm,height=48mm]{f2a.eps}\\
704: \includegraphics*[width=80mm,height=48mm]{f2b.eps}\\
705: \includegraphics*[width=80mm,height=48mm]{f2c.eps}\\
706: \caption{
707: Empirical CDF's (heavy dotted line) constructed by over-sampling the periodogram,
708: with best-fitting Scargle false alarm probability function (solid line) for (a)
709: $N_0=10$, (b) $N_0=50$, and (c) $N_0=75$ data points. The fit improves with
710: increasing $N_0$. The corresponding best-fit values of $N_i$ are (a) $N_i=9.09$, (b)
711: $N_i=54.00$, and (c) $N_i=85.82$. The light dashed line shows the Scargle function
712: for $N_i=[N_0/2]$. In all cases the best-fit value of $N_i$ exceeds $[N_0/2]$.
713: }
714: \label{figure1}
715: \end{figure}
716: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
717:
718: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Table %%%%%%%%%
719: \begin{table*}
720: \small
721: \centering
722: \label{TabulatedValues}
723: \begin{minipage}{140mm}
724: \caption{Results of Monte Carlo simulations} \vspace*{4mm}
725: \begin{tabular}{@{}ccccccc@{}}
726: \hline
727: &&&\multicolumn{2}{c}{\bf Over-sampled}&\multicolumn{2}{c}{\bf Natural Frequencies} \\
728: $N_0$ & HB Value&Number &Scargle Function & SC Function &Scargle Function
729: & SC Function \\
730: &of $N_i$&of Tests & Best-fit $N_i$ &Best Fit $N_i$& Best Fit $N_i$
731: & Best Fit $N_i$ \\
732: \hline
733: 10.00&9.70&1395.00&9.09&26.80&5.00&8.70 \\
734: 15.00&14.45&347.00&14.09&32.80&7.90&13.60 \\
735: 25.00&27.38&213.00&24.91&47.80&12.80&20.00 \\
736: 35.00&38.40&214.00&35.64&60.40&18.50&26.50 \\
737: 50.00&54.45&369.00&54.00&84.00&25.70&34.40 \\
738: 64.00&71.76&512.00&70.45&102.80&33.00&42.40 \\
739: 75.00&86.05&153.00&85.82&121.90&39.30&50.10 \\
740: 100.00&119.58&296.00&113.91&152.10&51.60&62.80 \\
741: 128.00&152.53&913.00&149.09&191.80&65.50&77.70 \\
742: 170.00&218.33&218.00&210.09&261.40&89.30&103.00 \\
743: 256.00&369.97&224.00&306.36&361.20&128.40&143.00 \\
744: 300.00&455.95&107.00&361.45&420.20&148.40&163.30 \\
745: 400.00&618.69&106.00&477.18&540.10&204.30&221.60 \\
746: \hline
747: \end{tabular}
748: \tablecomments{ Comparison of Horne and Baliunas values of $N_i$ with the results of
749: our numerical simulations, fitting both Scargle and Schwarzenberg-Czerny (SC) false
750: alarm functions to CDF's constructed from over-sampled periodograms and from
751: periodograms sampled at the natural frequencies. The corresponding best-fit
752: functions are displayed in Figures \ref{figure1} and \ref{figure2}, together with
753: the corresponding functions constructed with the correct value of $N_i=[N_0/2]$.}
754: \end{minipage}
755: \end{table*}
756: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% End Table %%%%
757:
758: \normalsize
759: \section{THE SCHWARZENBERG-CZERNY FALSE ALARM FUNCTION}
760:
761: \citet{koen90} pointed out an important implicit assumption in Scargle's derivation
762: of his false alarm probability function. Scargle assumed that the variance
763: $\sigma_X^2$ of the data $X_k$ is known {\em a priori}. There are situations in
764: which this condition is true, but it is satisfied neither in the case of real
765: astronomical data nor in that of the HB simulations. In the simulations, pseudo-data
766: are generated using a preselected variance and mean (chosen to be zero), but the
767: variance and mean of the generated sample will differ in general from those used in
768: their generation. Thus both variance and mean need to be estimated from the data.
769:
770: This changes the statistical analysis significantly. \citet{sc98}, in a particularly
771: clear and thorough exposition of the issues involved, has shown that the CDF of
772: maximum peak heights appropriate to the Lomb-Scargle periodogram and calculated from
773: a finite sample of Gaussian pure noise, is the (regularised) incomplete beta
774: function
775: \begin{equation}
776: I_{1-z/[N_0/2]}([N_0/2],1)=
777: \left ( 1-\frac{ z }{[N_0/2]} \right )^{[N_0/2]}
778: \label{SC-1}
779: \end{equation}
780: To construct the corresponding false alarm probability function, we need to use this
781: distribution in place of the exponential distribution used above. If in our
782: periodogram we can identify a set of frequencies at which the periodogram powers are
783: mutually independent, then the probability that the power at at least one of these
784: frequencies rises above given threshold power $z$ is given by
785: \begin{equation}
786: {\rm Pr\ }[Z_{\rm max} > z ]
787: = 1- \left[ 1 - \left(1-\frac {z/\sigma^2_X} {[N_0/2]} \right )^{[N_0/2]} \right]^{N_i}
788: \label{SC-2}
789: \end{equation}
790: where $N_i$ is the number of mutually independent frequencies inspected, and $Z_{\rm
791: max} = {\rm sup\ }\{ Z_1, Z_2, ... , Z_{N_i} \}$ is the maximum power among the
792: mutually independent powers $Z_\mu$. In our discussion, we shall call equation
793: (\ref{SC-2}) the {\em Schwarzenberg-Czerny false alarm probability function}. In
794: passing, note that \citet{sc98} provides a number of alternative distributions and
795: test statistics appropriate to other methods of data analysis and is able thereby to
796: resolve extant disputes about the ``correct" normalisation procedure for
797: periodograms.
798:
799: In the limit $N_0 \rightarrow \infty$, the distribution in equation (\ref{SC-1})
800: becomes exponential and coincides with that used by Scargle. Accordingly, in the
801: same limit, the associated false alarm probability function in equation (\ref{SC-2})
802: reduces to the Scargle false alarm function. A $Q-Q$ plot of the
803: Schwarzenberg-Czerny vs. Scargle false alarm functions (see \citet{sc98}, Figure 1,
804: p 835) shows that, while the agreement between them is good for large $N_0$, they
805: differ substantially for small data sets, with Schwarzenberg-Czerny's false alarm
806: function yielding consistently smaller false alarm probabilities than Scargle's.
807: According to this analysis, therefore, for given $N_i$, the Scargle false alarm
808: function consistently {\em underestimates} the statistical significance of
809: periodogram peaks.
810:
811: One reason for the failure of the Scargle function to reproduce the behaviour of our
812: empirical CDF's may be its implicit assumption that the variance $\sigma_X^2$ is
813: known {\em a priori}. To correct this error, we replaced the Scargle function by
814: Schwarzenberg-Czerny's and repeated the HB simulations for equally spaced data.
815: Using their method for determining $N_i$, we fitted the Schwarzenberg-Czerny false
816: alarm function to our empirical CDF's. We found very good, but not perfect,
817: agreement between the best-fit theoretical curves and the corresponding empirical
818: ones, with the greatest deviations occurring for small data sets (See Figure
819: \ref{figure2}).
820: %%%%%%%%%%%%%%%%%%%%%%%%%%
821: \begin{figure}
822: %\includegraphics*[bb = 0 0 260 180]{Fig2N0=10beta.bmp}
823: %\includegraphics*[bb = 0 0 260 180]{Fig2N0=50beta.bmp}
824: %\includegraphics*[bb = 0 0 260 180]{Fig2N0=75beta.bmp}
825: %\includegraphics[width=84mm]{f3a.ps}\\
826: %\vspace*{-7.7cm}\\
827: %\includegraphics[width=84mm]{f3b.ps}\\
828: %\vspace*{-7.7cm}\\
829: %\includegraphics[width=84mm]{f3c.ps}\\
830: %\vspace*{-7.7cm}\\
831: \includegraphics*[width=80mm,height=48mm]{f3a.eps}\\
832: \includegraphics*[width=80mm,height=48mm]{f3b.eps}\\
833: \includegraphics*[width=80mm,height=48mm]{f3c.eps}\\
834: %% to include a figure, or
835: %\vspace{3.5cm}
836: %% to leave a blank space
837: \caption{Empirical CDF's (heavy dotted line) constructed by over-sampling the
838: periodogram, with best-fitting Schwarzenberg-Czerny false alarm probability function
839: for (a) $N_0=10$, (b) $N_0=50$, and (c) $N_0=75$ data points.
840: The fits are significantly better than the corresponding ones for Scargle's function.
841: However, for low $N_0$, Schwarzenberg-Czerny's distribution is still significantly
842: different from the empirical one and overestimates the significance of high peaks.
843: The light dashed line shows the corresponding Schwarzenberg-Czerny false alarm
844: function for $N_i=[N_0/2]$. In all cases, the best-fit value of $N_i$ again exceeds
845: $[N_0/2]$.}
846: \label{figure2}
847: \end{figure}
848: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
849: For these, the theoretical best-fit curves consistently yield values that are {\em
850: lower} than those of the empirical curves, thus {\em overestimating} the
851: significance of peaks. For the larger values of $N_0$, the deviations of the fitted
852: from the empirical curves may be understood in the context of order statistics.
853:
854: In spite of the excellent nature of these fits, there is nevertheless an interesting
855: feature in these results that is worth noting. For the CDF's of periodogram powers
856: sampled at the natural frequencies, {\em the best-fit values of $N_i$ are
857: consistently larger than the theoretically expected number of independent
858: frequencies, which is at most $[N_0/2]$} (see Table 1). Correspondingly, a plot of
859: the Schwarzenberg-Czerny function for the value $[N_0/2]$ of independent frequencies
860: yields a curve that deviates badly from the corresponding empirical CDF and which
861: leads to a severe {\em overestimation} of the significance of periodogram peaks. We
862: have no option but to conclude from these results that, like the Scargle false alarm
863: function, the Schwarzenberg-Czerny false alarm function, given by equation
864: (\ref{SC-2}), appears not to describe the CDF's of our simulations. Note also from
865: Table 1 that the best-fit values of $N_i$ for CDF's constructed from over-sampled
866: periodograms are higher than those for the CDF's obtained by sampling at the natural
867: frequencies. This is consistent with our previous results for the Scargle false
868: alarm function. The results of our simulations again appear to be at variance with
869: the theory. For evenly spaced data, the theory (which seems unassailable) predicts
870: unambiguously the existence of at most $[N_0/2]$ independent frequencies, with the
871: CDF for periodogram powers sampled at these frequencies given by the
872: Schwarzenberg-Czerny false alarm function. Our empirical CDF's differ substantially
873: from those predicted by this theory, with HB best-fits occurring at values of $N_i$
874: that are consistently higher than expected. These results force us to the following
875: conclusions. First, when using the HB method to estimate $N_i$, {\em we cannot
876: interpret the best-fit value of $N_i$ as the number of independent frequencies in
877: our periodogram. Rather, we must treat $N_i$ as a floating parameter in a
878: one-parameter family of candidate CDF functions}. This conclusion is consistent with
879: that stated in the previous section. Second, as candidate CDF functions, the
880: Schwarzenberg-Czerny false alarm function appears to be superior to Scargle's.
881:
882: \section{FALSE ALARM FUNCTIONS FOR UNEVENLY SPACED DATA}
883:
884: The principal difficulty encountered when searching for a false alarm function in
885: the case of unevenly spaced data is the loss of the so-called independent
886: frequencies. This loss is not apparent. It is real. The problem is not that they are
887: difficult to identify but nevertheless present. It is that they are not there at
888: all, except perhaps for a small set that can be counted on the fingers of one hand.
889: A significance test based on so small a number of independent frequencies is not
890: useful. It would require us to sample the periodogram at no more than a few
891: frequencies, making it highly likely that we would miss most of the significant
892: periodicities in our data because of sparse sampling.
893:
894: The loss of a sizeable set of mutually independent frequencies puts us into a
895: difficult, possibly intractable, position vis-a-vis the search for a theoretical
896: formula in closed form for false alarm probabilities. Were such a formula available,
897: it would certainly be a great boon. Realistically however, it seems unlikely that
898: such a formula could ever be found for the general case of arbitrarily spaced data.
899:
900: The only alternative to a theoretical false alarm function is an empirically
901: generated one. \citet{sc98} expresses a distinct lack of confidence in this
902: approach. He states, p 832, ``We consider the opinion that all statistical problems
903: related to the periodograms can be solved by Monte Carlo simulations to be
904: over-optimistic". His skepticism regarding simulations is due to the unreliability
905: of random number generators. He says, p 832, ``The simulations have problems of
906: their own, related chiefly to the untested effects of the discrete random number
907: generators and periodogram algorithms on the tails of the continuous distributions",
908: and again on p 833, ``The Monte Carlo simulations rely on rare events of low
909: probability, for which neither the accuracy of random number generators nor the
910: accuracy of periodogram algorithms is well tested".
911:
912: The strong sentiments expressed by Schwarzenberg-Czerny offer little cheer to
913: observers, whose principal need is a reliable method for assessing candidate
914: periodogram peaks. The current generation of theoretical distributions are all based
915: on the assumption of independent frequencies and all require a value of $N_i$. In
916: the case of evenly spaced data, it might be argued that the correct value for $N_i$
917: is $[N_0/2]$, suitably reduced by the number of parameters already estimated from
918: the data. For the general case however, even were we to believe the conjecture that
919: independent frequencies exist, there appears to be no clear {\em a priori}
920: theoretical criterion for choosing the value of $N_i$, and the only practical method
921: offered is that of HB in which we fit some chosen theoretical distribution to the
922: simulated CDF's. Necessity therefore forces us, against Schwarzenberg-Czerny's
923: advice, into the route of Monte Carlo simulations.
924:
925: The realisation that the conjecture of the existence of independent frequencies is
926: false forces us to re-evaluate both the role of Monte Carlo simulations and the use
927: of theoretical false alarm probability functions. Schwarzenberg-Czerny's opinion
928: regarding random number generators is not unwarranted. However, the performance of
929: random number generators is continually being improved. There is every reason to
930: believe therefore that existing problems with random number generators will
931: eventually be resolved. In contrast, the problem of the lack of independent
932: frequencies is permanent. The Monte Carlo simulation option is therefore not as
933: bleak as may first appear. As regards the use of theoretical false alarm probability
934: functions, we do not really need them. The empirically generated CDF's contain all
935: the information that we need, whether or not we have a closed-form formula for them,
936: and can be used to determine significance thresholds. A closed-form formula would be
937: useful insofar as it facilitates calculation of the thresholds, but is not
938: essential. If one is needed, we can resort to fitting the empirical CDF as closely
939: as possible by {\em any} suitable form of trial function. In fact, we do not need
940: even to fit the entire CDF. We are interested only in the high-peak tail above a
941: certain minimum confidence threshold and so need only obtain a good fit in that
942: region. Should formulae be needed for other regions, we can resort to multiple fits
943: that together cover the entire CDF.
944:
945: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
946:
947: \section{THE PROBLEM OF OVER-SAMPLING}
948:
949: Theoretical false alarm probability functions are based on the assumption of the
950: existence of independent frequencies and contain the number $N$ of frequencies
951: inspected as a parameter. When the periodogram is inspected at the maximum number
952: $N_i$ of independent frequencies, $N=N_i$. For example, Scargle's function is given
953: by
954: \begin{eqnarray}
955: {\rm Pr} [ Z > z ] = 1 - F_{Z_{\rm max}} (z) = 1 - \left( 1- e^{-z} \right)^{N}
956: \end{eqnarray}
957: when $N$ is the number of independent frequencies {\em inspected}.
958:
959: %%%%%%%%%%%%%%%%%%%%%%%%%% Figure 3 %%%%%%%%%%%%%%%%%%%%%%%%
960: \begin{figure}
961: %\includegraphics*[bb = 0 0 260 180]{Fig3ScargleCDFs.bmp}
962: %\includegraphics[width=84mm]{f4.ps}\\
963: %\vspace*{-7.7cm}\\
964: \includegraphics*[width=84mm,height=55mm]{f4.eps}\\
965: \caption{
966: Scargle false alarm probability function as a function of $N$ for values $N= 10, 50,
967: 100, 200$. As $N$ increases, the probability of finding a peak above any given
968: threshold value increases. This illustrates Scargle's `statistical penalty': if many
969: independent frequencies are inspected for a spectral peak, we should expect to find
970: a large peak even when no signal is present. As $N$ increases, the CDF moves
971: progressively to larger peak-height values without limit.}
972: \label{figure3}
973: \end{figure}
974: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
975:
976:
977:
978: From Figure \ref{figure3}, it is seen that, for given $z$, this probability
979: increases as the number $N$ of sampled independent frequencies is increased.
980: Scargle, p 839, describes this property as the {\em statistical penalty} that we
981: must pay for inspecting a large number of frequencies. He explains this by saying
982: that ``if $N$ independent experiments are carried out, even if each one has a very
983: small probability of succeeding, the chance of one of them succeeding is very large
984: if $N$ is large enough (approaching certainty as $N$ approaches infinity)." He also
985: notes that the expected value of the maximum power $Z_{\rm max}$ of a white noise
986: spectrum over a set of $N$ frequencies at which the power is independent is given by
987: \begin{eqnarray}
988: \langle Z_{\rm max} \rangle = \sum_{k=1}^N \ \frac{1}{k}
989: \end{eqnarray}
990: which diverges logarithmically with $N$.
991:
992: These comments appear alarming. At face value, they seem to suggest that prodigious
993: sampling of the periodogram at the independent frequencies might lead eventually to
994: the dismissal of {\em all} periodogram peaks as spurious. They also appear strongly
995: to discourage over-sampling of the periodogram in an attempt to pin down more
996: precisely the frequency of a periodicity. Indeed, their effect has been so strong on
997: some that they refuse to evaluate periodogram power at any frequencies other than a
998: selected subset of the ``natural frequencies". Were these extreme conclusions drawn
999: from Scargle's comments correct, the periodogram method for searching for
1000: periodicities would be severely compromised.
1001:
1002: To understand Scargle's comments correctly, we need first to note that the false
1003: alarm function is deduced assuming that we are able to identify $N$ independent
1004: frequencies $\omega_k$. An evenly sampled time series consisting of $N_0$ data
1005: points guarantees the existence of {\em at most} $N=[N_0/2]$ mutually independent
1006: frequencies, namely the ``natural" ones. There can be no more. The original data set
1007: can be fully recovered from the DFT at these frequencies. So the information
1008: contained in periodogram powers at all other frequencies cannot be independent of
1009: these. For a given evenly sampled time series consisting of $N_0$ points, there is a
1010: maximum value of $N$ at which we can sample the periodogram independently, namely
1011: $[N_0/2]$, and hence a maximum value of $\langle Z_{\rm max} \rangle$. In practice
1012: therefore, there is no logarithmic divergence to fear.
1013:
1014: Second, if we over-sample the periodogram, the CDF is no longer correct. The powers
1015: at the sampled frequencies are no longer independent, and so equation
1016: (\ref{scargle-14}) ceases to be the correct description of the distribution. In
1017: these circumstances, it is not useful to look for a theoretical formula for the CDF.
1018: Even if it were mathematically tractable, it probably would not be worth the effort
1019: of obtaining an expression in closed form for it. The difficulties in obtaining a
1020: formula for this CDF, however, do not prevent us from obtaining an excellent
1021: approximation to it through numerical experiments. The results of our simulations in
1022: this respect are encouraging. Successive over-samplings produce progressively less
1023: effect on the CDF, until {\em it eventually converges to a limiting CDF beyond which
1024: no further refinement of the sampling grid changes the result.} (See Figure
1025: \ref{figure4}.)
1026: %%%%%%%%%%%%%%%%%%%%%%%%%% Figure 4 %%%%%%%%%%%%%%
1027: \begin{figure}
1028: %\includegraphics*[bb = 0 0 260 180]{Fig4EmpiricalCDFConv.bmp}
1029: %\includegraphics[width=84mm]{f5.ps}\\
1030: %\vspace*{-7.7cm}\\
1031: \includegraphics*[width=84mm,height=55mm]{f5.eps}\\
1032: \caption{Empirical CDF's as a function of over-sampling. The figure shows the CDF's
1033: corresponding to sampling at 1, 2, 3, 4, 5, and 10 times the Scargle sampling rate.
1034: The corresponding CDF's converge rapidly to a limiting CDF. The limiting CDF
1035: coincides almost perfectly with the CDF for an over-sampling factor of 10.}
1036: \label{figure4}
1037: \end{figure}
1038: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1039: With hindsight, we should have expected this. The original time domain data contain
1040: a finite amount of information. There is therefore a limit to how much information
1041: they can be forced to yield.
1042:
1043: Based on the numerical experiments described in this paper, we would therefore like
1044: to refine Scargle's lesson, drawn from a consideration of statistical penalties. The
1045: (gloomy) lesson he drew was: ``If many frequencies are inspected for a spectral
1046: peak, expect to find a large peak power even if no signal is present" \citep[p 840,
1047: column 1]{scargle82}. Our revision of Scargle's lesson is this: {\em If many
1048: frequencies are inspected for a spectral peak, expect to find a large peak power
1049: even if no signal is present - but the total number of independent frequencies
1050: present in any given time series is limited, so don't expect the number of large
1051: peaks produced by white noise to increase without limit. More importantly, {\bf \em
1052: over-sampling the periodogram does not dramatically increase the number of large
1053: peaks expected}}.
1054:
1055: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1056:
1057:
1058: \section{A PRACTICAL METHOD FOR DETERMINING FALSE ALARM PROBABILITIES}
1059:
1060: The theoretical false alarm probability functions extant in the literature all rely
1061: for their validity on the existence of independent frequencies. Such a set is
1062: guaranteed for evenly spaced data, but not for data that are unevenly spaced. Even
1063: in the case where the data are evenly spaced, we may wish to inspect the periodogram
1064: at frequencies that do not coincide with Scargle's natural ones. Such is the case
1065: when a pronounced peak occurs at an intermediate frequency. How do we assess the
1066: significance of periodogram peaks in these cases?
1067:
1068: Based on our investigations described above, we suggest the following method:
1069: \begin{enumerate}
1070: \item
1071: {\em Using the sampling times of the actual data set to be analysed}, construct a
1072: large number of pseudo-Gaussian random time series.
1073:
1074: \item
1075: Select a convenient grid of frequencies that cover the frequency range in the
1076: periodogram that is to be inspected. (We discuss how to choose these frequencies in
1077: the next paragraph. For the moment, assume that they have been selected.)
1078:
1079: \item
1080: Construct the periodogram for each pseudo-random time series, sampling it at
1081: each of the selected frequencies.
1082:
1083: \item
1084: In each periodogram, identify the highest periodogram power that occurs {\em at the
1085: pre-selected frequencies only}, and use these highest values to construct the CDF of
1086: these highest power values.
1087: \end{enumerate}
1088: The CDF thus obtained is an empirically generated graphical representation of the
1089: probability function ${\rm Pr}[Z_{\rm max}\leq z]$. It gives the probability that
1090: pure noise alone could have produced power values less than or equal to a given
1091: threshold
1092: value $z$ at each of the selected sampling frequencies.\\
1093:
1094: \begin{minipage}[h]{7.4cm}
1095: % \begin{quote}
1096: {\normalsize \bf The plot of $1-{\rm Pr}[Z_{\rm max}\leq z]$ is thus the required
1097: false alarm probability function. It gives the probability that pure noise alone
1098: could produce a peak {\em at the inspected frequencies} of value higher than the
1099: threshold $z$. \\}
1100: %\end{quote}
1101: \end{minipage}
1102:
1103: How do we choose the frequencies at which to sample the periodogram? In a sense, it
1104: makes little difference how we choose them since, once chosen, we generate an
1105: empirical false alarm probability function that is tailor-made for our particular
1106: choice. However, for each choice, there is a price to be paid, and the final
1107: decision on how to choose the sampling frequencies is determined by what we consider
1108: to be the best compromise between the price paid and the advantage gained. For a
1109: given false alarm probability $p_A$, the denser the sampling, the higher the
1110: associated threshold $z$, with the heaviest penalty being paid for over-sampling
1111: sufficiently dense as to produce a fully resolved periodogram curve. In our
1112: simulations, this occurred at approximately five times the Scargle sampling rate,
1113: that is, using $\Delta \omega = \frac{1}{5}(\pi N_0/T)$. (See Figure \ref{figure5}.)
1114: %%%%%%%%%%%%%%%%%%%%%%%%%% Figure 4 %%%%%%%%%%%%%%
1115: \begin{figure}
1116: %\includegraphics*[bb = 0 0 260 180]{Fig5EmpericalCDFxlog.bmp}
1117: \includegraphics*[width=84mm,height=48mm]{f6.eps}\\
1118: %\vspace*{-7.7cm}\\
1119: \caption{
1120: Logarithmic plot of the sum of square deviations (from the limiting CDF) of the CDF
1121: for $\nu$ times over-sampling vs. the over-sampling factor $\nu$. The convergence to
1122: the limiting curve is seen to be very rapid. For the data set used in this
1123: simulation, the convergence occurs approximately at an over-sampling factor of 5.
1124: The convergence shows up in this plot as a sharp levelling off of the graph.}
1125: \label{figure5}
1126: \end{figure}
1127: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1128: %%%%%%%%%%%%%%%%%%%%%%%%%% Figure 4 %%%%%%%%%%%%%%
1129: \begin{figure}
1130: %\includegraphics*[bb = 0 0 260 180]{Fig4aEmpericalCDFloglog.bmp}
1131: \includegraphics*[width=84mm,height=48mm]{f7.eps}\\
1132: %\vspace*{-7.7cm}\\
1133: \caption{
1134: Log-log plot of the sum of square deviations (from the limiting CDF) of the CDF for
1135: $\nu$ times over-sampling vs. the over-sampling rate $\nu$. }
1136: \label{figure5a}
1137: \end{figure}
1138: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1139: The sampling rate sufficient to guarantee convergence to the limiting CDF must be
1140: established individually for each data set. This can be done using plots like those
1141: shown in Figures \ref{figure5} and \ref{figure5a}.
1142:
1143: If we are interested in pinpointing precisely the frequency of a peak (as we are in
1144: asteroseismology), then gross over-sampling may be the route to follow. However,
1145: there is a limit to the amount of information contained in the periodogram of a
1146: finite time series. There is therefore also a limit to how finely the frequency axis
1147: should be subdivided. This limit is given by $\Delta \omega_{\rm min} = \pi/T$,
1148: which is the smallest frequency interval that can reasonably be resolved by the data
1149: set. Dense over-sampling in pursuit of the convergence limit of the CDF may lead to
1150: a choice of $\Delta \omega$ smaller than this interval. If the limiting CDF differs
1151: substantially from that obtained from $\Delta \omega_{\rm min}$, then limiting
1152: the sampling interval to $\Delta \omega_{\rm min}$ may be a better option.
1153:
1154: As noted previously, \citet{sc98} has little confidence in this method. Apart from
1155: the comments already reported, he further says, p 832, ``Experiments often
1156: demonstrate difficulties in the reproduction of theoretical single-trial
1157: distributions by simulations. Hence the analytical single-trial probabilities
1158: discussed here are essential for the verification of Monte Carlo simulations". He
1159: has a similar comment on p 833 from which he draws the conclusion that
1160: ``single-trial analytical probability distributions are indispensable in any
1161: strategy for bandwidth correction". It is not clear how Schwarzenberg-Czerny intends
1162: the phrase ``single-trial probabilities" to be understood, but however we interpret
1163: it, these comments leave us with the same dilemma. It seems to us that our only
1164: recourse in assessing periodogram peak significance in the general case of unevenly
1165: spaced data is Monte Carlo simulation. Though the problems with Monte Carlo
1166: simulations pointed out by Schwarzenberg-Czerny are real, they are not problems of
1167: principle, but of practical implementation. They therefore can, and will, be
1168: overcome in time, if they have not been overcome already.
1169:
1170: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1171:
1172:
1173:
1174: \section{SUMMARY AND CONCLUSIONS}
1175:
1176: Currently available theoretical false alarm probability functions are all derived
1177: from what appear to be reasonable assumptions about the data to be analysed and
1178: about the periodograms that they yield. Their validity, reliability and usefulness
1179: therefore strongly depend on how well these assumptions are met in practice.
1180:
1181: A key assumption made by all authors is that the frequency range inspected in the
1182: periodogram contains a set of $N_i$ frequencies ${\omega_k : k = 1, ..., N_i}$ at
1183: which the periodogram powers ${P_X(\omega_k)}$ are mutually independent. In
1184: theoretical statistical analysis, we have little hope of obtaining a false alarm
1185: probability function in the absence of this assumption. Without independence, very
1186: few general statistical results are available, and none are relevant to the problem
1187: at hand. The assumption of the existence of independent frequencies is therefore
1188: necessary in any theoretical discussion of the problem of significance of
1189: periodogram peaks and poses the first and most important obstruction to its
1190: resolution.
1191:
1192: The existence of independent frequencies is guaranteed when the data are evenly
1193: spaced. We should therefore be able to test the validity of proposed false alarm
1194: probability functions for this case against the results of Monte Carlo simulations.
1195: Reasonable requirements on candidate functions include a good fit to the empirical
1196: CDF's, and their ability to predict correctly the number of independent frequencies
1197: known to exist from the theory.
1198:
1199: Though they seem not to have viewed their work in this light, HB effectively
1200: performed this test for Scargle's false alarm function. They constructed the
1201: empirical CDF for periodogram peak heights produced by a pure noise time series
1202: consisting of $N_0$ evenly spaced data points, and fitted the Scargle false alarm
1203: function to it by least squares using the number $N_i$ of independent frequencies as
1204: the fitting parameter. According to the theory, they should have obtained $N_i
1205: \lesssim [N_0/2]$. However, their results consistently yielded $N_i > N_0$. Horne
1206: and Baliunas did not comment on this anomaly.
1207:
1208: We have repeated their simulations, obtaining results similar to theirs only for
1209: gross over-sampling of the periodogram, and only for data sets with $N_0\leq 170$
1210: data points. For gross over-sampling and data sets with $N_0 > 170$, we were unable
1211: to reproduce their results. The values of $N_i$ obtained by HB are consistently and
1212: systematically larger than ours. In our simulations, the best-fit value of $N_i$
1213: increases linearly with $N_0$, in conflict with the quadratic dependence claimed by
1214: HB. Inspection of a plot of the values published in HB appears to indicate that
1215: their points lie on two straight lines, with a disjunction of slope at $N_0 = 170$
1216: data points. We conjecture from these results that HB constructed their empirical
1217: CDF's by gross over-sampling of the periodogram. This might explain why they
1218: consistently obtained $N_i > N_0$. We also conjecture that the sharp disjunction in
1219: slope at $N_0 = 170$, which is not observed in our simulations, is due to a
1220: systematic error in theirs. If so, the quadratic dependence of $N_i$ on $N_0$,
1221: sometimes exploited by astronomers in the analysis of their data, might not be
1222: a real feature of real astronomical data but rather a spurious artifact of the HB
1223: simulations.
1224:
1225: Given the assumption of independence that lies at the heart of Scargle's derivation
1226: of his false alarm probability function, it seemed unreasonable to suppose that it
1227: would provide an adequate description of the empirical CDF's obtained by
1228: over-sampling the periodograms. Accordingly, we initially ran the HB simulations for
1229: CDF's constructed by sampling the periodograms only at the natural frequencies. The
1230: best-fit values of $N_i$ were very close to the theoretically expected value of
1231: $[N_0/2]$. Though heartening, the results of these simulations displayed a
1232: disconcerting feature: the best-fit Scargle functions were very poor fits to the
1233: empirical CDF's, displaying large deviations from the empirical CDF's in the domain
1234: of most interest when assessing the significance of periodogram peaks. The
1235: theoretical false alarm functions were consistently substantially higher in value
1236: than the empirical CDF's, leading to severe under-estimation of peak significance.
1237: This same behaviour was observed for the best-fit curves to the CDF's constructed by
1238: over-sampling the periodograms. Researchers using the Scargle false alarm function,
1239: with or without the HB algorithm, are thus at significant risk of rejecting peaks
1240: that reflect real periodicities in their data.
1241:
1242: A flaw in Scargle's derivation of his false alarm function was pointed out by
1243: \citet{koen90} and by \citet{sc96}): Scargle assumes that the variance $\sigma_X^2$
1244: of the noise is known {\em a priori}. This condition is not satisfied either in the
1245: simulations (where we {\em sample} pure pseudo-Gaussian noise), nor in real data
1246: sets (where the data variance must be estimated from the data themselves). Both Koen
1247: and Schwarzenberg-Czerny correct this error in their respective treatments of the
1248: problem. \citet{koen90} concludes that Scargle's false alarm function should be
1249: replaced by the Fisher (or, Fisher-Snedecor) distribution. \citet{sc98} pointed out
1250: that the Fisher distribution is applicable only for ratios of {\em independent}
1251: random variables. With ratios of random variables that are not independent, the
1252: Fisher distribution must be replaced by the incomplete $\beta$-function. He also
1253: showed that, in the case of the Lomb-Scargle periodogram, the correct distribution
1254: is given by the incomplete beta function. On the strength of the work of these
1255: authors, we tested Schwarzenberg-Czerny's proposed function on CDF's constructed by
1256: over-sampling periodograms and also on CDF's constructed by sampling only at the
1257: natural frequencies. In both cases, we have found the best-fit Schwarzenberg-Czerny
1258: function, obtained by the HB algorithm, consistently to fit the empirical CDF's far
1259: more closely than Scargle's function, with impressively good agreement on all but
1260: the smallest data sets, where the theoretical function deviates only slightly from
1261: the empirical CDF's.
1262:
1263: In spite of the excellent fits provided by the Schwarzenberg-Czerny false alarm
1264: function, our simulations display an alarming feature: the best-fit values of $N_i$
1265: that yield such excellent agreement with the empirical CDF's are all consistently
1266: higher than the theoretically expected value of $[N_0/2]$. This is not unexpected
1267: for CDF's constructed by over-sampling. In the case of CDF's constructed by sampling
1268: only at the natural frequencies, however, this result is in conflict with the
1269: theory. This means that, as in the case of the Scargle function, we cannot interpret
1270: the best-fit value of the parameter $N_i$ as the number of independent frequencies.
1271: It must be regarded rather as a fitting parameter in a one-parameter family of
1272: candidate CDF functions that fit the empirical CDF's better than Scargle's candidate
1273: functions. Note that the Schwarzenberg-Czerny false alarm functions constructed
1274: independently of simulations, relying exclusively on the use of {\em a priori}
1275: theoretical values for $N_i$ {\em badly overestimate the significance of periodogram
1276: peaks and may result in the acceptance of spurious peaks as genuine}. It would seem
1277: therefore that unqualified confidence in analytical single-trial probability
1278: distributions in the construction of false alarm probability functions may be
1279: misplaced. Even in those cases where they ought to provide a good description of the
1280: behaviour of the empirical CDF's, they apparently fail to do so, leaving us no
1281: option but to resort to Monte Carlo simulations and to treat the theoretical
1282: distributions as nothing more than candidate CDF functions to be accepted or
1283: rejected according to their utility in providing a good fit to the empirical curves
1284: in the region of interest.
1285:
1286: Ultimately, our principal interest is in the case of unevenly spaced data, not data
1287: that are evenly spaced. The loss of independence of the variables $P_X(\omega)$ in
1288: this case calls into question the validity and the expediency of searching for a
1289: formula in closed form for a false alarm probability function. All formulae proposed
1290: hitherto are based on the assumption of the existence of a set of mutually
1291: independent periodogram powers. This assumption is not realistic in uneven sampling
1292: schemes, as shown by \citet{koen90}. Were a set of approximately uncorrelated
1293: periodogram powers to be found, in the sense outlined by Scargle, this still would
1294: not guarantee their approximate independence. The currently proposed closed-form
1295: formulae therefore cannot be expected to provide accurate false alarm criteria. Even
1296: if we adopt the attitude that the proposed formulae are no more than candidate CDF
1297: functions, the value of the parameter $N_i$ is not known {\em a priori},
1298: independently of Monte Carlo simulations. Therefore, {\em theoretical probability
1299: distributions provide no predictive power in determining false alarm criteria
1300: appropriate to a given data set which is independent of the empirical CDF's
1301: generated by simulations}. In the final analysis, the only way to obtain the
1302: appropriate false alarm probability function is by first constructing empirical
1303: CDF's for the maximum peak heights by using Monte Carlo methods, and then fitting
1304: these distributions with the false alarm function of choice. If a sufficiently good
1305: fit is obtained, the fitted function can then be used to calculate the significance
1306: levels for the given data set. If the fit is not good however, the significance
1307: levels predicted by these fitted functions are likely to lead to erroneous rejection
1308: or acceptance of periodogram peaks, making them almost useless in the assessment of
1309: the significance of peaks.
1310:
1311: At first, this dilemma appears irresolvable. On reflection, however, its resolution
1312: is staring us in the face. What we need is a reliable false alarm probability
1313: function. Though we do not possess this function as a closed-form formula, we
1314: nevertheless have a numerical plot of it in the form of the CDF of maximum peak
1315: heights. This plot can be used just as easily as any closed form formula to get the
1316: answers that we want. If we insist on having a closed-form formula to facilitate
1317: significance estimation, the empirical CDF can be fitted {\em in the region of
1318: interest} by any number of candidate fitting-functions. This renders the need to
1319: search for a theoretical formula obsolete. Of course, it would be nicer, more
1320: convenient and more satisfying to have a theoretical formula, but a numerical plot
1321: of the same function is almost as good.
1322:
1323: In this paper we have studied almost exclusively significance tests for the
1324: rejection of spurious peaks in periodograms of data sets that are evenly-spaced in
1325: time. An analogous study for unevenly-spaced data sets is currently in preparation.
1326:
1327: \acknowledgments
1328:
1329: We sincerely thank Mike Gaylard, Chris Koen and Melvyn Varughese for their critical
1330: reading of draft versions of the manuscript. We also thank the South African SKA
1331: Office in Johannesburg for use of their facilities.
1332:
1333: \appendix
1334:
1335:
1336: \section{CLASSICAL PERIODOGRAM}
1337: \label{classicalP}
1338:
1339: The {\em classical periodogram} is essentially a Fourier power spectrum estimator
1340: for an infinite continuous-time signal $X(t)$ that has been discretely sampled for a
1341: finite time at equally spaced time intervals. The data for this estimator form a
1342: finite discrete time series consisting of $N$ values $X_i=X(t_i)$, $i=1,..., N$, of
1343: the physical parameter $X$ at times $t_i= t_0, t_0+\Delta t, t_0+2\Delta t,..., t_0+
1344: (N-1)\Delta t$. The discrete Fourier transform (DFT), $DFT_X (\omega)$, of this time
1345: series $X_i$, which is defined by
1346: \begin{eqnarray}
1347: DFT_X (\omega) = \sum_{r=1}^{N} \ X(t_r)\, e^{-i\omega t_r} \label{cp-1}
1348: \end{eqnarray}
1349: may be regarded as an estimator of the Fourier transform $FT_X(t)$ of $X(t)$. The
1350: power spectral density of the signal may then be estimated by the function
1351: \begin{eqnarray*}
1352: \left| DFT_X (\omega) \right|^2 %\label{cp-2}
1353: \end{eqnarray*}
1354: with some suitably chosen normalising coefficient. A commonly used normalisation is
1355: \begin{eqnarray}
1356: CP_X (\omega) &=& \frac{1}{N} \, \left| DFT_X (\omega) \right|^2
1357: = \frac{1}{N} \, \left| \sum_{r=1}^{N} \ X(t_r)\, e^{-i\omega t_r} \right|^2
1358: \label{cp-2}
1359: \end{eqnarray}
1360: A simple calculation then yields the formula,
1361: \begin{eqnarray}
1362: CP_X (\omega) = \frac{1}{N} \, \left[ \left( \sum_{r=i}^N\ X(t_r) \cos \omega t_r \right)^2
1363: + \left( \sum_{r=i}^N\ X(t_r) \sin \omega t_r \right)^2 \right] \label{cp-3}
1364: \end{eqnarray}
1365: Following \citet{scargle82}, we call this function the {\em classical periodogram}.
1366: This definition agrees with that given originally by Schuster in \citet{schuster98},
1367: but not with that in his later publications. It also agrees with the definitions
1368: used in \citet{thompson71} and \citet{deeming75}, and differs by a factor of two
1369: from that used by \citet{priestley81}.
1370:
1371: It is easy to see from equation (\ref{cp-2}) why the classical periodogram is useful
1372: in identifying the frequencies of harmonic components in the signal $X$. Suppose $X$
1373: contains an harmonic component of frequency $\tilde{\omega}$. Then, when $\omega$ is
1374: very different from $\tilde{\omega}$, $X(t)$ and $e^{-i\omega t}$ are out of phase,
1375: and the product $X(t) e^{-i\omega t}$ oscillates rapidly. The sum of the products
1376: $X(t_r) e^{-i\omega t_r}$, which is a discrete estimator of the integral $\int
1377: \,X(t) e^{-i\omega t}dt$ will thus have a value close to zero, albeit masked by
1378: whatever other signal is present in $X(t)$. As $\omega$ approaches the value of
1379: $\tilde{\omega}$, the factors $X(t)$ and $e^{-i\omega t}$ get closer in phase, so
1380: the product $X(t) e^{-i\omega t}$ oscillates more slowly. The value of the sum of
1381: the products $X(t_r) e^{-i\omega t_r}$ will thus rise, reaching a maximum at $\omega
1382: = \tilde{\omega}$. The presence of a harmonic signal of frequency $\tilde{\omega}$
1383: thus produces a peak in the periodogram with maximum at $\tilde{\omega}$.
1384:
1385: The converse however is not true. A peak in the periodogram does not necessarily
1386: reflect the presence of an harmonic component in the signal $X$. Peaks might be
1387: produced by other effects. Thus, the presence of measurement error, signal noise, or
1388: random physical processes in the observed system might, by a spurious random
1389: fluctuation, also produce a peak. Peaks may also be produced by aliasing and / or
1390: spectral leakage, and the observing window. The potential for producing peaks that
1391: are not due to harmonic components in the observed signal makes the interpretation
1392: of peaks in the periodogram very difficult and presents many hazards and pitfalls
1393: for the unwary. The dangers posed by these effects were already noted by Arthur
1394: Schuster as early as 1906, ``... it has generally been assumed that each maximum in
1395: the amplitude of a harmonic term corresponded to a true periodicity. The extent to
1396: which this fallacious reasoning has been made use of would surprise anyone not
1397: familiar with the literature of the subject." (\citet{schuster06}, p 71-72).
1398: Strangely, his warning has often been ignored, and sometimes even disdainfully
1399: brushed aside.
1400:
1401: \section{LOMB-SCARGLE PERIODOGRAM}
1402: \label{scargleP}
1403:
1404: Following \citet{scargle82}, Appendix B, we define the Lomb-Scargle periodogram by
1405: the formula
1406: \begin{eqnarray}
1407: P_X (\omega ) &=& \frac{1}{2} \left\{
1408: \frac{\left[ \displaystyle \sum_{i=1}^N x_i \cos \omega (t_i - \tau)\right]^2 }
1409: {\displaystyle \sum_{i=1}^N \cos^2 \omega (t_i - \tau)}
1410: \ + \ \frac{\left[ \displaystyle \sum_{i=1}^N x_i \sin \omega (t_i - \tau)\right]^2 }
1411: {\displaystyle \sum_{i=1}^N \sin^2 \omega (t_i - \tau)} \right\}
1412: \label{lomb-scargle}
1413: \end{eqnarray}
1414: where the epoch translation parameter $\tau(\omega)$ is defined implicitly by the
1415: formula
1416: \begin{eqnarray}
1417: \tan ( 2\omega \tau ) &=& \frac{ \displaystyle \sum_{i=1}^N \sin ( 2 \omega t_i )}
1418: {\displaystyle \sum_{i=1}^N \cos ( 2 \omega t_i )}
1419: \label{lomb-scargle-A}
1420: \end{eqnarray}
1421: The data used to calculate $P_X(\omega)$ form a finite discrete time series
1422: consisting of $N$ values $X_i=X(t_i)$, $i=1,..., N$, of the physical parameter $X$
1423: measured at times $\{t_i | i= 1,2,...,N\}$ which are arbitrarily spaced in time.
1424: \citet{lomb76}, following \citet{barning63} and \citet{vanicek69}, arrived at this
1425: formula via a least squares fitting procedure in which sampled values $X(t_i)$ are
1426: fitted with an harmonic signal of frequency $\omega$. For these three authors
1427: therefore, $P_X(\omega)$ does {\em not} represent and attempt at estimating the
1428: Fourier power spectrum of any continuous time physical signal $X(t)$, but is rather
1429: a {\em spectral best-fit parameter} that displays how closely the data may be fitted
1430: with a single harmonic function of frequency $\omega$. The larger the value of
1431: $P_X(\omega)$, the better the fit.
1432:
1433: In contrast with these authors, \citet{scargle82} arrived at the same spectral
1434: function by first relaxing the definition of the DFT for application to the case of
1435: unevenly spaced data \citep[Appendix~A]{scargle82}, and then imposing two demands on
1436: the periodogram (which he calls the {\em modified}, or {\em generalised
1437: periodogram}) calculated from this relaxed definition: the statistical distribution
1438: of the generalised periodogram will be made as closely as possible the same as it is
1439: in the evenly spaced case; and, the generalised periodogram will be made invariant
1440: to translations in time. These two requirements yield uniquely the formulae in
1441: equations (\ref{lomb-scargle}) and (\ref{lomb-scargle-A}). Arguably, this restores
1442: the interpretation of the modified periodogram as an estimator of the power spectrum
1443: of the physical signal $X(t)$ in the case where the signal is unevenly sampled in
1444: time. However, it is probably more accurate to regard it as a spectral
1445: goodness-of-fit parameter. This view also enables one better to understand a variety
1446: of other, alternative, periodogram formulae currently offered in the literature.
1447:
1448: \section{Pure Noise}
1449: \label{noiseApp}
1450:
1451: A random process $X(t)$ is said to be a {\em purely random process}, {\em pure
1452: noise}, or {\em white noise}, if it consists of a sequence of {\em uncorrelated
1453: random variables}. This means that, for all $t'\neq t$,
1454: $$ {\rm cov}\left(X(t), X(t') \right)=0 $$
1455: Pure noise is the simplest of all random process models. It corresponds to a case
1456: where the process has ``no memory" in the sense that the value of the random
1457: variable $X(t)$ at time $t$ has no relation whatever to its value $X(t')$ at any
1458: other time $t'$, no matter how close or distant $t$ and $t'$ are to each other. In
1459: this sense, $X(t)$ neither remembers its past, nor is aware of its future. Knowing
1460: the value of $X(t_0)$ at any time $t_0$ therefore provides no way at all, other than
1461: by the probability distribution $p_{X(t)}=p(x,t)$, of predicting within reasonable
1462: limits and uncertainties the value of $X(t)$ at time $t$. This is to be contrasted
1463: with {\em correlated noise} where the values $X(t)$ and $X(t')$ are in general
1464: related or `correlated'. In this case, we can do better in predicting the value of
1465: $X(t+\tau)$ from $X(t)$ than in the case of uncorrelated, or pure, noise. From a
1466: knowledge of the value $X(t)$, we can set narrower limits on the probable values of
1467: $X(t+\tau)$ than is possible from the distribution $p_{X(t+\tau)}$ alone. \citep[p
1468: 114]{priestley81}.
1469:
1470: Pure noise is said to be {\em Gaussian} if the random variables $X(t)$ are jointly
1471: normally distributed. Noise of this kind is often called {\em Gaussian white noise}.
1472: In this case, the random variables $\{X(t)\}$ are also mutually independent.
1473:
1474: Note that some authors define pure noise more stringently. For them, a random
1475: process $X(t)$ is pure noise if the random variables $\{X(t)\}$ are {\em
1476: independent}, and {\em identically distributed with zero mean.}
1477:
1478: In this paper, a data set $\{ X_k | k=1,2,...,N_0\}$ is said to be pure noise if the
1479: values $X_k$ are {\em independent}, and {\em identically distributed random
1480: variables with zero mean}. For simplicity, we assume also that the $X_k$ are each
1481: normally distributed. Denote their common variance by $\sigma_X^2$. Since the $X_k$
1482: have zero mean, their covariance matrix is given by
1483: \begin{eqnarray}
1484: C_{jk} = E [ (X_j-\mu_{X_j}) ( X_k -\mu_{X_k})]
1485: = E [ X_j X_k ] = \sigma^2_X
1486: \delta_{jk}
1487: \end{eqnarray}
1488:
1489:
1490: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1491: %\begin{thebibliography}{}
1492: %\bibitem[Auri\`ere(1982)]{aur82} Auri\`ere, M. 1982, \aap, 109, 301
1493:
1494: \begin{thebibliography}{99}
1495:
1496: % Baglin
1497: \bibitem[\protect\citeauthoryear{Baglin et al.}{2002}]{baglin02} Baglin, A. et al., 2002,
1498: in First Eddington Workshop, Cordoba 11-15 June 2001, ed. J. Christensen-Dalsgaard
1499: \& I. Roxburgh, ESA-SP, 485, 17
1500:
1501: % Barning
1502: \bibitem[\protect\citeauthoryear{Barning}{1963}]{barning63}
1503: Barning, F. J. M., 1963, Bull. Astr. Inst. Netherlands, 17, 22
1504:
1505: % Cumming et al
1506: \bibitem[\protect\citeauthoryear{Cumming, Marcy \& Butler}{1999}]{cumming99} Cumming, A., Marcy, G. W.,
1507: \& Butler R. P., 1999, ApJ, 526, 890
1508:
1509:
1510: % Deeming
1511: \bibitem[\protect\citeauthoryear{Deeming}{1975}]{deeming75} Deeming, T. J., 1975, Ap\&SS, 36, 137
1512:
1513: % Horne and Baliunas
1514: \bibitem[\protect\citeauthoryear{Horne \& Baliunas}{1986}]{hb86} Horne, J. H., \& Baliunas, S. L.,
1515: 1986, ApJ, 302, 757
1516:
1517: % Koen
1518: \bibitem[\protect\citeauthoryear{Koen}{1990}]{koen90} Koen, C., 1990, ApJ, 348, 700
1519:
1520: % Lomb
1521: \bibitem[\protect\citeauthoryear{Lomb}{1976}]{lomb76} Lomb, N. R., 1976, Ap\&SS, 39,
1522: 447
1523:
1524: % Pojmanski
1525: \bibitem[\protect\citeauthoryear{Pojmanski}{1998}]{pojmanski98} Pojmanski, G., 1998,
1526: Acta Astron, 48, 35
1527:
1528: % Priestley
1529: \bibitem[\protect\citeauthoryear{Priestley}{1981}]{priestley81}
1530: Priestley, M. B., 1981, {\em Spectral Analysis and Time Series}, Vols 1\& 2,
1531: Elsevier Academic Press, London, ISBN 0-12-564922-3.
1532:
1533: % Scargle
1534: \bibitem[\protect\citeauthoryear{Scargle}{1982}]{scargle82}
1535: Scargle, J. D., 1982, ApJ, 263, 835
1536:
1537: % Schuster
1538: \bibitem[\protect\citeauthoryear{Schuster}{1898}]{schuster98}
1539: Schuster, A., 1898, Terrestrial Magnetism (now J.G.R.), 3, 13
1540:
1541: \bibitem[\protect\citeauthoryear{Schuster}{1906}]{schuster06}
1542: Schuster, A., 1906, Phil. Trans. Roy. Soc. Lond., 206, 69
1543:
1544: % Schwarzenberg-Czerny
1545: \bibitem[\protect\citeauthoryear{Schwarzenberg-Czerny}{1996}]{sc96}
1546: Schwarzenberg-Czerny, A., 1996, ApJ, 460, L107
1547:
1548: \bibitem[\protect\citeauthoryear{Schwarzenberg-Czerny}{1998}]{sc98}
1549: Schwarzenberg-Czerny, A., 1998, MNRAS, 301, 831
1550:
1551: % Thompson
1552: \bibitem[\protect\citeauthoryear{Thompson}{1971}]{thompson71}
1553: Thompson, R. O. R. Y., 1971, IEEE Trans., GE-9, 107
1554:
1555:
1556: % Vanicek
1557: \bibitem[\protect\citeauthoryear{Vanicek}{1969}]{vanicek69}Vanicek, P., 1969,
1558: Ap\&SS, 4, 387
1559:
1560: \end{thebibliography}
1561:
1562: \end{document}
1563:
1564: %%***************************************************************************
1565:
1566:
1567:
1568: This makes the search for a false alarm probability function in closed form
1569: intractable by currently available theoretical methods.
1570: