1: % \usepackage{lscape} - w thesis.tex
2:
3: %%% --------------------------------------------------
4: \chapter{Data Analysis}
5: \label{sec_dataanalysis}
6: \ifpdf
7: \graphicspath{{DataAnalysis/DataAnalysisFigs/PNG/}{DataAnalysis/DataAnalysisFigs/PDF/}{DataAnalysis/DataAnalysisFigs/}}
8: \else
9: \graphicspath{{DataAnalysis/DataAnalysisFigs/EPS/}{DataAnalysis/DataAnalysisFigs/}}
10: \fi
11:
12: \def\deg{$^\circ$}
13:
14: Data analysis in the "Pi of the Sky" experiment consists of on-line and off-line parts.
15: The on-line data analysis is required to control performance of the detector, but
16: it is also responsible for finding optical flashes in timescale 10 or 22 sec in real time.
17: Fast identification of optical flashes gives a possibility to distribute
18: alerts in the community for follow-up observations.
19: The off-line data analysis acts on the reduced data. The reduction
20: pipeline consists of three main stages : photometry, astrometry and
21: cataloging to the database. Final, reduced data consists of star brightness
22: measurements stored in the database, which provides easy and effective access.
23: Off-line analysis algorithms act on the database, there are several algorithms
24: developed for different purposes. In this thesis two off-line algorithms implemented
25: by author will be described in detail. These are flare recognition algorithm,
26: looking for brightness increase of existing stars and algorithm for finding new objects in the sky.
27: The database provides easy data access for broad spectrum of data analysis.
28: Except algorithms presented in this thesis there are also other
29: off-line algorithms implemented ( \cite{scan_nova}, \cite{mbiskup_var} ).
30:
31: \section{On-line data reduction}
32: \label{sec_online_reduction}
33: On-line data analysis is required for detector control. The most important
34: task is astrometry, it is a transformation of chip coordinates (x,y) to
35: celestial coordinates ($\lambda,\delta$) : \\
36:
37: \begin{equation}
38: T: (x,y) \rightarrow (\lambda,\delta)
39: \label{eq_astro_transform}
40: \end{equation}
41:
42: In order to obtain this transformation the following steps are performed :
43: \begin{itemize}
44: \item Fast image reduction - subtraction of dark image ( this is step is also
45: required by on-line analysis algorithm )
46: \item Photometry - identification of stars in an image and determination of instrumental magnitudes
47: and chip coordinates (x,y)
48: \item Astrometry - determination of non-linear transformation T:(x,y) $\rightarrow$ ($\lambda,\delta$)
49: It is iterational minimization procedure comparing stars identified in the
50: image by photometry with reference stars from external star catalog.
51: This procedure will be described in more detail in Section \ref{sec_image_reduction}.
52: \end{itemize}
53: After finding astrometry transform it is possible to calculate celestial
54: coordinates of any object on an image from its (x,y) coordinates.
55: The ($\lambda,\delta$) coordinates of image center are calculated and
56: compared against the expected position and can be used to correct mount
57: tracking speeds ( so called \texttt{autoguiding} procedure ).
58: Before analyzing an image with on-line algorithm fast image processing is
59: performed. The first step is dark image subtraction.
60: In the next step, image is transformed by special transformation called \texttt{laplace}\footnote{because it resembles a discrete version of Laplace operator}.
61: Value of each pixel is calculated as simple function of several surrounding
62: pixels. Values in pixels just around transformed pixels are summed and values in
63: other pixels far from it are subtracted. The idea of this transformation is
64: to calculate simple aperture brightness for every pixel.
65:
66: % laplace_type.eps
67: \begin{figure}[!htbp]
68: \begin{center}
69: \leavevmode
70: \ifpdf
71: \includegraphics[width=6in]{laplace_type.gif}
72: \else
73: \includegraphics[width=6in]{laplace_type.eps}
74: \fi
75: \caption{Different \texttt{laplace} types tested in on-line flash recognition algorithms}
76: \label{fig_laplacjans}
77: \end{center}
78: \end{figure}
79:
80:
81: Several types of filters which were tested are shown in Figure \ref{fig_laplacjans}.
82: Images before and after applying the \texttt{g54 laplace} filter ( aperture 4 in Fig. \ref{fig_laplacjans} ) are shown in Figure \ref{fig_before_and_after_laplace}
83: it can be clearly seen that stars are sharper on the filtered image.
84: Finally, one filter was chosen according to lowest ratio of $\sigma_{g} / g_{avg}$
85: value calculated for faint stars, where $g_{avg}$ denotes average of maximum \texttt{laplace}
86: value for given star and $\sigma_{g}$ stands for its dispersion.
87: For old prototype version with Carl Zeiss f=50mm lenses \texttt{laplace} 4 (\texttt{g54}) was used and for the Canon f=85mm \texttt{laplace} 12 is used.
88: On-line algorithm is based on transformed images, distribution of pixel values after
89: such transformation is centered around zero ( Fig. \ref{fig_laplace_distr} ).
90: For every collected image a Gaussian curve is fitted and signal threshold $T_n$ is calculated
91: as multiplicity of sigma value ( typically $T_n$ = 5$\sigma_B$ or 6$\sigma_B$ ).
92:
93: % /pi20/msok/praca/MY/badania/Laplace/
94: % at home : phd/images/laplace
95: \begin{figure}[!htbp]
96: \begin{center}
97: \leavevmode
98: \ifpdf
99: \includegraphics[width=6in]{before_and_after_laplace.gif}
100: \else
101: \includegraphics[width=6in]{before_and_after_laplace.eps}
102: \fi
103: \caption{Sky image before and after applying the \texttt{laplace} filter}
104: \label{fig_before_and_after_laplace}
105: \end{center}
106: \end{figure}
107:
108:
109:
110: % /pi20/msok/praca/MY/badania/Laplace/
111: \begin{figure}[!htbp]
112: \begin{center}
113: \leavevmode
114: \ifpdf
115: \includegraphics[width=6in]{g12_distr.gif}
116: \else
117: \includegraphics[width=6in]{g12_distr.eps}
118: \fi
119: \caption{Distribution of \texttt{laplace} 12 values on a single image}
120: \label{fig_laplace_distr}
121: \end{center}
122: \end{figure}
123:
124:
125: \section{On-line flash recognition algorithms}
126: \label{sec_online_flashes_algo}
127:
128: The aim of this type of algorithms is to find optical flashes occurring in
129: single image time scale ( 10 sec ). The signature of such events is the
130: following. Interesting event is an object which appears in a new image of
131: the sky and was not present in the previous image of the same field which was
132: taken a moment before.
133: However, this simple idea is not so simple in practical realization. Most of
134: the events found by such simple comparison of two images are due to
135: background. The crucial task of the efficient algorithm is to
136: reject most of the background without too much loss of the signal.
137: It is realized by a multilevel triggering system based on ideas
138: used in particle physics.
139: Every image consists of $4 \cdot 10^6$ pixels suspected of being potential
140: interesting event. Image should be analysed while next image is being
141: collected which takes $\approx 12$s in the current configuration. This means
142: that the algorithm must be fast.
143: First trigger levels are simple and fast, they reject big amount of event
144: candidates with simple criteria. Higher trigger levels have more time and
145: can use more sophisticated criteria to reject background events.
146:
147: \subsection{First Level Trigger}
148: This level of algorithm must handle the highest data rate, so it must be
149: very fast and simple. It should preserve most of the signal and reject big fraction of non-interesting pixels.
150: At this stage flash-like events in single camera are identified and saved to
151: log files and optionally to the database. The following list of cuts are applied to every
152: pixel in the image : \\
153: \begin{itemize}
154:
155: \item[$\bullet$] \boldmath{$T_n$} - this cut selects stars on new image by requiring
156: signal in the analysed pixel. The condition for signal presence is :
157:
158: \begin{equation}
159: N(x,y) > T_n
160: \label{eq_tn_cut_cond}
161: \end{equation}
162:
163: where N(x,y) is ADU value in pixel (x,y) of the new image and the threshold $T_n$ is specified by DAQ
164: configuration parameters in multiplicities of $\sigma_B$ ( see Tab. \ref{tab_fltparams} and Fig. \ref{fig_laplace_distr} ).
165: The goal of this cut is selection of all stars in new image.
166:
167: \item[$\bullet$] \boldmath{$T_v$} - this cut rejects constant stars. It
168: requires that there is no signal on the previous frame. "Previous frame" in this case
169: means not just one single image, but average of several previous images.
170: The condition imposed on value of pixel in the "previous image" is the following :
171:
172: \begin{equation}
173: P(x,y) < T_v
174: \label{eq_tv_cut_cond}
175: \end{equation}
176:
177: where P(x,y) is the value in pixel (x,y) on the average of $N_{aver}$ previous images.
178: Pixels remaining after this cut should be new objects which appeared on new
179: image and were not present on previous images.
180:
181: \item[$\bullet$] \textbf{MinLaplace} - rejects pixels which have value on previous
182: image lower then minimal accepted value ( $T_{MinLap}$ ). This cut allows to reject edges of
183: bright stars where values of pixels after \texttt{laplace} filter often become negative, but
184: can also vary to values exceeding $T_n$.
185:
186: \item[$\bullet$] \textbf{IfMoreAfterTv} - rejects the whole image if number of pixels
187: accepted after $T_v$ cut exceeds certain limit $N_{MaxTv}$.
188: This cut allows to reject images with big number of events which are due to system
189: error, Moon light or clouds. The image is bad, all events are certainly
190: rubbish so they are rejected and no further analysis of this image is performed.
191:
192: \item[$\bullet$] \textbf{LocalMax} - requires that pixel value is a local maximum. This
193: cut allows to choose only one pixel of star like object for further analysis.
194:
195: \item[$\bullet$] \textbf{SkipOverlaps} - checks number of accepted pixels in
196: certain radius $R_{overlap}$ from current pixel and leaves only
197: one event and skips the others. This cut narrows the number of pixels
198: to be analysed which are related to the same object to a single one.
199:
200: % PATH : /pi20/msok/praca/MY/badania/Shape
201: % show_shape! - lub z
202: % piphoto : /pi20/msok/praca/MY/badania/cosmics/20070221/piphoto/
203: % allevents : piphoto : /pi20/msok/praca/MY/badania/cosmics/20070221/allevents
204: % dlaczego - tak duzo jest gwiazd 1-pixlowych ??? - zwlaszcza z piphoto ?
205: % moze tabelke pokazac 1pixel -> shape=0.64 , 2 pixele=... , etc ...
206: %
207: \item[$\bullet$] \textbf{Shape} - object shape indicator S is calculated. It is defined as :
208:
209: \begin{equation}
210: S = \frac{S_{cluster}}{S_{circle}}
211: \label{eq_shape_def}
212: \end{equation}
213:
214: % circumcircle
215: where $S_{cluster}$ is area of cluster and $S_{circle}$ is the area of the
216: smallest circle circumscribed on this cluster.
217: Cluster is defined as group of pixels around current pixel
218: with values satisfying $N(x,y) \geq T_{cluster}$.
219: Shape is required not to be elongated by imposing :
220:
221: \begin{equation}
222: S > T_{shape}
223: \label{eq_shape_cond}
224: \end{equation}
225:
226: The idea of shape calculation is shown in Figure \ref{fig_stars_shape_distr}. The distribution of shape
227: value for stars in single image is shown in Figure \ref{fig_stars_shape_distr}.
228:
229: % /pi20/msok/praca/MY/badania/shape/20070221/allevents/
230: % .x histo_shape.C("shape.txt",0,0)
231: \begin{figure}[!htbp]
232: \begin{center}
233: \leavevmode
234: \ifpdf
235: \includegraphics[width=2.7in,height=2.7in]{shape.gif}
236: \includegraphics[width=2.7in,height=2.7in]{shape_allevents.gif}
237: \else
238: \includegraphics[width=2.7in,height=2.7in]{shape.eps}
239: \includegraphics[width=2.7in,height=2.7in]{shape_allevents.eps}
240: \fi
241: \caption{Idea of shape indicator calculation (left) and its distribution for events from a single night and camera (right)}
242: \label{fig_stars_shape_distr}
243: \end{center}
244: \end{figure}
245:
246:
247: \item[$\bullet$] \textbf{BlackPixels} - this cut rejects pixels which have much
248: smaller signal then neighboring pixels causing \texttt{laplace} filter to be high
249: due to underestimation of the background level. The following requirement is imposed
250: on every pixel :
251:
252: \begin{equation}
253: \frac{Min(P_{-})}{\sum P_{-}} \geq T_{black}
254: \label{eq_black_cond}
255: \end{equation}
256:
257: where $P_{-}$ is value of pixel entering the \texttt{laplace} function with minus sign.
258: Black pixels on reduced image can be due to CCD defects or temperature
259: fluctuations, but are rather very seldom ( Tab. \ref{tab_fltcuts} ).
260:
261:
262: \item[$\bullet$] \textbf{HotPixels} - due to CCD chip defects some pixels can give
263: much higher signal then normal good pixels. Such effects should generally be
264: subtracted by the dark image subtraction. However, sometimes it is not
265: enough, because new hot pixels can appear temporary during a night and
266: become quiet again later. The main reason for this are excitations
267: from cosmic ray hits. Two ways of rejecting such events have been implemented.
268: The first one is calculation of average value in pixel on previous images,
269: the imposed criteria is :
270:
271: \begin{equation}
272: \frac{\sum_{i=-1}^{N_{aver}}{P_i(x,y)}}{N_{aver}} < T_{hot}
273: \label{eq_hotpixel_cond}
274: \end{equation}
275:
276: where $P_i(x,y)$ is ADU value in pixel in image \textit{i} before current one.
277: In the case of tracking mount this cut is neutral due to $T_v$ cut which is stronger.
278: Second anti-hotpixel cut is rejection of pixels by the list of known hot-pixels.
279: This list is updated regularly when new defects are found. This is done
280: "manually" by running report which shows events described as hot pixel. In case
281: certain pixel is regularly giving false alerts it can be added to list of known
282: hot pixels which is stored in the database ( Fig. \ref{fig_starcat} ).
283:
284: \item[$\bullet$] \textbf{IfMore} - checks nearby events in distance of $R_{ifmore}$ pixels, in
285: case number of events exceeds limit of $N_{ifmore}$ all nearby events are rejected.
286: This cut allows to reject events caused by planes or satellites.
287: \end{itemize}
288:
289: Parameters for algorithms are passed as described in the Section \ref{sec_daq_cfg}.
290: The parameters used in First Level Trigger (FLT) are listed in Table \ref{tab_fltparams}.
291:
292: \begin{table}[htbp]
293: \begin{center}
294: \begin{tiny}
295: \begin{tabular}{|c|c|c|c|c|}
296: \hline
297: \textbf{Cut} & \textbf{Parameter Name} & \textbf{\begin{minipage}{1.8cm}{Value for confirmation on next}\end{minipage}} & \textbf{\begin{minipage}{1.8cm}Value for \hspace{3mm} coincidence\end{minipage}} & \textbf{Notes} \\
298: & & & & \\
299: \hline
300: LaplaceType & LaplaceType & 12 & 12 & For Carl Zeiss f=50 \texttt{laplace}=4 \\
301: \hline
302: $T_n$ & $T_n$ & 6 & 5 & For Carl Zeiss f=50 $T_n$=5 \\
303: \hline
304: $T_v$ & $T_v$ & 2 & 2 & For Carl Zeiss f=50 $T_v$=2.5 \\
305: & $N_{aver}$ & 7 & 7 & Number of averaged previous images \\
306: \hline
307: MinLaplace & $T_{MinLap}$ & 0 & 0 & - \\
308: \hline
309: IfMoreAfterTv & $N_{MaxTv}$ & 3000 & 3000 & - \\
310: \hline
311: SkipOverlaps & SkipOverlaps & 1 & 1 & enable/disable \\
312: SkipOverlaps & $R_{overlap}$ & 4 & 4 & radius in which overlaps are skipped \\
313: \hline
314: Shape & $T_{shape}$ & 0.2 & 0.2 & - \\
315: \hline
316: BlackPixels & $T_{black}$ & 0.5 & 0.5 & - \\
317: % & CCD\_BLACK\_PIXELS\_IF\_N\_SIGMA\_BELOW & 3 & ??? \\
318: \hline
319: HotPixels & $T_{hot}$ & 3.6 & 3.6 & Threshold in anti-hotpixel cut \\
320: \hline
321: IfMore & $N_{IfMore}$ & 20 & 20 & \\
322: & $R_{IfMore}$ & 150 & 150 & \\
323: \hline
324: \end{tabular}
325: \end{tiny}
326: \caption{FLT parameters summary, real names of parameters used in
327: configuration files can be found in Table \ref{tab_fltparams_trans} in Appendix \ref{sec_appendix2}}
328: \label{tab_fltparams}
329: \end{center}
330: \end{table}
331:
332:
333: The output from FLT is basically a list of events from single camera. These
334: events are saved to a log file allevents\_N.log ( N stands for index of camera ).
335: Optionally they can be saved to database to provide easy access for further analysis.
336: Table \ref{tab_fltcuts} shows background rejection efficiency of subsequent FLT cuts.
337:
338: % na laptopie :
339: % C:\doc\msok\phd\obliczenia\bacground\ccddouble\20060527.ods
340: \begin{table}[htbp]
341: \begin{center}
342: \begin{tiny}
343: \begin{tabular}{|c|c|c|c|}
344: \hline
345: \textbf{Cut} & \textbf{\% of all events} & \textbf{\% of events from previous level} & \textbf{Number of events} \\
346: \hline
347: All & - & - & $3.212 \cdot 10^9$ \\
348: \hline
349: Tn & $2.06 \cdot 10^{-2}$ & 2.06\% & 66434214 \\
350: \hline
351: Tv & $2 \cdot 10^{-5}$ & 0.1\% & 68345 \\
352: \hline
353: MinTv & $2 \cdot 10^{-5}$ & 96.99\% & 66293 \\
354: \hline
355: Overlap & $5 \cdot 10^{-6}$ & 25.79\% & 17100 \\
356: \hline
357: Black & $5 \cdot 10^{-6}$ & 100\% & 17100 \\
358: \hline
359: Hot & $5 \cdot 10^{-6}$ & 100\% & 17100 \\
360: \hline
361: IfMore & $5 \cdot 10^{-6}$ & 99.85\% & 17075 \\
362: \hline
363: Coinc & $2.5 \cdot 10^{-6}$ & 46.28\% & 8099 \\
364: \hline
365: Satellites Catalog & $2.49 \cdot 10^{-6}$ & 98.96\% & 8015 \\
366: \hline
367: Star Catalog & $2.32 \cdot 10^{-6}$ & 93.11\% & 7463 \\
368: \hline
369: Shape & $2.25 \cdot 10^{-6}$ & 97.10\% & 7247 \\
370: \hline
371: Tracks & $1.55 \cdot 10^{-9}$ & 0.06\% & 5 \\
372: \hline
373: Accepted & $1.55 \cdot 10^{-9}$ & 0.06\% & 5 \\
374: \hline
375: \end{tabular}
376: \end{tiny}
377: \caption{Number of events remaining after subsequent cuts of coincidence algorithm for night 2006-05-27/28}
378: \label{tab_fltcuts}
379: \end{center}
380: \end{table}
381:
382:
383:
384: \subsection{Second Level Trigger}
385: \label{sec_slt}
386:
387: The action at this level depends on the type of the system setup. Generally three
388: configurations are possible :\\
389: \begin{itemize}
390: \item[$\bullet$] two cameras on a single mount working in coincidence.
391: In this configuration events found by the first camera are verified in the
392: corresponding image from the second camera. Only events present in images from both cameras are
393: accepted. This configuration is realized by the prototype system in LCO.
394:
395: \item[$\bullet$] single camera, coincidence is replaced by confirmation of
396: signal on next image. This version of algorithm was also realized in LCO
397: when one of the cameras was not working due to technical problems.
398: This can be used as cross-check algorithm for two cameras working in coincidence.
399:
400: \item[$\bullet$] two cameras in separate locations working in coincidence.
401: This will be realized in the full version of the system. Cameras will be
402: paired, both cameras in the pair will observe the same field in the sky.
403: Spatial and time coincidence of the flash in both cameras will be required.
404: \end{itemize}
405:
406: % script : show_shape!
407: % cosmics : /pi20/msok/praca/MY/badania/cosmics/20070221/
408: % shape : /pi20/msok/praca/MY/badania/cosmics/20070221/piphoto/
409: % /pi20/msok/praca/MY/badania/cosmics/20070221/hist.txt
410: % trzeba wziasc z danych kiedy byly 2 kamery !!!
411: % heplx48 : /disk03/results/fits/20060527/cosmics - get_cosmics!
412: % heplx48 : getpart2 k2a_060527_01665.fitc,k2a_060527_01666.fitc,k2a_060527_01667.fitc - -part_size=50 -dark=dark1.fit -ra_hms=16h25m25.94s -dec=3.8670 -outdir=cosmic01666/Cam0/
413: % heplx48 : getpart2 k2b_060527_01665.fitc,k2b_060527_01666.fitc,k2b_060527_01667.fitc - -part_size=50 -dark=dark1.fit -ra_hms=16h25m25.94s -dec=3.8670 -outdir=cosmic01666/Cam0/
414: % + script fit2show!
415: % /pi20/msok/praca/MY/badania/cosmics/20060527/final/cosmic01118/final/
416:
417: \begin{figure}[!htbp]
418: \begin{center}
419: \leavevmode
420: \ifpdf
421: \includegraphics[width=6in]{cosmic01666.gif}
422: \includegraphics[width=6in]{cosmic01118.gif}
423: \else
424: \includegraphics[width=6in]{cosmic01666.eps}
425: \includegraphics[width=6in]{cosmic01118.eps}
426: \fi
427: \caption{Examples of events caused by cosmic ray hitting the CCD chip, with PSF easy to distinguish from stars (upper image) and very similar to PSF of stars (lower image)}
428: \label{fig_cosmics}
429: \end{center}
430: \end{figure}
431:
432:
433: In any case coincidence requirement is one of the most important cuts.
434: The main goal is rejection of cosmic rays hitting the CCD chip and imitating astrophysical
435: flashes (Fig. \ref{fig_cosmics}). In many cases cosmic rays have Point
436: Spread Function (PSF) completely different then PSF of stars and they could
437: be rejected by a shape recognition procedure. However, in some cases
438: they are very similar to PSF of the stars. Even if this is a
439: very small fraction of all cosmic ray events this would cause all flashes
440: found by the algorithm to be uncertain. A way of definite rejection of all
441: cosmic rays events is required for credible flash recognition algorithm.
442: Probability that different cosmic ray particles will hit two chips in
443: the same time and in the same positions ( with respect to stars ) is negligible.
444: Coincidence is also very effective way of rejecting background events due to sky
445: background fluctuations, edges of bright stars and clouds.
446: In the prototype version cameras take images parallely so the only parameter
447: is the angular distance of events in both cameras, currently used default value is
448: $R_{coinc} = 150$ arcsec ( 250 arcsec was used for Carl Zeiss f=50mm lenses ).
449: It was determined from distribution of angular distances
450: of corresponding stars in both cameras ( Fig. \ref{fig_coinc_radius} ).
451:
452: % /pi20/msok/praca/MY/badania/coinc/
453: % coinc_radius!
454: \begin{figure}[!htbp]
455: \begin{center}
456: \leavevmode
457: \ifpdf
458: \includegraphics[width=6in]{coincidence_radius.gif}
459: \else
460: \includegraphics[width=6in]{coincidence_radius.eps}
461: \fi
462: \caption{Distribution of angular distances between events from corresponding images collected by cameras k2a and k2b during night
463: 2006-05-28/29. Events with the distance < 250 arcsec are accepted.}
464: \label{fig_coinc_radius}
465: \end{center}
466: \end{figure}
467:
468:
469: In the case of coincidence between cameras in separated sites, the
470: significance of coincidence is even bigger. In this configuration
471: it is possible to reject close Earth flashing objects using the parallax ( Fig. \ref{fig_parallax} ).
472: Artificial satellites orbiting the Earth may rotate and sometimes reflex Sun light
473: causing flash-like events. The best method to distinct such kind of flashes is to use parallax.
474: In the prototype version of the experiment two cameras are installed on
475: single mount. However, this method was tested by coincidence with the RDOT
476: experiment \cite{rdot_web_page} located on La Silla at the distance of $\approx 30$ km.
477:
478: % wikpedia : http://en.wikipedia.org/wiki/Parallax
479: \begin{figure}[!htbp]
480: \begin{center}
481: \leavevmode
482: \ifpdf
483: \includegraphics[width=6in]{parallax_new.gif}
484: \else
485: \includegraphics[width=6in]{parallax_new.eps}
486: \fi
487: \caption{A simplified example of parallax ( image taken from \cite{wiki_parallax} ) }
488: \label{fig_parallax}
489: \end{center}
490: \end{figure}
491:
492:
493:
494: % see: http://grb.fuw.edu.pl/pi/rdot/sat050205/index.html
495: % http://grb.fuw.edu.pl/pi/rdot/index.html
496: % /home/8/msok/doc/PHD/DataAnalysis/DataAnalysisFigs/RDOT
497: \begin{figure}[!htbp]
498: \begin{center}
499: \leavevmode
500: \ifpdf
501: \includegraphics[width=6in]{pi_vs_rdot_20050205.gif}
502: \else
503: \includegraphics[width=6in]{pi_vs_rdot_20050205.eps}
504: \fi
505: \caption{Stereo observations of near Earth satellite by experiments PI and RDOT}
506: \label{fig_rdot_parallax}
507: \end{center}
508: \end{figure}
509:
510:
511:
512: Image collected by the prototype containing an optical flash was compared with
513: image of same area of the sky taken by RDOT telescope at the same time.
514: It can be clearly seen in Figure \ref{fig_rdot_parallax} that optical flash
515: is visible in different positions with respect to the stars. Requirement
516: of spatial coincidence would reject such event. It is probably caused by a
517: satellite in distance of $< 25000$ km from Earth.\\
518: This method has been tested for a few nights only. During normal operation it is
519: not possible for the prototype to use parallax to reject flashing satellites.
520: In order to reject most of such events a database of known satellites is used.
521: It is retrieved every night from the Internet and contains $\approx 10000$ orbital elements.
522: For every image, positions of all satellites in the database are calculated and every flash candidate is verified.
523: In case it is closer then $R_{sat} = 0.5^\circ$ from any of the satellites it
524: is rejected. The rejection radius was determined from distribution of angular distances
525: from flashes to closest satellite from the database which is clearly peaked around zero ( see Fig. \ref{fig_satradius_distr}). \\
526: The red dots on this plot represent distribution of distances to the closest
527: satellite from the catalog for randomly distributed flashes.
528: A clear peak at $R_{sat} < 0.5^\circ$ is visible, it corresponds to real satellites.
529:
530: % /pi20/msok/praca/MY/badania/sat_radius/20041028/
531: % sat_radius!
532: % wyszlo ze satellitesdb.qth byl corrupted ! po poprawce dorobilem symulacje :
533: % cd heplx43:/pi20/msok/praca/MY/badania/sat_radius/20070524
534: % doit! :
535: % nohup satgen -max_flash_no=100000 -start_gmtime=20070524_230000 -end_gmtime=20070525_100000 >20070524_gen.txt 2>&1 &
536: % sat_radius_both! sat_radius_20070524.txt 20070524_gen.txt 20070524
537: % laptop : C:\doc\msok\phd\obliczenia\sat_radius\20070524
538: %
539: % heplx43:/pi20/msok/praca/MY/badania/sat_radius/
540: % 20070524/20070524_vs_mc.eps
541: % 20070526/20070526_vs_mc.eps
542: \begin{figure}[!htbp]
543: \begin{center}
544: \leavevmode
545: \ifpdf
546: \includegraphics[width=6in]{sat_radius_distr.gif}
547: \includegraphics[width=6in]{20070526_vs_mc.gif}
548: \else
549: \includegraphics[width=6in]{sat_radius_distr.eps}
550: \includegraphics[width=6in]{20070526_vs_mc.eps}
551: \fi
552: \caption{Distribution of distance from flash event candidate to the closest satellite from the catalog. For events found by coincidence algorithm during night 2004.10.28/29 (upper plot) and for single camera events from night 2007.05.26/27 (lower plot). }
553: \label{fig_satradius_distr}
554: \end{center}
555: \end{figure}
556:
557:
558: The orbital elements databases are not complete, there are many satellites
559: which are not included ( e.g. spy satellites ). In order to reject such kind of objects event candidates
560: from many consecutive images are examined against track conditions.
561: In case it is possible to fit track to set of events from
562: different images and velocity of object is constant all events on the
563: track are rejected. This rejects big fraction of flashing satellites and
564: planes ( see Fig. \ref{fig_night_tracks} ), however it is possible that rarely flashing ( rotating ) satellites
565: survive this cut.
566:
567: % http://grb.fuw.edu.pl/pi0/daq/events/200501/20050123/
568: % http://grb.fuw.edu.pl/pi0/daq/events/200501/20050116/Tracks/Cam0/
569: % plot_tracks!
570: \begin{figure}[!htbp]
571: \begin{center}
572: \leavevmode
573: \ifpdf
574: \includegraphics[width=6in]{20050116.gif}
575: \else
576: \includegraphics[width=6in]{20050116.eps}
577: \fi
578: \caption{Events rejected by track cut during single night 2005-01-16/17}
579: \label{fig_night_tracks}
580: \end{center}
581: \end{figure}
582:
583:
584: At this level of the trigger each event
585: candidate is checked against the catalog of constant stars. TYCHO-2 star catalog \cite{tycho}
586: is used for this purpose. The event candidate is rejected in case there is a
587: star brighter then $Mag_{max}$ in radius $R_{star}$ (see Tab. \ref{tab_sltparams}).
588: Stars can imitate flashes mainly due to clouds. When cloud moves and uncovers a star the FLT identifies
589: such an event as flash.
590: All events accepted by coincidence are saved to files \texttt{verifiedevents\_N.log} and
591: optionally to the database.
592: Events accepted by the SLT are saved to \texttt{finalevents\_N.log} and to the database.
593: Parameters used in the SLT are listed in Table \ref{tab_sltparams}.
594: Block diagram of on-line flash recognition algorithm is shown in Figure \ref{fig_online_algo}.
595: Rejection efficiency of subsequent FLT and SLT cuts are show in Figure \ref{fig_backgr_cuts_eff}.
596:
597: \begin{table}[htbp]
598: \begin{center}
599: \begin{tiny}
600: \begin{tabular}{|c|c|c|c|}
601: \hline
602: \textbf{Cut} & \textbf{Parameter Name} & \textbf{Current Value} & \textbf{Notes} \\
603: \hline
604: Coincidence & $R_{coinc}$ & 150'' & It was 250'' for objectives f=50mm \\
605: \hline
606: Confirmation on next & $N_{confirm}$ & 1 & in case of 2 cameras working it is 0 \\
607: \hline
608: Satellite Catalog & $R_{sat}$ & 1800'' & angular distance from satellite to reject event \\
609: \hline
610: Star Catalog & $R_{star}$ & 120'' & angular distance from catalog star to reject event \\
611: & $Mag_{max}$ & 13 & minimum brightness of used catalog stars \\
612: \hline
613: Track & $N_{track}$ & 200 & number of subsequent images used for track fit \\
614: & & & \\
615: & $\chi^2_{add}$ & 700 & \begin{minipage}{6cm}{Maximum allowed distance ( in pixels squared ) from existing track to match event to this track }\end{minipage} \\
616: & & & \\
617: & $\chi^2_{seed}$ & 100 & Maximum value of $\chi^2$ to initialize new track \\
618: & & & \\
619: \hline
620: \end{tabular}
621: \end{tiny}
622: \caption{SLT parameters summary, real names of parameters used in
623: configuration files can be found in Table \ref{tab_sltparams_trans} in Appendix \ref{sec_appendix2}}
624: \label{tab_sltparams}
625: \end{center}
626: \end{table}
627:
628: % scripts : calc_backgr! calc_backgr_slt! calc_backgr_slt_confirm! show_cuts_confirm!
629: % show_cuts.C show_cuts_double.C
630: % DATA :
631: % ccdsingle : heplx43:/pi20/msok/data/20070308/RESULTS/20070308_200414
632: % show_cuts_confirm! 20070308
633: % ccddouble : heplx43:/pi20/msok/praca/MY/badania/cuts/20060527
634: % show_cuts_coinc! 20060527
635: \begin{figure}[!htbp]
636: \begin{center}
637: \leavevmode
638: \ifpdf
639: \includegraphics[width=2.7in,height=2.7in]{20070308_confirm.gif}
640: \includegraphics[width=2.7in,height=2.7in]{20060527_coinc_k2a.gif}
641: \else
642: \includegraphics[width=2.7in,height=2.7in]{20070308_confirm.eps}
643: \includegraphics[width=2.7in,height=2.7in]{20060527_coinc_k2a.eps}
644: \fi
645: \caption{Background rejection efficiency of algorithm requiring confirmation of flash on next image (left) and coincidence of two cameras ( right )}
646: \label{fig_backgr_cuts_eff}
647: \end{center}
648: \end{figure}
649:
650: \begin{figure}[!htbp]
651: \begin{center}
652: \leavevmode
653: \ifpdf
654: \includegraphics[width=5in]{online_algo.gif}
655: \else
656: \includegraphics[width=5in,height=8in]{online_algo.eps}
657: \fi
658: \caption{Block diagram of on-line flash recognition algorithm}
659: \label{fig_online_algo}
660: \end{center}
661: \end{figure}
662:
663: \subsection{Third Level Trigger}
664: The first two levels of the trigger retain very small number of events, on average
665: it is not more then 10 per night. It depends strongly on weather conditions and in case of
666: cloudy night this number can reach hundreds, but in this case fast inspection
667: can reject most of them. However, in the full system the number of events will
668: be 16 time larger reaching 100-200 per night and this will be much more
669: difficult to inspect. For this reason the third level of the
670: trigger has been implemented. It checks final events accepted by previous levels
671: which ensures that only small number of events will be examined.
672: Thus it is possible to implement much more sophisticated and time consuming algorithms to check
673: every event. Current implementation of the TLT consists of the following cuts :\\
674:
675: % hough plots : heplx43:/pi20/msok/praca/MY/badania/HOUGH
676: % http://grb.fuw.edu.pl/pi0/daq/events/200412/20041204/EventsHtml/Frame01660/event5.php
677: % if( ( max_val > (m+t*r) ) && ( max_val > gCCDParams.m_HoughDistrMaxLimit*m)){
678: % /disk03/results/DAQ_RESULTS/20041204/RESULTS/SLT/RESULTS/Event04120400015/Cam0
679: \begin{itemize}
680: \item[$\bullet$] Comparison of event on both cameras and require signal to
681: be similar on both cameras, by imposing condition : $L_{min} / L_{max} < L_{diff}$
682: \item[$\bullet$] Checks sphericity again with optionally more strict criteria $Shape < T_{shape}^{TLT}$
683: \item[$\bullet$] Simple Hough transform \footnote{Hough transform is a technique
684: of image transform from (x,y) to cylindrical (r,$\phi$) coordinates in order to find particular shapes in an image} - uses small image part surrounding event. It
685: finds pixels with signal above certain level $T_{hough}$ and creates
686: distribution of $\phi$ coordinate ( $\phi = atan( (y-y0)/(x-y0) )$ ).
687: In case this distribution has significant peak this means it is probably due
688: to straight line from a plane or a satellite ( Fig. \ref{fig_hough_phi} )
689:
690: % /pi20/msok/praca/MY/badania/HOUGH
691: % http://grb.fuw.edu.pl/pi0/daq/events/200412/20041204/EventsHtml/Frame01660/event5.php
692: % plot_hough.C( "hough.log",0,0,0,3.2,50,0,"Phi angle coordinate of pixel","Number of pixels ")
693: \begin{figure}[!htbp]
694: \begin{center}
695: \leavevmode
696: \ifpdf
697: \includegraphics[width=2.8in,height=2.8in]{eventframe1340_camera0_eventno5_currframe1340.gif}
698: \includegraphics[width=2.8in,height=2.8in]{hough.gif}
699: \else
700: \includegraphics[width=2.8in,height=2.8in]{eventframe1340_camera0_eventno5_currframe1340.eps}
701: \includegraphics[width=2.8in,height=2.8in]{hough.eps}
702: \fi
703: \caption{Original event image (left image) and distribution of $\phi$ angle coordinate for background event (right image)}
704: \label{fig_hough_phi}
705: \end{center}
706: \end{figure}
707:
708: Event will be considered as straight line if the maximum of distribution of angle $\phi$
709: satisfies the following condition :
710:
711: % + \usepackage{amsmath}
712: \begin{equation}
713: \begin{split}
714: N_{max}(\phi) > MEAN_Y + T_{hough\_distr} * RMS_y \hspace{2mm} \\
715: AND \hspace{2mm} \\
716: N_{max}(\phi) > T_{hough\_height} * MEAN_Y
717: \end{split}
718: \label{eq_hough_cond}
719: \end{equation}
720:
721: \item[$\bullet$] Track check : checks event against fitted tracks if the
722: event was not correctly merged to existing track by the on-line algorithm
723: \item[$\bullet$] Algorithm on parts : algorithm is re-run on small parts
724: around the event, with less strict threshold $T_n^{TLT}= 4\cdot \sigma_B$.
725: The event is rejected if tracks are found and it belongs to one of these tracks.
726: \item[$\bullet$] Cloud check : checks number of stars in the full image and
727: rejects event if $N_{stars} < 8000$.
728: \item[$\bullet$] Frame line : in case line can be fitted to events from a
729: single frame rejected by Hough transform, all events from the same frame and not
730: yet rejected, are matched to this line. In case event matches this line it is rejected
731: \item[$\bullet$] All line check : checks if straight line can be fitted to
732: events from final list. It rejects final events matching this line.
733: \end{itemize}
734:
735: % configuration file slt.cfg
736: \begin{table}[htbp]
737: \begin{center}
738: \begin{tiny}
739: \begin{tabular}{|c|c|c|c|}
740: \hline
741: \textbf{Cut} & \textbf{Parameter Name} & \textbf{Current Value} & \textbf{Notes} \\
742: \hline
743: Hough Transform & $T_{hough}$ & 1.5 & Threshold for choosing pixels \\
744: Hough Transform & $T_{hough\_distr}$ & 4.5 &
745: \begin{minipage}{4cm}
746: Threshold for peak in $\phi$ distribution ( in $\sigma$ )
747: \end{minipage} \\
748: Hough Transform & $T_{hough\_height}$ & 2.0 &
749: \begin{minipage}{4cm}
750: Threshold for minimum peak height ( in multiplicity of mean value )
751: \end{minipage} \\
752: & & & \\
753: \hline
754: clouds cut & $N_{tstars}$ & 40000 & typical number of stars on whole image \\
755: clouds cut & $R_{clouds}$ & 0.2 &
756: \begin{minipage}{4cm}
757: reject if when number of stars in the image < $R_{clouds} \cdot N_{tstars}$
758: \end{minipage} \\
759: & & & \\
760: \hline
761: Event Comparison & $L_{diff}$ & 0.25 & requires similar signal on both cameras \\
762: & & & \\
763: \hline
764: & & & \\
765: Algorithm on parts & $T_n^{TLT}$ & 4.00 & more loose then normal $T_n$ \\
766: \hline
767: \end{tabular}
768: \end{tiny}
769: \caption{TLT parameters summary, real names of parameters used in
770: configuration files can be found in Table \ref{tab_tltparams_trans} in Appendix \ref{sec_appendix2}}
771: \label{tab_tltparams}
772: \end{center}
773: \end{table}
774:
775:
776: Events rejected by this level of algorithm are not excluded from final list
777: of alerts claimed by the system. They are only flagged, this flag can be
778: useful indication for a person inspecting all night events.
779: Results of TLT are saved to log file and database. Parameters used in TLT are listed in Table \ref{tab_tltparams}.
780:
781: \subsection{Optimization of algorithm parameters}
782: \label{sec_opt_of_algo}
783:
784: Optimization of algorithm parameters was a very important step of algorithm
785: development. Algorithm parameters can be changed by settings in the \texttt{ccd.cfg}
786: file as described in section \ref{sec_daq_cfg}.
787: Tables \ref{tab_fltparams} and \ref{tab_sltparams} list most important parameters of algorithms on
788: first and second levels of the trigger.
789:
790: Some parameters were optimized by specific studies and the others were
791: optimized by running algorithms on sky data with simulated optical flashes.
792: Testing of algorithms was performed on regular sky images.
793: Exactly the same software was used, but instead of reading images from the camera like during regular night observations,
794: the images were read from the \texttt{fits} files stored on a disk.
795: Images were analysed and found events were considered as background events.
796: Optical flashes were simulated in such a way
797: that samples of stars of given brightness were extracted from images and pasted in an image in
798: random positions. Single test of given parameters set was performed
799: on all images from a single night, program was analyzing subsequent images
800: and one generated flash was added in every image.
801: After all images were analysed the number of background events $N_{bkg}$
802: was calculated, as those which were found by the algorithm, but were not generated.
803: Generated events were checked on the list of identified events and
804: number $N_{genident}$ of those generated and identified was determined.
805: Efficiency of identifying optical flashes of given magnitude was determined
806: as : \\
807: \begin{equation}
808: \epsilon ( mag ) = \frac{N_{genident}(mag)}{N_{generated}(mag)} \\
809: \label{eq_eff_calc}
810: \end{equation}
811: Every set of tested parameters was plotted as a single point in the plot of $\epsilon$ vs $N_{bkg}$.
812: Different points on this plot show values determined for different settings
813: of the algorithm. For every star magnitudo separate plot was created.
814: Figures \ref{fig_eff_vs_bkg_9mag} and \ref{fig_eff_vs_bkg_10mag} show results of
815: efficiency and background rejection tests for algorithm requiring confirmation of flash in the next image
816: performed on $\approx$500 images from night 2007.04.25/26.
817: These plots were created for simulated flashes of $9^m$ and $10^m$ respectively. The best values of efficiency are also shown
818: in Table \ref{tab_best_eff_vs_bkg}.
819: According to these results optimal parameter values were chosen ( see Tables \ref{tab_fltparams} and \ref{tab_sltparams} ).
820:
821: % heplx43:/pi20/msok/praca/MY/badania/eff_vs_bkg/20070425/mag9
822: % plot_eff_bkg!
823: \begin{figure}[!htbp]
824: \begin{center}
825: \leavevmode
826: \ifpdf
827: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag9/laplace4/eff_bkg_9.00_lap4.gif}
828: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag9/laplace12/eff_bkg_9.00_lap12.gif}
829: \else
830: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag9/laplace4/eff_bkg_9.00_lap4.eps}
831: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag9/laplace12/eff_bkg_9.00_lap12.eps}
832: \fi
833: \caption{Efficiency of $9^m$ flash recognition and background rejection
834: by an algorithm requiring confirmation of event on next image.
835: Test was performed on data from night 2007-04-25/26 for laplace=4 (left plot) and laplace=12 (right plot)}
836: \label{fig_eff_vs_bkg_9mag}
837: \end{center}
838: \end{figure}
839:
840: % heplx48:/disk03/results/msok/simul/ccdsingle/20070425/FINAL_RESULTS/mag10/
841: % plot_eff_bkg! eff_bkg_10.00.txt 10
842: \begin{figure}[!htbp]
843: \begin{center}
844: \leavevmode
845: \ifpdf
846: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag10/eff_bkg_10.00_lap4.gif}
847: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag10/eff_bkg_10.00_lap12.gif}
848: \else
849: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag10/eff_bkg_10.00_lap4.eps}
850: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/mag10/eff_bkg_10.00_lap12.eps}
851: \fi
852: \caption{Efficiency of $10^m$ flash recognition and background rejection
853: by an algorithm requiring confirmation of event on next image.
854: Test was performed on data from night 2007-04-25/26 for laplace=4 (left plot) and laplace=12 (right plot)}
855: \label{fig_eff_vs_bkg_10mag}
856: \end{center}
857: \end{figure}
858:
859:
860:
861: % cd /pi20/msok/praca/MY/badania/eff_vs_bkg/20070425/mag9/laplace12
862: % show_best_effbkg! - 0.75
863: % wykonac : show_best_effbkg! eff_bkg_9.00_lap12.txt 0.72 50
864: \begin{table}[htbp]
865: \begin{center}
866: % \begin{tiny}
867: \begin{tabular}{|c|c|c|c|c|c|}
868: \hline
869: \textbf{Magnitude} & \textbf{Laplace} & \textbf{Tv} & \textbf{Tn} & \textbf{Eff} & \textbf{Bkg} \\
870: \hline
871: \hline
872: 9 & 4 & 2.0 & 4.0 & 0.71 & 21 \\
873: 9 & 4 & 2.0 & 5.0 & 0.71 & 4 \\
874: 9 & 4 & 2.0 & 6.0 & 0.70 & 0 \\
875: 9 & 4 & 2.5 & 4.0 & 0.74 & 209 \\
876: 9 & 4 & 2.5 & 5.0 & 0.74 & 4 \\
877: 9 & 4 & 2.5 & 6.0 & 0.73 & 2 \\
878: 9 & 4 & 3.0 & 5.0 & 0.75 & 18 \\
879: 9 & 4 & 3.0 & 6.0 & 0.74 & 3 \\
880: \hline
881: \hline
882: 9 & 12 & 2.0 & 5.0 & 0.71 & 2 \\
883: 9 & 12 & 2.0 & 6.0 & 0.71 & 0 \\
884: 9 & 12 & 2.5 & 5.0 & 0.74 & 5 \\
885: 9 & 12 & 2.5 & 5.0 & 0.74 & 0 \\
886: 9 & 12 & 3.0 & 5.0 & 0.75 & 43 \\
887: 9 & 12 & 3.0 & 6.0 & 0.75 & 2 \\
888: \hline
889: \end{tabular}
890: % \end{tiny}
891: \caption{Best values of efficiency of confirmation on next image algorithm obtained for simulated flashes of brightness 9$^m$, tested on 500 images from night 2007-04-25/26.}
892: \label{tab_best_eff_vs_bkg}
893: \end{center}
894: \end{table}
895:
896:
897: The efficiency and background tests were also performed for the coincidence
898: algorithm. The Figures \ref{fig_fig_eff_vs_bkg_coinc_9mag} show best points of $T_n$ and
899: $T_v$ thresholds.
900:
901: % heplx48:/disk03/results/msok/simul/ccddouble/20060526/FINAL_RESULTS/
902: % plot_eff_bkg!
903: % heplx40:http://grb.fuw.edu.pl/pi0/user/msok/doc/phd/badania/eff/ccddouble/mag9/20060526/
904: \begin{figure}[!htbp]
905: \begin{center}
906: \leavevmode
907: \ifpdf
908: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccddouble/mag9/laplace12/eff_bkg_9.00_lap12.gif}
909: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccddouble/mag9/laplace4/eff_bkg_9.00_lap4.gif}
910: \else
911: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccddouble/mag9/laplace12/eff_bkg_9.00_lap12.eps}
912: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccddouble/mag9/laplace4/eff_bkg_9.00_lap4.eps}
913: \fi
914: \caption{Results of efficiency of $9^m$ flash recognition and background rejection of coincidence algorithm. Tests were performed
915: on data from night 2006-05-26/27 for laplace=4 (left plot) and laplace=12 (right plot)}
916: \label{fig_fig_eff_vs_bkg_coinc_9mag}
917: \end{center}
918: \end{figure}
919:
920: % GENERAL :
921: % heplx48:/disk03/results/msok/simul/ccddouble/20060526/FINAL_RESULTS/
922: % heplx40:/var/www/html.orig/pi0/user/msok/doc/phd/badania/eff/ccddouble/mag9/20060526
923: % DETAILS :
924: % heplx48:/disk03/results/msok/simul/ccddouble/20060526/FINAL_RESULTS_DETAIL/
925: % nohup ./run_eff_bkg_ccddouble_detail! > out 2>&1 &
926: % http://grb.fuw.edu.pl/pi0/user/msok/doc/phd/badania/eff/ccddouble/mag9/20060526_det/
927: \begin{table}[htbp]
928: \begin{center}
929: % \begin{tiny}
930: \begin{tabular}{|c|c|c|c|c|c|}
931: \hline
932: \textbf{Magnitude} & \textbf{Laplace} & \textbf{Tv} & \textbf{Tn} & \textbf{Eff} & \textbf{Bkg} \\
933: \hline
934: \hline
935: 9 & 4 & 2.0 & 5.0 & 0.75 & 14 \\
936: 9 & 4 & 2.0 & 6.0 & 0.74 & 7 \\
937: 9 & 4 & 3.0 & 5.0 & 0.74 & 63 \\
938: 9 & 4 & 3.0 & 6.0 & 0.77 & 17 \\
939: \hline
940: \hline
941: 9 & 12 & 2.0 & 5.0 & 0.78 & 20 \\
942: 9 & 12 & 2.0 & 6.0 & 0.77 & 5 \\
943: 9 & 12 & 3.0 & 6.0 & 0.82 & 40 \\
944: 9 & 12 & 3.0 & 7.0 & 0.82 & 16 \\
945: \hline
946: \end{tabular}
947: % \end{tiny}
948: \caption{Best values of efficiency of coincidence algorithm obtained for simulated flashes of brightness 9$^m$, tested on 1952 images from night 2006-05-26/27.}
949: \label{tab_best_eff_vs_bkg}
950: \end{center}
951: \end{table}
952:
953: The efficiency losses due to subsequent cuts of on-line algorithm was tested.
954: It was done by counting how many of generated samples were rejected by
955: on-line algorithm cuts. Figure \ref{fig_efflosses} shows efficiency losses
956: due to subsequent cuts for data collected during single night.
957:
958: % heplx43:/pi20/msok/praca/MY/badania/online_eff/EffLoss/c
959: % skrypty : show_cuts_effloss_coinc! show_cuts_effloss_confirm!
960: % plot_effloss! - sciaga najswiezsze dane z pi2 i robi plot
961: \begin{figure}[!htbp]
962: \begin{center}
963: \leavevmode
964: \ifpdf
965: \includegraphics[width=2.8in,height=2.8in]{EffLoss/ccdsingle/20070707_confirm.gif}
966: \includegraphics[width=2.8in,height=2.8in]{EffLoss/ccddouble/20060527_coinc_k2a.gif}
967: \else
968: \includegraphics[width=2.8in,height=2.8in]{EffLoss/ccdsingle/20070707_confirm.eps}
969: \includegraphics[width=2.8in,height=2.8in]{EffLoss/ccddouble/20060527_coinc_k2a.eps}
970: \fi
971: \caption{Efficiency losses due to subsequent cuts of on-line algorithm : confirmation on next image (left plot), coincidence (right plot)}
972: \label{fig_efflosses}
973: \end{center}
974: \end{figure}
975:
976:
977: The efficiency of on-line algorithm cuts was determined for several
978: different nights ( Fig. \ref{fig_eff_vs_night} ). The mean efficiency is $\approx 70$\%.
979: The testing procedure which pasts samples of stars into image does not take
980: clouds into account. In fact this procedure allows to estimate efficiency
981: of all cuts after Tn cut. This is because sample is pasted in an image on
982: top of clouds, so there is no possibility to have large loss of efficiency
983: due to clouds in such kind of testing procedure.
984: The average efficiency of Tn cut was estimated as $\epsilon_{Tn}\approx$0.49 with usage of the TYCHO-2 star catalog
985: and cataloging procedure ( see Sec. \ref{sec_red_eff} ). This gives average
986: efficiency of flash identification algorithm $\epsilon_{algo}\approx$0.35.
987:
988: % heplx43:/pi20/msok/praca/MY/badania/online_eff/doit!
989: % zrobilem to na podstawie kilku nocy srednia eff :
990: % select avg(ns_online_eff) from nightstat where ns_online_eff is not null and ns_online_eff>0;
991: % wyszla ~70%
992: \begin{figure}[!htbp]
993: \begin{center}
994: \leavevmode
995: \ifpdf
996: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/EffVsNight/eff_vs_stars.gif}
997: \else
998: \includegraphics[width=2.8in,height=2.8in]{EffBkg/ccdsingle/EffVsNight/eff_vs_stars.eps}
999: \fi
1000: \caption{Acceptance of on-line algorithm cuts in function of average number of stars on all images collected during a single night. Each point represents efficiency during a single night.}
1001: \label{fig_eff_vs_night}
1002: \end{center}
1003: \end{figure}
1004:
1005:
1006: \subsection{Sources of background}
1007: Final list of events from one night of the prototype work did not exceed 100 events
1008: , but typically remained on the level of 10 events per night.
1009: These events were mainly due to background events. The main sources of
1010: the background were flashing satellites, planes and meteors.
1011: There was also background due to clouds, in this case usually number of
1012: events on cloudy images was big so it was easy to simply reject whole image.
1013: Summary of background events statistics is given in Table
1014: \ref{tab_online_background}. This table is divided into periods and
1015: types of algorithms in the cases where more then one algorithm was running.
1016: Example of background event images are given in Figures
1017: \ref{fig_bkg_cosmic_coinc}, \ref{fig_bkg_plane}, \ref{fig_bkg_meteor} and \ref{fig_bkg_satellites}.
1018:
1019: % heplx43:/pi20/msok/praca/MY/badania/tlo/
1020: % 2004_2005!
1021: % show_bkg! / show_bkg_stat!
1022: % runtype : 0 - coinc, 1 - sum8 , 3 - offline single, 4 - online single , 5 - offline coinc
1023: % new categories since :
1024: \begin{landscape}
1025: \begin{table}[htbp]
1026: \begin{center}
1027: \begin{tiny}
1028: \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
1029: \hline
1030: % \footnotemark
1031: % \footnote{Before 2006-10-01 Other was category for all later types in table}
1032: \textbf{Period} & \textbf{Algo} & \textbf{Flash} & \textbf{Satellite} & \textbf{\begin{minipage}{1.2cm}{Flash or satellite}\end{minipage}} & \textbf{Plane} & \textbf{\begin{minipage}{1.cm}{Satellite or plane}\end{minipage}} & \textbf{Other}\footnotemark & \textbf{Meteor} & \textbf{Clouds} & \textbf{Hot Pixel} & \textbf{\begin{minipage}{1.2cm}{Saturated Star}\end{minipage}} & \textbf{\begin{minipage}{1.2cm}{Opened Shutter}\end{minipage}} & \textbf{\begin{minipage}{1.cm}{System Error}\end{minipage}} \\
1033: \hline
1034: % Period | Algo | Flash | Sat | Flash or Sat | Plane | Sat or Plane | Other |
1035: % Meteor | Clouds | Hot Pixel | Saturated Star | Opened Shutter | Sys. Error |
1036: & & & & & & & & & & & & & \\
1037: \hline
1038: 2004.06.25 - 2005.01.20 & coinc. & 70 & 572 & 29 & 392 & 134 & 435 & - & - & - & - & - & - \\
1039: \hline
1040: 2004.06.25 - 2005.08.09 & coinc. & 125 & 961 & 53 & 819 & 455 & 1810 & - & - & - & - & - & - \\
1041: \hline
1042: 2005.04.21 - 2005.08.09 & conf. next & 23 & 31 & 8 & - & 2 & 3676 & - & - & - & - & - & - \\
1043: \hline
1044: 2006.05.20 - 2006.08.08 & coinc. & 42 & 352 & 10 & 262 & 324 & 1968 & - & - & - & - & - & - \\
1045: \hline
1046: 2006.05.20 - 2006.12.31 & conf. next & 3 & 118 & 2 & 6 & 12 & 4817 & 0 & 113995 & 168 & 12 & 60 & 64 \\
1047: \hline
1048: 2006.10.01 - 2006.12.31 & conf. next & 1 & 2 & 1 & - & 0 & 51 & 0 & 113995 & 168 & 12 & 60 & 64 \\
1049: \hline
1050: 2007.01.01 - 2007.05.29 & conf. next & 2 & 363 & 7 & - & 7 & 46725 & 0 & 34729 & 2151 & 298 & 329 & 64445 \\
1051: \hline
1052: \end{tabular}
1053: \end{tiny}
1054: \caption{Statistics of classification of events from on-line algorithm in period 2006.06 - 2007.06}
1055: \label{tab_online_background}
1056: \end{center}
1057: \end{table}
1058: \footnotetext{Before 2006-10-01 Other was category for all later types in table}
1059: \end{landscape}
1060:
1061:
1062: % /home/8/msok/doc/PHD/DataAnalysis/DataAnalysisFigs/bkg
1063: % + script : jpg2show! 1142_1059_01939.jpg "2005.05.02 07:08:30 UT" 15
1064: % ten skrypt wpisuje date + godzine ( trzeba ja podac )
1065: \begin{figure}[!htbp]
1066: \begin{center}
1067: \leavevmode
1068: \ifpdf
1069: \includegraphics[width=6in]{cosmic20040915_x2.gif}
1070: \else
1071: \includegraphics[width=6in]{cosmic20040915_x2.eps}
1072: \fi
1073: \caption{Rare example of coincidence of two cosmic ray hits}
1074: \label{fig_bkg_cosmic_coinc}
1075: \end{center}
1076: \end{figure}
1077:
1078: \begin{figure}[!htbp]
1079: \begin{center}
1080: \leavevmode
1081: \ifpdf
1082: \includegraphics[width=6in]{plane20040912.gif}
1083: \else
1084: \includegraphics[width=6in]{plane20040912.eps}
1085: \fi
1086: \caption{Plane-like background event}
1087: \label{fig_bkg_plane}
1088: \end{center}
1089: \end{figure}
1090:
1091: \newpage
1092: \begin{figure}[!htbp]
1093: \begin{center}
1094: \leavevmode
1095: \ifpdf
1096: \includegraphics[width=6in]{meteor20050501.gif}
1097: \else
1098: \includegraphics[width=6in]{meteor20050501.eps}
1099: \fi
1100: \caption{Meteor trace blown by the wind}
1101: \label{fig_bkg_meteor}
1102: \end{center}
1103: \end{figure}
1104:
1105: \begin{figure}[!htbp]
1106: \begin{center}
1107: \leavevmode
1108: \ifpdf
1109: \includegraphics[width=6in]{flotilla20060915.gif}
1110: \else
1111: \includegraphics[width=6in]{flotilla20060915.eps}
1112: \fi
1113: \caption{Flotilla of artificial satellites}
1114: \label{fig_bkg_satellites}
1115: \end{center}
1116: \end{figure}
1117:
1118:
1119: \subsection{Final verification of events}
1120: \label{sec_flashes_verification}
1121: The final list of accepted events is very small. However, the prototype does
1122: not have a possibility to definitely reject all background events.
1123: The most difficult part of the background are flashing satellites.
1124: They are mostly rejected by catalog and track cuts described in the previous
1125: sections, but some objects are not cataloged and flash too rarely to
1126: be rejected by these criteria.
1127: Final events can be evaluated by several checks. In the case when object suddenly
1128: appears and remains visible in the next several images in the same position
1129: it is very probable that it is not a satellite. Assuming the object is
1130: Earth's satellite the following formula can be derived : \\
1131:
1132: \begin{equation}
1133: R > \left( \frac{\sqrt{G \cdot M} \cdot \delta t }{\alpha} \right)^{(2/3)} \approx 16700 \cdot \left( \frac{\delta t_{sec}}{\alpha_{arcmin}} \right)^{(2/3)} [km]
1134: \label{many_pixels_contraint}
1135: \end{equation}
1136:
1137: where $\alpha$ is the angular distance of the object in consecutive 2 images and
1138: $\delta$t is the time separation of images. They can be substituted in
1139: arcmin and seconds respectively if formula on the right is used.
1140: In the case of the prototype $\alpha \approx 0.6'$ which corresponds to a single
1141: pixel and $\delta t \approx 12s$ which corresponds to time separation of 2
1142: subsequent images ( $T_{exposure} + T_{dead}$= 10s $+$ 2s ). The minimal
1143: distance of the object visible on 2 consecutive images in the same position
1144: derived from these values is D $\approx$ 123 000 km.
1145: For comparison, the geostationary orbit is $R_{geostat} \approx $42160 km
1146: ( from the Earth center ).
1147:
1148: % /pi20/msok/praca/MY/badania/sattest/20041212
1149: % sat_distance!
1150: % sat_distance_20041212.eps
1151: \begin{figure}[!htbp]
1152: \begin{center}
1153: \leavevmode
1154: \ifpdf
1155: \includegraphics[width=6in]{sat_distance_20041212.gif}
1156: \else
1157: \includegraphics[width=6in]{sat_distance_20041212.eps}
1158: \fi
1159: \caption{Distribution of distances of artificial satellites from the center of the Earth. Peak at $\approx$ 42000 km is due to geostationary satellites.}
1160: \label{fig_sat_distance_distr}
1161: \end{center}
1162: \end{figure}
1163:
1164:
1165: The distribution of the distance from the Earth to satellites in the catalog
1166: is shown in Figure \ref{fig_sat_distance_distr}. Peak from geostationary
1167: satellites is clearly visible. There are not many satellites
1168: more distant then 50000 km, which supports "double image" events.
1169: However, it is possible that these flashes are caused by spacecrafts on the
1170: long missions which are very far objects and can also reflect sun light
1171: towards the Earth. The probability of such events is very small.
1172: The above check can not be applied to events visible only in a single image.
1173: For this class of events another check was implemented. Flashing satellites
1174: can only reflect sun light when they are outside the Earth shadow cone and
1175: not on the illuminated side of the Earth (Fig. \ref{fig_earth_shadow}).
1176:
1177: \begin{figure}[!htbp]
1178: \begin{center}
1179: \leavevmode
1180: \ifpdf
1181: % width=2.7in,height=2.7in
1182: \includegraphics[width=6in]{satcone.gif}
1183: \includegraphics[width=6in]{earth_shadow.gif}
1184: \else
1185: \includegraphics[width=6in]{satcone.eps}
1186: \includegraphics[width=6in]{earth_shadow.eps}
1187: \fi
1188: \caption{Positions where satellites can reflect sun light}
1189: \label{fig_earth_shadow}
1190: \end{center}
1191: \end{figure}
1192:
1193:
1194: % WZOR : CSatCone::calc_min_dist - ale jest tak skaplikowany ze go tu nie zamieszczam !!!
1195: Knowing the position and time of the flash it is possible to calculate the
1196: satellite's minimal distance from the Earth to be outside of the shadow
1197: cone which is required for the satellite to cause a flash-like event.
1198: The Earth shadow cone size is $H_{cone} \approx$ 1,37 mln km ( Fig. \ref{fig_earth_shadow} ), which is more then
1199: Moon's orbit size ( $R_{moon}$ $\approx$ 400000 km). In case any of the flash candidates would have $D_{cone} > R_{moon}$
1200: it would be most probably an event of astrophysical origin.
1201:
1202:
1203: \section{Off-line data analysis}
1204: Off-line data analysis acts on data reduced and cataloged to the database.
1205: The reduction chain consists of several steps which will be described in
1206: detail in the next subsection. After this chain, star brightness measurements
1207: are stored in the database which is optimized for fast access.
1208: The structure of the database will be described in subsection \ref{sec_star_catalog}.
1209: In the last subsection algorithms for detecting brightness increases and
1210: new object appearance will be described.
1211:
1212: \subsection{Reduction pipeline}
1213: The aim of the reduction procedure is to reduce raw data stored as images
1214: in \texttt{fits} files into essential data describing stars coordinates and brightness.
1215: This allows to reduce the amount of data by factor of $\approx 10$ in the case of
1216: single images and $\approx 100$ in the case of 20 averaged images.
1217:
1218: \subsubsection{Image reduction}
1219: \label{sec_image_reduction}
1220:
1221: Every image collected during a night is processed in the same way.
1222: Off-line data analysis described in this thesis was performed on data obtained by
1223: averaging 20 subsequent images. There are also other reduction pipelines
1224: for reducing single image and scan images ( averaging 3 scan images ), but
1225: almost all of the steps are the same. The main difference is that in pipelines
1226: acting on averaged images there is an additional step calculating
1227: average of specified number of images.
1228: Image reduction consists of the following steps :
1229: \begin{itemize}
1230: \item image averaging - it is present in reduction pipelines acting on
1231: averaged images. In the case of \texttt{aver20} pipeline 20 subsequent images are averaged
1232: and in the case of pipeline \texttt{scan3} 3 subsequent images are averaged.
1233: Image coordinates are controlled and in case they change, average chain
1234: is stopped not to allow for averaging images from different positions.
1235: In the case of single image reduction the image averaging step is skipped.
1236: \item Dark frame subtraction - the reason for dark frame subtraction was already
1237: described in Section \ref{sec_online_reduction}.
1238: As described in Section \ref{system_nightly_performance} in order to reduce
1239: fluctuations the dark image is calculated as a median of several dark images.
1240: This step allows to subtract signal offset produced by dark current and electronics.
1241: It also reduces the effect of hot pixels.
1242: \item Division by flat image - this step allows to correct for
1243: non-uniformity of the optics and differences between pixels amplifications.
1244: Standard way of finding this correction is taking images of
1245: uniformly illuminated field. It is usually the sky just after dusk or just
1246: before dawn, when the sky is bright and stars are not visible.
1247: An alternative way is to use uniformly illuminated screen. In case of wide
1248: field observations it is very difficult to obtain proper flat image.
1249: It is due to difficulty of obtaining uniformly illuminated field of size
1250: of FOV $\ge 20^\circ$. The evening sky just after sunset is uniformly illuminated in
1251: scale of arc minutes, but in scale of degrees non uniformities due to sky
1252: gradient are significant. Due to this problem flat image is obtained by taking
1253: images of evening sky with the mount tracking switched off. After taking many images and
1254: calculating median image stars are eliminated, finally the image is normalized
1255: to one. This procedure requires collection of many ($\geq$200) images so it is
1256: performed rarely and for most of the time the same flat image is used in
1257: analysis.
1258: \end{itemize}
1259:
1260: After the above operations the image is ready for photometry. The photometry
1261: is a procedure which finds stars in the image and determines their chip coordinates
1262: (x,y) and brightness.
1263: In the "Pi of the Sky" data analysis two photometry procedures are
1264: used depending on the type of reduction pipeline :
1265: \begin{itemize}
1266: \item ASAS photometry - aperture photometry adopted from ASAS experiment \cite{ASAS}.
1267: It is rather slow so cannot be used for reduction of all single
1268: images from a night. It is used for photometry of 20 averaged images ( $\approx 20 \cdot 12 = 240$ sec timescale )
1269: and in reduction of scan images ( 3 images averaged ).
1270: \item Fast photometry - it is fast, aperture photometry algorithm. Simple
1271: aperture is used to calculate star brightness. This photometry is used
1272: on-line by \texttt{DAQ} to perform astrometry every 300 sec and in reduction of all
1273: night images ( ASAS photometry is too slow for this purpose )
1274:
1275: \begin{figure}[!htbp]
1276: \begin{center}
1277: \leavevmode
1278: \ifpdf
1279: \includegraphics[width=4in,height=4in]{apert9x9+9.gif}
1280: \else
1281: \includegraphics[width=4in,height=4in]{apert9x9+9.eps}
1282: \fi
1283: \caption{Aperture used in fast photometry algorithm}
1284: \label{fig_fast_aperture}
1285: \end{center}
1286: \end{figure}
1287:
1288: Aperture used for brightness calculation is shown in Figure \ref{fig_fast_aperture}.
1289: Final brightness of star is determined as :\\
1290: \begin{equation}
1291: I = \sum_{i}^{N_+} P_{+} - N_{+} \cdot B_{sky}
1292: \label{eq_piphoto_mag}
1293: \end{equation}
1294: where $B_{sky}$ stands for average value of sky background near analysed
1295: star. This value is obtained as median value of pixels in gray ring around
1296: the star ( Fig. \ref{fig_fast_aperture} ). $P_{+}$ denotes signal pixels, shown as
1297: dark gray pixels inside the star, they form 3x3 square around the maximum
1298: brightness pixel with 3 most bright pixels contiguous to the square sides.
1299: This procedure is not good for dense fields where sky background can be
1300: calculated incorrectly due to stars entering the "background ring" and also
1301: $\sum_{i}^{N_+} P_{+}$ is affected by overlapping stars.
1302: Instrumental magnitude is calculated from the formula : \\
1303: \begin{equation}
1304: m_{\pi} = - 2,5 log(I)
1305: \label{eq_mag_def}
1306: \end{equation}
1307: Star coordinates (x,y) are determined as centroid of cluster of pixels according
1308: to the formula : \\
1309: \begin{equation}
1310: % \begin{split}
1311: X_{star} = \frac{\sum_{cluster} x_i \cdot V_i}{\sum_{cluster} V_i} \hspace{2mm},\hspace{2mm}
1312: Y_{star} = \frac{\sum_{cluster} y_i \cdot V_i}{\sum_{cluster} V_i}
1313: % \end{split}
1314: \label{eq_star_center}
1315: \end{equation}
1316: Cluster of pixels is determined as pixels around the star which satisfy
1317: P(x,y) > $T_{cluster} = 3.5 \cdot \sigma$.
1318: \end{itemize}
1319:
1320:
1321: Both photometry procedures write resulting list of stars with (x,y)
1322: coordinates and magnitudes to output \texttt{mag} files. The format of this
1323: files is similar to \texttt{fits} format. They consist of header which is taken from
1324: \texttt{fits} header with some additional fields added, after the header section,
1325: list of stars in binary format is written. The \texttt{mag} files are input files for astrometry procedure. \\
1326: This procedure finds transformation T:(x,y) $\rightarrow (\lambda,\delta)$, the
1327: transformation T is described by the following formula :\\
1328:
1329: % a moze w postaci macierzowej to zapisac - see zeszyt
1330: \begin{equation}
1331: \lambda = \sum_{i,j \leq O} P_{ij} \cdot x^i \cdot y^i \hspace{2mm},\hspace{2mm}
1332: \delta = \sum_{i,j \leq O} Q_{ij} \cdot x^i \cdot y^i
1333: \label{eq_astro_transform}
1334: \end{equation}
1335:
1336: Where O is the order of the transformation, in current configuration O=4
1337: and due to this fact coefficients $P_{14},P_{23},P_{24},P_{32},P_{33},P_{34},P_{41},P_{42},P_{43},P_{44}=0$
1338: and also corresponding Q coefficients are zero.
1339: It allows to calculate equatorial coordinates for any chip coordinate (x,y).
1340: Astrometry requires input information about image center position,
1341: pixscale \footnote{pixscale is an angular size of the CCD pixel} and rotation angle of the image with respect to the celestial
1342: coordinates, these settings are read from header of \texttt{mag} file.
1343: The astrometry procedure was adopted from the ASAS experiment, it is an
1344: iteration procedure where stars in \texttt{mag} file are matched against catalog
1345: stars in given position in the sky. Star catalog currently used in the
1346: procedure is based on TYCHO catalog, however any star catalog can be used
1347: instead. The procedure consists of the following main steps : \\
1348:
1349: \begin{itemize}
1350: \item loading of \texttt{mag} file
1351: \item read stars from the reference star catalog
1352: \item estimation of shift from the expected position $(\lambda,\delta)_{mount}$
1353: and real position $(\lambda,\delta)$ using the correlation function between
1354: image stars and stars in the catalog
1355: \end{itemize}
1356:
1357: After the above initialization steps the iterational procedure begins, every
1358: iteration consists of the following main steps :\\
1359:
1360: \begin{itemize}
1361: \item recalculation of reference and image stars coordinates to standard coordinates (
1362: with respect to image center )
1363: \item matching of image stars from \texttt{mag} file against reference stars
1364: \item determination of transformation parameters by using the method of
1365: singular value decomposition (SVD).
1366: \item check the convergence condition requiring astrometry error $\delta \alpha < \delta \alpha_{MAX}$
1367: \item recalculation of image center coordinates $(\lambda,\delta)_{center}$
1368: \end{itemize}
1369:
1370: The iteration steps are repeated until the convergence conditions are satisfied.
1371: In the case astrometry procedure converges, for all stars in the \texttt{mag}
1372: file coordinates $(\lambda,\delta)$ are calculated from the formula \ref{eq_astro_transform}.
1373: The results are saved to \texttt{ast} file which consists of same information as \texttt{mag}
1374: file with additional fields for $(\lambda,\delta)$.
1375: All night images are processed in the same way and every sky image ( \texttt{fits}
1376: file ) has a corresponding \texttt{ast} file.
1377:
1378: \subsubsection{Star Catalog}
1379: \label{sec_star_catalog}
1380:
1381: The star catalog is developed as relational PostgreSQL database \cite{POSTGRES}.
1382: The database structure is shown in Figure \ref{fig_starcat}.
1383: It consists of tables described earlier in Section \ref{online_db_structure} with
1384: additional tables :
1385: \begin{itemize}
1386: \item \textbf{Star} - this table contains all objects observed by given camera.
1387: Same physical star can be observed by many cameras so same physical star
1388: has N$\geq$1 records in the table \textbf{Star}, where N stands for number of cameras by
1389: which this star was observed.
1390: \item \textbf{Measurements} - this table stores information on every observation of
1391: the star. It is linked to table \textbf{Star} by reference field \texttt{star}, it is also linked
1392: by field \texttt{id\_frm} to \textbf{Frame} on which the star was observed.
1393: \item \textbf{SuperStar} - it is a table containing real physical stars. In case
1394: star is observed by different cameras it has multiple records in table \textbf{Star},
1395: but only one record in the table \textbf{SuperStar}. Every record in the table \textbf{Star} is linked
1396: to corresponding \textbf{SuperStar} record by field \texttt{sstar\_id}. The relation between
1397: \textbf{SuperStar} and \textbf{Star} table is one to many.
1398: \item \textbf{ObsFieldStat} - statistical table containing information on number
1399: of images collected for a specific field
1400: \item \textbf{Field\_Def} - definitions of fields observed by the system
1401: \end{itemize}
1402:
1403: \begin{figure}[!htbp]
1404: \begin{center}
1405: \leavevmode
1406: \ifpdf
1407: \includegraphics[width=6in,height=8.5in]{star_catalog_all.gif}
1408: \else
1409: \includegraphics[width=6in,height=8.5in]{star_catalog_all.eps}
1410: \fi
1411: \caption{Structure of star catalog database}
1412: \label{fig_starcat}
1413: \end{center}
1414: \end{figure}
1415:
1416: Star catalog database can be huge, after year of data collection it can reach 50-200 GB ( \texttt{aver20} database ), so in order to
1417: be efficiently used it must be optimized. An important element of the database structure are indexes.
1418: They allow searching indexed fields by fast binary search algorithm.
1419: The most important database queries were optimized by creating indexes on fields used in conditional statements.
1420: Another optimization performed on the database is placing \textbf{Measurements}
1421: records for a given star in the same physical location
1422: on the disk. This is very important for fast reading of star light curves.
1423: This optimization is executed by PostgreSQL command CLUSTER which must be
1424: called from time to time after large amount of new data is added to catalog.
1425: % called every time new data is added to the database.
1426: Also table \textbf{Star} is optimized by CLUSTER command according to the celestial
1427: coordinates in order to optimize reading of stars from database during cataloging.
1428: The most important indexes are listed in Table \ref{tab_indexes_in_db}.
1429: There are also database optimizations on PostgreSQL server configuration
1430: level which set parameter value proper for huge database.
1431:
1432: \begin{table}[htbp]
1433: \begin{center}
1434: % \begin{tiny}
1435: \begin{tabular}{|c|c|c|c|}
1436: \hline
1437: \textbf{Index Name} & \textbf{Table} & \textbf{Indexed Fields} & \textbf{Clustered} \\
1438: \hline
1439: stars\_id\_index & STARS & ID & no \\
1440: stars\_ra\_dec\_index & STARS & ra,dec & yes \\
1441: stars\_dec\_index & STARS & dec & no \\
1442: superstar\_gcvs\_id\_index & SUPERSTAR & gcvs\_id & no \\
1443: measurements\_star\_index & MEASUREMENTS & star & yes \\
1444: measurements\_id\_frm\_index & MEASUREMENTS & id\_frm & no \\
1445: \hline
1446: \end{tabular}
1447: % \end{tiny}
1448: \caption{Indexes most important for optimizing star catalog}
1449: \label{tab_indexes_in_db}
1450: \end{center}
1451: \end{table}
1452:
1453:
1454: \subsubsection{Cataloging procedure}
1455: \label{sec_cataloging}
1456:
1457: Reduced data is a set of \texttt{ast} files, these files have to be loaded to the
1458: database structure described in the previous section. This task is called
1459: cataloging and is performed by \texttt{piaddast2} program.
1460: Except of loading data to the database this program normalizes star
1461: magnitudes according to V filter magnitudes in catalog of reference stars.
1462: The block diagram of the \texttt{piaddast2} program is shown in Figure
1463: \ref{fig_piaddast2}. Generally, cataloging procedure reads stars already
1464: observed before from the database and matches new observations to these stars.
1465: Subsequent \texttt{ast} files are organized in the memory until observation field
1466: changes which triggers dump of data from memory to the database and
1467: selection of stars for new position. Every \texttt{ast} file is processed in the following way :
1468:
1469: \begin{figure}[!htbp]
1470: \begin{center}
1471: \leavevmode
1472: \ifpdf
1473: \includegraphics[width=6in,height=8.5in]{piaddast2.gif}
1474: \else
1475: \includegraphics[width=6in,height=8.5in]{piaddast2.eps}
1476: \fi
1477: \caption{Block diagram of the cataloging program}
1478: \label{fig_piaddast2}
1479: \end{center}
1480: \end{figure}
1481:
1482: % /pi20/msok/praca/MY/badania/cat_cuts/
1483: \begin{itemize}
1484: \item Read next \texttt{ast} file, determine range of celestial coordinates in \texttt{ast} file
1485: $(\lambda_{min}^{ast},\delta_{min}^{ast})-(\lambda_{max}^{ast},\delta_{max}^{ast})$
1486: \item Save image header information to database tables \textbf{FRAME} and \textbf{FRAME\_DET}.
1487: \item Check if all stars in \texttt{ast} file have coordinates in the range $(\lambda_{min}^{prev},\delta_{min}^{prev})-(\lambda_{max}^{prev},\delta_{max}^{prev})$.
1488: In case there are stars from outside this range program dumps stars and measurements stored in memory to \texttt{sql}
1489: files and loads \texttt{sql} files to the database, then selects stars from database with coordinates matching
1490: range of coordinates in new \texttt{ast} file ( as determined in first step )
1491: \item Check if the image represented by the \texttt{ast} file is good enough to be cataloged, the following criteria are
1492: checked :\\
1493: \begin{itemize}
1494: \item number of images used for averaging ( in the case of cataloging averaged images )
1495: for 20 averaged images the lower limit is 10 images and for \texttt{scan3} pipeline lower limit is 3. \\
1496: \item check if $N_{star}^{image} \geq N_{star}^{MIN} ( = 5000 ) $ , in
1497: case number of stars is lower it means that it is probably cloudy image ( Fig. \ref{fig_image_stars} ) .
1498: \item check if number of stars is not too high : $N_{star}^{image} \leq N_{star}^{MAX} ( = 70000 ) $ ( Fig. \ref{fig_image_stars} ), which indicates bad image.
1499: \item check if average astrometry error is not too large : $A_{err} \leq A_{err}^{max} ( = 0.3 )$ ( Fig. \ref{fig_avg_ast_err} )
1500: \end{itemize}
1501:
1502: % WWW : http://grb.fuw.edu.pl/pi0/user/msok/doc/phd/badania/cat_cuts/
1503: % heplx43 : /pi20/msok/praca/MY/badania/cat_cuts/istarcount
1504: % script : show_star_count!
1505: \begin{figure}[!htbp]
1506: \begin{center}
1507: \leavevmode
1508: \ifpdf
1509: \includegraphics[width=4in,height=4in]{cat_cuts/image_stars.gif}
1510: % \includegraphics[width=2.7in,height=2.7in]{cat_cuts/matched_to_allstars.gif}
1511:
1512: % /pi20/msok/praca/MY/badania/cat_cuts/istarcount/20070828 :
1513: \includegraphics[width=4in,height=4in]{cat_cuts/ratio_vs_nstars.gif}
1514: \else
1515: \includegraphics[width=4in,height=4in]{cat_cuts/image_stars.eps}
1516: % \includegraphics[width=2.7in,height=2.7in]{cat_cuts/matched_to_allstars.eps}
1517:
1518: % /pi20/msok/praca/MY/badania/cat_cuts/istarcount/20070828 :
1519: \includegraphics[width=4in,height=4in]{cat_cuts/ratio_vs_nstars.eps}
1520: \fi
1521: \caption{Distribution of number of stars in an image (upper plot) and distribution of ratio of number of stars in an image to number of catalog stars in the observed field (lower plot)}
1522: \label{fig_image_stars}
1523: \end{center}
1524: \end{figure}
1525:
1526: \begin{figure}[!htbp]
1527: \begin{center}
1528: \leavevmode
1529: \ifpdf
1530: \includegraphics[width=4in,height=4in]{cat_cuts/ast_err_log.gif}
1531: \else
1532: \includegraphics[width=4in,height=4in]{cat_cuts/ast_err_log.eps}
1533: \fi
1534: \caption{Distribution of average astrometry error in an image}
1535: \label{fig_avg_ast_err}
1536: \end{center}
1537: \end{figure}
1538:
1539:
1540:
1541:
1542: In case all criteria are satisfied \texttt{ast} file is accepted and stars are
1543: cataloged, otherwise it is skipped.
1544: \item Each star in ast file is examined against certain quality cuts :\\
1545: \begin{enumerate}
1546: \item Star altitude is required to by h>$h_{min}$=15$^\circ$, in order to
1547: reject measurements close to horizon which are of very poor quality
1548: \end{enumerate}
1549: \item Star magnitudes are normalized by comparing with the catalog of reference
1550: stars. Matching allows to create correction image ( Fig. \ref{fig_corrimage} )
1551: which can be used to normalize magnitudes of all stars in the image.
1552: The normalization is performed in the following way :\\
1553:
1554: \begin{itemize}
1555: \item catalog of reference stars is read to memory
1556: \item for each star in \texttt{ast} file program finds a corresponding star in
1557: the reference catalog. In case it is found the field $mag_{cat}$ is filled with
1558: the magnitude of the star from the reference catalog.
1559: Correction for this star is calculated as :\\
1560: \begin{equation}
1561: \delta_{mag} = mag_{cat} - mag_{\pi}
1562: \label{eq_corr_value}
1563: \end{equation}
1564: this is value which must be added to instrumental magnitude to obtain
1565: normalized one. Typically about 50\% of \texttt{ast} stars have corresponding
1566: star in the currently used reference catalog (TYCHO).
1567: \item after matching procedure, corrections values are known only for pixels
1568: where stars matched to catalog stars are present.
1569: In order to calculate correction for each star in the image,
1570: correction values are calculated for all pixels of the image by
1571: extrapolating values determined for reference stars.
1572: Correction value C(x,y) for each pixel in the image is calculated as average
1573: of corrections for nearby reference stars weighted by distance to pixel where
1574: this star was observed :
1575:
1576: \begin{equation}
1577: C(x,y) = \sum_{ref. stars \hspace{1mm} : R<R_{max}} w( \sqrt{ (x-x_i)^2+(y-y_i)^2 } ) \cdot C_(x_i,y_i)
1578: \label{eq_corr_image_formula}
1579: \end{equation}
1580:
1581: An example of resulting correction image is shown in Figure \ref{fig_corrimage}.
1582: \item normalized magnitude is calculated for every star in the image according to formula :\\
1583: \begin{equation}
1584: mag_{norm} = mag_{cat} + C(x,y)
1585: \label{eq_mag_norm}
1586: \end{equation}
1587: where (x,y) are chip coordinates of the normalized star.
1588: \end{itemize}
1589:
1590: % /pi20/msok/data/20070302/cat/corr/
1591: % cd heplx49:/disk03/results/msok/astro_root
1592: % astroroot
1593: % .x image_display2.C
1594: % eps is gif converted by : convert 20070302_corr_smooth_18.gif 20070302_corr_smooth_18.eps
1595: \begin{figure}[!htbp]
1596: \begin{center}
1597: \leavevmode
1598: \ifpdf
1599: \includegraphics[width=6in]{20070302_corr_smooth_18.gif}
1600: \else
1601: \includegraphics[width=6in]{20070302_corr_smooth_18.eps}
1602: \fi
1603: \caption{Correction image obtained in cataloging to normalize instrumental magnitudes to magnitudes of stars in the reference catalog.}
1604: \label{fig_corrimage}
1605: \end{center}
1606: \end{figure}
1607:
1608:
1609: \item Stars from the \texttt{ast} file are matched to stars read from the database ( see step 3 ) and kept in the memory.
1610: In case given star is found on the list of stars kept in the memory, the star measurement from current \texttt{ast} file is linked to the list of
1611: measurements of this star. In case this star was not yet observed, it is
1612: added to the list of stars in the memory and flagged as a new object in the catalog.
1613: After all stars in \texttt{ast} file are processed next \texttt{ast} file is read and process starts from
1614: the beginning ( step 1 ). The procedure continues until all \texttt{ast} files are processed.
1615: \end{itemize}
1616:
1617: After the above steps all good quality data in \texttt{ast} files is saved to
1618: the database. However, there are several technical details which should be
1619: mentioned here. First of all, as can be seen in Figure \ref{fig_starcat}
1620: the \textbf{Star} table has statistical fields like magnitude, dispersion of magnitudo (\texttt{sigma\_mag}), \texttt{no\_measurements}
1621: which are not updated during data loading, but they are very useful in further
1622: data analysis. These fields are recalculated by
1623: \texttt{pg/sql} procedure \texttt{ReCalcNight} after the data is loaded to the database.
1624: The second step is optimization of the database, before loading of
1625: night \texttt{ast} files most of the indexes on tables \textbf{Star} and \textbf{Measurements} must be
1626: dropped in order to load data efficiently. After loading is finished these
1627: indexes are re-created. After loading new data its location on the disk must be
1628: re-organized in order to provide fast access. It is realized by PostgreSQL
1629: command \texttt{CLUSTER}.
1630: After all optimizations and recalculations are finished the database is unlocked and
1631: off-line algorithms can be executed on new night data.
1632:
1633: \subsubsection{Efficiency, purity and precision of observations}
1634: \label{sec_red_eff}
1635:
1636: Efficiency and purity of reduction and cataloging was tested by the
1637: following procedure :\\
1638: \begin{enumerate}
1639: \item Initialization of database - star catalog database was initialized
1640: with stars from TYCHO-2 star catalog
1641: \item Initial stars were flagged as TYCHO-2 stars in the database
1642: \item Test data was loaded to catalog initialized with \mbox{TYCHO-2} stars
1643: \end{enumerate}
1644:
1645: After this steps database was filled with data and there was an easy way to
1646: distinguish stars which are present in TYCHO-2 star catalog from those which
1647: are not and were observed only in "Pi of the Sky" images.
1648: The test was performed on two samples of images one from night 2007-04-25/26
1649: taken with shutter in normal mode and second sample were images from night
1650: 2007-05-12/13 taken with shutter permanently opened.
1651: Total star identification efficiency which is defined as : \\
1652:
1653: \begin{equation}
1654: \epsilon_{\pi-red} = \frac{N_{TYCHO2/PI}}{N_{TYCHO2}}
1655: % \epsilon_{red} =
1656: \label{eq_red_eff}
1657: \end{equation}
1658:
1659: where $N_{TYCHO2/PI}$ stands for number of TYCHO-2 stars identified by "Pi
1660: of the Sky" photometry and $N_{TYCHO2}$ is number of TYCHO-2 stars in the
1661: observed field.
1662: The Table \ref{tab_red_eff_total} shows total efficiencies for single image
1663: and set of images.
1664:
1665: % heplx48
1666: % /disk03/results/msok/piphoto_efficiency/results/20070425/NEW/eff_total
1667: % /disk03/results/msok/piphoto_efficiency/results/20070512/NEW/eff_total/NEW
1668: % /disk03/results/msok/piphoto_efficiency/results/20070512/NEW/eff_total/
1669: \begin{table}[htbp]
1670: \begin{center}
1671: % \begin{tiny}
1672: \begin{tabular}{|c|c|c|c|}
1673: \hline
1674: \textbf{Night} & \textbf{Number of images} & \textbf{Magnitude range} & \textbf{Efficiency} \\
1675: \hline
1676: 2007.04.25/26 & 1 & 0 - 12 & 0.74 \\
1677: 2007.04.25/26 & 41 & 0 - 12 & 0.84 \\
1678: 2007.04.25/26 & 41 & 0 - 15 & 0.77 \\
1679: \hline
1680: 2007.05.12/13 & 1 & 0 - 12 & 0.78 \\
1681: 2007.05.12/13 & 41 & 0 - 12 & 0.87 \\
1682: 2007.05.12/13 & 41 & 0 - 15 & 0.82 \\
1683: \hline
1684: \end{tabular}
1685: % \end{tiny}
1686: \caption{Total efficiencies for night 2007.04.25/26 ( data collected with
1687: shutter in normal mode ) and 2007.05.12/13 ( data collected with permanently
1688: opened shutter ) }
1689: \label{tab_red_eff_total}
1690: \end{center}
1691: \end{table}
1692:
1693:
1694: The Figure \ref{fig_eff_vs_mag} shows the efficiency of
1695: star identification in function of star brightness for data collected with
1696: permanently opened shutter ( 2007.05.12 ) and shutter in normal open/close
1697: mode ( 2007.04.25 ).
1698:
1699: % /pi20/msok/praca/MY/badania/photo_eff/eff_vs_mag/eff_vs_mag
1700: \begin{figure}[!htbp]
1701: \begin{center}
1702: \leavevmode
1703: \ifpdf
1704: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_mag/20070425_shutter_normal.gif}
1705: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_mag/20070512_shutter_open.gif}
1706: \else
1707: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_mag/20070425_shutter_normal.eps}
1708: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_mag/20070512_shutter_open.eps}
1709: \fi
1710: \caption{Efficiency of TYCHO-2 stars identification in function of magnitude for data collected during night 2007.04.25/26 with shutter in normal (open/close) mode (left plot ) and data collected with shutter permanently opened during night 2007.05.12/13 ( right plot ) }
1711: \label{fig_eff_vs_mag}
1712: \end{center}
1713: \end{figure}
1714:
1715: The Figure \ref{fig_eff_vs_image_many_nights}
1716: shows efficiency in function of number of field observations. It can be seen
1717: that at least 10 images of field must be catalogued before running nova
1718: identification algorithm and the safe limit is 20 exposures. This ensures
1719: that nova candidates will not be mainly due to normal faint stars added to catalog
1720: for the first time. The best way would be to initialize star catalog with
1721: all objects up to 12-13$^m$, which is planned to be done.
1722:
1723: % heplx43:/pi20/msok/praca/MY/badania/photo_eff/eff_vs_image
1724: % .x plotfile_multi.C("all.txt",0,1)
1725: \begin{figure}[!htbp]
1726: \begin{center}
1727: \leavevmode
1728: \ifpdf
1729: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_image/eff_vs_image_many_nights.gif}
1730: \else
1731: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_image/eff_vs_image_many_nights.eps}
1732: \fi
1733: \caption{Average efficiency ( stars 0-12$^m$ ) of TYCHO-2 stars identification in function of
1734: number of field observations for 3 different nights and after applying
1735: correction of shutter opened effect.}
1736: \label{fig_eff_vs_image_many_nights}
1737: \end{center}
1738: \end{figure}
1739:
1740:
1741:
1742: The Figures \ref{fig_eff_vs_xy} and \ref{fig_bkg_vs_xy} show efficiency and background in function of chip coordinates.
1743: It is clear that efficiency and background drop in the corners of the CCD chip and reach the
1744: highest values in the center of the chip which is due to the fact that
1745: less light reaches corners of CCD because of properties of the optical system.
1746:
1747: \begin{figure}[!htbp]
1748: \begin{center}
1749: \leavevmode
1750: \ifpdf
1751: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070425_shutter_normal_eff.gif}
1752: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070512_shutter_open_eff.gif}
1753: \else
1754: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070425_shutter_normal_eff.eps}
1755: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070512_shutter_open_eff.eps}
1756: \fi
1757: \caption{Efficiency of TYCHO-2 stars identification in function of star position (x,y) on the CCD chip.
1758: Left plot shows efficiency of single average of 20 images from
1759: night 2007.04.25/26 collected with shutter in normal mode and right plot
1760: shows efficiency on single image from night 2007.05.12/13 collected with permanently opened shutter}
1761: \label{fig_eff_vs_xy}
1762: \end{center}
1763: \end{figure}
1764:
1765: \begin{figure}[!htbp]
1766: \begin{center}
1767: \leavevmode
1768: \ifpdf
1769: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070425_shutter_normal_bkg.gif}
1770: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070512_shutter_open_bkg.gif}
1771: \else
1772: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070425_shutter_normal_bkg.eps}
1773: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/eff_vs_xy/20070512_shutter_open_bkg.eps}
1774: \fi
1775: \caption{Number of observed objects not existing in TYCHO-2 catalog in function of star position (x,y) on the CCD chip.
1776: Left plot shows objects found on single average of 20 images from
1777: night 2007.04.25/26 collected with shutter in normal mode and right plot
1778: shows objects found on single image from night 2007.05.12/13 collected with permanently opened shutter}
1779: \label{fig_bkg_vs_xy}
1780: \end{center}
1781: \end{figure}
1782:
1783:
1784:
1785: The situation is slightly more difficult in the case of purity
1786: determination. The objects in the TYCHO-2 star catalog are only stars.
1787: Objects observed by "Pi of the Sky" which are not present in the TYCHO-2
1788: catalog are not only background events. They can be non-star objects like
1789: galaxies etc. However, after many observations of given field all objects in
1790: the range of the telescope should already be observed and new objects
1791: appearing in the catalog after many field observations can be considered as
1792: background ( assuming signal events are very seldom ).
1793: The Figure \ref{fig_new_vs_imageno_20070425} shows number of new objects
1794: added to star catalog on subsequent images of the same field and also total
1795: number of new objects in function of number of field observations is shown.
1796: The data presented on this plot was collected during night 2007.04.25/26 with
1797: shutter in normal open/close mode. The steps at frame id=40 and id=34 are
1798: due to trace of satellite ( or plane ) which is shown in Figure \ref{fig_red_bkg_plane}.
1799: It is clear that after many ( >40 ) field observations number of new objects
1800: added to star catalog on every image is very small and equals few events per
1801: image, unless background event ( like satellite, plane etc ) is observed.
1802: For comparison the same plots for data collected on the same field with shutter permanently
1803: opened ( night 2007.05.12/13 ) is shown in Figure \ref{fig_new_vs_imageno_20070512}.
1804: The conditions during both night were similar, the number of objects added to
1805: star catalog was much higher during night when data was collected with permanently
1806: opened shutter.
1807:
1808: Figure \ref{fig_bkg_vs_image_many_nights} shows cumulative number of events added to
1809: catalog in function of image number for couple of fields observed during
1810: many nights. These plots indicate that the minimum number of field observations
1811: to consider new objects as potential nova candidate is 20-30.
1812:
1813: % heplx48:/disk03/results/msok/piphoto_efficiency/results/20070425/NEW/bkg_vs_image_no
1814: % doit2!
1815: % heplx40:/lhome/piwww/www/pi0/user/msok/doc/phd/badania/photo_eff/20070425/bkg_vs_image_no
1816: \begin{figure}[!htbp]
1817: \begin{center}
1818: \leavevmode
1819: \ifpdf
1820: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070425/new_vs_i.gif}
1821: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070425/bkg_vs_i.gif}
1822: \else
1823: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070425/new_vs_i.eps}
1824: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070425/bkg_vs_i.eps}
1825: \fi
1826: \caption{Number of new objects added to catalog on subsequent images
1827: (left plot) and total number of new objects added to catalog in function of
1828: image number right plot. The data was collected on field 0800$+$20 ( average number of stars ~ 18000 )
1829: during night 2007.04.25/26 with shutter in normal (open/close) mode }
1830: \label{fig_new_vs_imageno_20070425}
1831: \end{center}
1832: \end{figure}
1833:
1834: % heplx43:/opt/pi/dev/pisys/daq/ndir/data/20070425/GIF/
1835: \begin{figure}[!htbp]
1836: \begin{center}
1837: \leavevmode
1838: \ifpdf
1839: \includegraphics[width=2.7in,height=5.4in]{PhotoEff/bkg_vs_image/20070425/sat/no_plane.gif}
1840: \includegraphics[width=2.7in,height=5.4in]{PhotoEff/bkg_vs_image/20070425/sat/plane.gif}
1841: \else
1842: \includegraphics[width=2.7in,height=5.4in]{PhotoEff/bkg_vs_image/20070425/sat/no_plane.eps}
1843: \includegraphics[width=2.7in,height=5.4in]{PhotoEff/bkg_vs_image/20070425/sat/plane.eps}
1844: \fi
1845: \caption{The reason for step visible on previous image at frame id=34. Part of image without satellite (or plane) trace is visible on left
1846: image and with the trace on right image. This trace causes addition of new objects to star catalog}
1847: \label{fig_red_bkg_plane}
1848: \end{center}
1849: \end{figure}
1850:
1851: % heplx48:/disk03/results/msok/piphoto_efficiency/results/aver20_tycho/0800+20/NEW/ALL
1852: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/new_vs_image/NEW
1853: % root : .x plotfile_multi.C("bkg_vs_i_all.txt",100,20000,1,0,"Number of images of field","Sum of all new objects added to catalog","Number of new objects added to catalog so far","Field")
1854: % script : check_new_vs_image_many!
1855: \begin{figure}[!htbp]
1856: \begin{center}
1857: \leavevmode
1858: \ifpdf
1859: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/bkg_vs_i_all.gif}
1860: \else
1861: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/bkg_vs_i_all.eps}
1862: \fi
1863: \caption{Number of new objects added to catalog from the beginning to
1864: given frame number. Data for different fields collected during many nights is
1865: shown.}
1866: \label{fig_bkg_vs_image_many_nights}
1867: \end{center}
1868: \end{figure}
1869:
1870:
1871: % /disk03/results/msok/piphoto_efficiency/results/20070512/NEW/bkg_vs_image_no
1872: % doit2!
1873: % heplx40:/var/www/html.orig/pi0/user/msok/doc/phd/badania/photo_eff/20070604/bkg_vs_image_no
1874: % select avg(istarcount) from frame where sobject='S0800+20' and falt>30;
1875: \begin{figure}[!htbp]
1876: \begin{center}
1877: \leavevmode
1878: \ifpdf
1879: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070512/new_vs_i.gif}
1880: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070512/bkg_vs_i.gif}
1881: \else
1882: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070512/new_vs_i.eps}
1883: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070512/bkg_vs_i.eps}
1884: \fi
1885: \caption{Number of new objects added to catalog in function of number of images
1886: (left plot) and total number of new objects added to catalog in function of
1887: image number right plot. The data was collected on field 0800$+$20 ( average number of stars ~ 18000 )
1888: during night 2007.05.12/13 with shutter permanently opened. }
1889: \label{fig_new_vs_imageno_20070512}
1890: \end{center}
1891: \end{figure}
1892:
1893: In Figure \ref{fig_new_vs_imageno_20070604} results for field ( 0851-70 )
1894: are shown, the number of background events added
1895: to catalog on every image is much larger. Most of these objects are
1896: artifacts coming from photometry of strip of charge which appears when
1897: images are collected with permanently opened shutter ( see Fig. \ref{fig_opened_shutter_effect} ).
1898: On field 0851-70 number of stars ( ~33000 ) is much higher then in field 0800$+$20 ( ~18000 )
1899: which causes much higher number of stars causing significant "open shutter" strips.
1900:
1901: % heplx48:/disk03/results/msok/piphoto_efficiency/results/20070604/bkg_vs_image_no
1902: % doit2!
1903: % heplx40:/var/www/html.orig/pi0/user/msok/doc/phd/badania/photo_eff/20070604/bkg_vs_image_no
1904: % select avg(istarcount) from frame where sobject='I0851-70' and falt>30;
1905: \begin{figure}[!htbp]
1906: \begin{center}
1907: \leavevmode
1908: \ifpdf
1909: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604/new_vs_i.gif}
1910: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604/bkg_vs_i.gif}
1911: \else
1912: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604/new_vs_i.eps}
1913: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604/bkg_vs_i.eps}
1914: \fi
1915: \caption{Number of new objects added to catalog on subsequent images
1916: (left plot) and total number of new objects added to catalog in function of
1917: image number right plot. Data was collected during night 2007.06.04/05 with
1918: permanently opened shutter, on field 0851-70 ( average number of stars ~ 33000 ) }
1919: \label{fig_new_vs_imageno_20070604}
1920: \end{center}
1921: \end{figure}
1922:
1923: Open shutter causes that charge is collected also in pixels above the
1924: readout pixel during chip readout ( Fig. \ref{fig_opened_shutter_effect} ). It is possible to reduce this effect by
1925: subtracting from every pixel fraction of values in pixels below. The image before and after
1926: the correction is shown in Figure \ref{fig_opened_shutter_effect}. The
1927: correction is not perfect, but significantly reduces this effect.
1928: The data from night 2007.06.04/05 was corrected and cataloged, number of new objects
1929: in function of image number is shown in Figure \ref{fig_new_vs_imageno_20070604_sc}.
1930: The number of new objects in every image is reduced approximately by a
1931: factor of 2.
1932:
1933: % opened shutter corrections :
1934: % heplx43:/opt/pi/dev/pisys/daq/ndir/data/20070604/FINAL/
1935: \begin{figure}[!htbp]
1936: \begin{center}
1937: \leavevmode
1938: \ifpdf
1939: \includegraphics[width=2.7in,height=3.24in]{PhotoEff/OpenedShutterCorr/image100_1000_1200.gif}
1940: \includegraphics[width=2.7in,height=3.24in]{PhotoEff/OpenedShutterCorr/image100_1000_1200_sc.gif}
1941: \else
1942: \includegraphics[width=2.7in,height=3.24in]{PhotoEff/OpenedShutterCorr/image100_1000_1200.eps}
1943: \includegraphics[width=2.7in,height=3.24in]{PhotoEff/OpenedShutterCorr/image100_1000_1200_sc.eps}
1944: \fi
1945: \caption{Image taken with permanently opened shutter before correction (left image)
1946: and after correction (right image) }
1947: \label{fig_opened_shutter_effect}
1948: \end{center}
1949: \end{figure}
1950:
1951:
1952: % heplx48:/disk03/results/msok/piphoto_efficiency/results/20070604_so/bkg_vs_image
1953: % heplx40:/lhome/piwww/www/pi0/user/msok/doc/phd/badania/photo_eff/20070604_so/
1954: \begin{figure}[!htbp]
1955: \begin{center}
1956: \leavevmode
1957: \ifpdf
1958: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604_sc/new_vs_i.gif}
1959: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604_sc/bkg_vs_i.gif}
1960: \else
1961: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604_sc/new_vs_i.eps}
1962: \includegraphics[width=2.7in,height=2.7in]{PhotoEff/bkg_vs_image/20070604_sc/bkg_vs_i.eps}
1963: \fi
1964: \caption{Number of new objects added to catalog on subsequent images
1965: (left plot) and total number of new objects added to catalog in function of
1966: image number right plot. Data was collected during night 2007.06.04/05 with
1967: shutter with opened shutter, on field 0851-70 and correction of opened shutter effect was applied }
1968: \label{fig_new_vs_imageno_20070604_sc}
1969: \end{center}
1970: \end{figure}
1971:
1972: The efficiency of star identification can be parameterized in function of
1973: number of stars in image. Figures \ref{fig_eff_vs_stars} and \ref{fig_eff_vs_stars_many_fields} show star identification efficiency in function of
1974: number of stars in image for star catalog created from images averaged over
1975: 20 and star catalog created from single images ( respectively ) . These plots were obtained for
1976: single sky field 0800$+$20.
1977:
1978: % heplx48:/disk03/results/msok/piphoto_efficiency/results/aver20_tycho/0800+20/eff_vs_stars/NEW/
1979: % doit!
1980: % nohup plot_eff_vs_imagestars\! aver20_tycho - 68 1677 > out2 2>&1 &
1981: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/eff_vs_stars/
1982: \begin{figure}[!htbp]
1983: \begin{center}
1984: \leavevmode
1985: \ifpdf
1986: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_stars/eff_vs_stars_aver20.gif}
1987: \else
1988: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_stars/eff_vs_stars_aver20.eps}
1989: \fi
1990: \caption{Efficiency of star identification in function of number of
1991: stars in image for sky field 0800$+$20. Star catalog obtained from images averaged over 20}
1992: \label{fig_eff_vs_stars}
1993: \end{center}
1994: \end{figure}
1995:
1996: However, as can be observed from Figures \ref{fig_eff_vs_stars} and
1997: \ref{fig_eff_vs_stars_many_fields} this is not the best parametrization.
1998: Much better parametrization is efficiency in function of ratio $R_{cat}$,
1999: defined as :
2000:
2001: \begin{equation}
2002: R_{cat} = \frac{N_{image\_stars}}{N_{cat\_stars}}
2003: \label{eq_ratio_cat}
2004: \end{equation}
2005:
2006: where $N_{image\_stars}$ is number of stars in image and $N_{cat\_stars}$ is
2007: number of stars found in reference catalog ( used in cataloging ) in the
2008: observed field.
2009: Star identification efficiency in function of ratio $R_{cat}$ is show in
2010: Figure \ref{fig_eff_vs_ratio}, this efficiency was calculated for different
2011: sky fields and different nights. As expected this dependency is linear.
2012: The average efficiency of star identification on single 10s exposure was
2013: determined by averaging efficiency from many single images collected during
2014: different nights ( every 25 image from single night was cataloged ).
2015: The resulting average efficiency for stars of brightness 0-12$^m$ is :
2016:
2017: \begin{equation}
2018: \epsilon_{star} \approx 0.49
2019: \label{eq_aver_single_image_star_eff}
2020: \end{equation}
2021:
2022: This number can be used to calculate total efficiency of the on-line flash
2023: recognition algorithm ( Sec. \ref{sec_opt_of_algo} ). Multiplication of
2024: efficiency of algorithm cuts determined as $\epsilon_{cuts} \approx $70\% by efficiency of
2025: finding stars in image $\epsilon_{star} \approx $50\% gives the overall efficiency of the
2026: flash recognition algorithm $\epsilon_{on-line} \approx$35\% ( for stars 0-12$^m$ ).
2027: It is larger for brighter stars and can reach nearly 70\% for stars 0-10$^m$.
2028: % which multiplied by average efficiency of on-line algorithm gives average
2029: % efficiency of on-line algorithm on the level of $\epsilon_{on-line} \approx$ 35\% .
2030:
2031: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/eff_vs_stars/single_tycho_tresh5_more
2032: % TUTAJ SKRYPTY I BAZA DANYCH : single_tycho_tresh5_more_new
2033: % heplx48:/disk03/results/msok/piphoto_efficiency/results/single_tycho_tresh5_more/NEW/
2034: \begin{figure}[!htbp]
2035: \begin{center}
2036: \leavevmode
2037: \ifpdf
2038: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_ratio/eff_vs_stars.gif}
2039: \else
2040: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_ratio/eff_vs_stars.eps}
2041: \fi
2042: \caption{Efficiency of star identification in function of number of stars on single 10s image }
2043: \label{fig_eff_vs_stars_many_fields}
2044: \end{center}
2045: \end{figure}
2046:
2047: \begin{figure}[!htbp]
2048: \begin{center}
2049: \leavevmode
2050: \ifpdf
2051: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_ratio/eff_vs_fract.gif}
2052: \else
2053: \includegraphics[width=6in,height=3.0in]{PhotoEff/eff_vs_ratio/eff_vs_fract.eps}
2054: \fi
2055: \caption{Efficiency of star identification in function of ratio $R_{cat}$}
2056: \label{fig_eff_vs_ratio}
2057: \end{center}
2058: \end{figure}
2059:
2060:
2061: The precision of star brightness measurements can be determined by finding
2062: dispersion of magnitude measurements for individual stars and plotting
2063: dispersion in function of star brightness. The Figure \ref{fig_sigmamag_vs_mag} shows
2064: distribution of dispersion in function of star brightness for stars observed during
2065: night 2007.06.06/07.
2066:
2067: % dane pi1:/data1/results/msok/sigmamag_vs_mag/
2068: % heplx43:/pi20/msok/praca/MY/badania/sigmamag_vs_mag/20070606/
2069: % plot_sigmamag_vs_mag!
2070: % wynik paramteryzacji Van Laskena :
2071: % RMS(mag) = 0.66221 * exp( 0.698794 * ( mag - 14.6308 ) ) + 0.02
2072: % it can be used for star lightcurves generation
2073: \begin{figure}[!htbp]
2074: \begin{center}
2075: \leavevmode
2076: \ifpdf
2077: \includegraphics[width=6in]{sigmamag_vs_mag_20070606_aver20_2006_at_pi3.gif}
2078: \else
2079: \includegraphics[width=6in]{sigmamag_vs_mag_20070606_aver20_2006_at_pi3.eps}
2080: \fi
2081: \caption{Precision of star brightness measurements in the photometry on images obtained by averaging 20 single (10s) images.}
2082: \label{fig_sigmamag_vs_mag}
2083: \end{center}
2084: \end{figure}
2085:
2086:
2087:
2088: \subsection{Off-line algorithms}
2089: \label{sec_offline_algo}
2090:
2091: Off-line algorithms act on data stored in the database. The data is
2092: cataloged in the way described in previous section and it is stored in tables
2093: \textbf{frame}, \textbf{star} and \textbf{measurements}. It is optimized for certain types of queries
2094: which are executed by analyzing programs. The algorithms described here have been
2095: implemented and tested on a star catalog created from images averaged over twenty - so
2096: called \texttt{aver20} database. However, they can also be used to analyse
2097: the catalog of single images data. Two algorithms developed by author will be
2098: described.
2099: The first one looks for new objects appearing in the star catalog which
2100: correspond to new objects appearing in the sky.
2101: Such kind of processes may be due to nova stars explosions or other
2102: kinds of processes when object below detection limit suddenly increases its
2103: brightness and appears as a new object.
2104: The second algorithm looks for sudden increases of
2105: stars brightness, such events can occur in flare stars, but this can
2106: also happen in other objects like blasars or AGNs.
2107: In both cases rejection of false event was the most important task.
2108: Both algorithms were implemented in \texttt{perl} scripting language.
2109:
2110: \subsubsection{Nova identification algorithm}
2111: \label{sec_nova_ident_algo}
2112: % script : do_flareevents.pl
2113: This algorithm was developed to find new objects which were not present in
2114: the star catalog before. The algorithm is invoked for data collected during
2115: specified night. It performs the analysis of all new objects added to the catalog during analysed
2116: night. The objects which are expected to be found by this kind of
2117: algorithm are those which are normally below detection threshold of "Pi of
2118: the Sky" detector, but due to intrinsic reasons increase their brightness
2119: and become possible to be detected. The astrophysical processes can
2120: be nova star explosion, supernova or GRB, but it can also be a flare star or
2121: even variable star of large amplitude of brightness variations.
2122: There is also large amount of background processes which must be efficiently
2123: rejected.
2124: The algorithm relies on flag \texttt{Measurements.new\_star} which is filled during
2125: cataloging. This field is true for the record which is the first measurement of
2126: the object. The algorithm is realized by \texttt{perl} script
2127: \texttt{do\_flareevents.pl}. The main steps of the algorithm are the following :\\
2128:
2129: \begin{enumerate}
2130: \item connection to the database star catalog
2131: \item selection of fields which were observed during the given night and have
2132: total number of observations $N_{obsfield}^{total} > N_{obsfield}^{min}$ .
2133: Only images from selected fields will be considered in next steps.
2134: \item every field selected in the previous step is analysed now. The first step is
2135: selection of all images of specific field from given night to list \texttt{id\_frm\_table}
2136: \item images are verified against clouds and moonlight, average pixel value $P_{avg}$
2137: in the image is compared with the limit $T_{avg}$, calculated as :
2138: \begin{equation}
2139: T_{avg}=<P_{avg}> + N_{avgcut} \cdot \sigma_{avg}
2140: \label{eq_avg_cut}
2141: \end{equation}
2142: values $<P_{avg}>$ and $\sigma_{avg}$ are obtained from distribution of
2143: average pixel values from all images collected so far.
2144: In case the image does not satisfy condition
2145: $P_{avg} < T_{avg}$ it is rejected and skipped from further analysis.
2146: Cloudy images are also rejected in cataloging phase by condition imposed on
2147: the number of stars in image ( see Sec. \ref{sec_cataloging} ).
2148:
2149: Every image from the analysed field kept in the array \texttt{id\_frm\_table} is now
2150: analysed, and the following subsequent criteria are applied to every image :\\
2151:
2152: \item all measurements from an image, flagged as new object ( with \texttt{Measurements.new\_star} = true )
2153: are selected with additional conditions for brightness of the new
2154: object \mbox{mag $\leq mag_{min}$} and for number of measurements \mbox{$N_{obs} \geq N_{obs}^{min}$}.
2155: Every object accepted by the above criteria is analysed as potential
2156: nova-like event and subsequent criteria are applied.
2157: \item verification of object in the second camera ( if available )
2158: \item determination of the number of object observations during the given night ( in case
2159: the second camera is not working this is the best indication of the event quality )
2160: \item The event is checked if there is no nearby ( R<$R_{bigstar}^{min}$ ) very bright star ( $mag_{bigstar} < mag_{bigstar}^{max}$)
2161: in "Pi of the Sky" or TYCHO-2 \cite{tycho} catalog.
2162: \item Despite the fact that the field was already observed at least
2163: $N_{obsfield}^{min}$ times, it is possible ( most probably due to bad weather conditions
2164: or faintness of the object ) that it is just a normal star which remained
2165: undetected before. Thus every event is checked against the
2166: TYCHO catalog and in case a star of brightness $mag < mag_{tycho}^{max}$ is found in distance R <
2167: $R_{star}^{tycho}$ the event candidate is rejected.
2168: \item In case the image was collected with permanently opened shutter,
2169: an additional check for bright stars below and above ( in CCD y coordinate )
2170: is performed. In case bright star ( $mag < mag_{above}$ )
2171: is found in the same image in the column close to the event candidate column ( $|X_{event} - X_{bigstar}| < \Delta X_{bigstar}^{max}$ ),
2172: then the event candidate is rejected.
2173: \item Event is verified against the list of known hot pixels, which are stored
2174: in database table \textbf{HotPixels} ( Fig. \ref{fig_starcat} )
2175: \item In case coordinates of the event are close to one of the Solar System
2176: planets it is rejected ( $D_{planet} < R_{planet}$ )
2177: \item Dispersions of mount tracking speeds are checked for each image. In case they
2178: exceed certain limit, the image is flagged as bad quality image ( shifted image ). Events
2179: found on such images are automatically rejected.
2180: \end{enumerate}
2181:
2182: All events which survived criteria 6) are saved to database table \textbf{FlareEvents}
2183: ( Fig. \ref{fig_starcat} ). Later cuts do not reject events definitely, only
2184: rejection flag field \texttt{fl\_rej\_tycho} is set in the database. Its value
2185: depends on the cut which rejected the event.
2186: Table \ref{tab_rejection_codes} lists possible values of rejection flag.
2187:
2188: \begin{table}[htbp]
2189: \begin{center}
2190: \begin{tabular}{|c|c|}
2191: \hline
2192: \textbf{Rejection Flag Value} & \textbf{Description} \\
2193: \hline
2194: \hline
2195: 0 & accepted event \\
2196: \hline
2197: 1 & star found in TYCHO \\
2198: \hline
2199: 2 & nearby bright star found in TYCHO \\
2200: \hline
2201: 3 & saturated star found above event ( $Y_{star} > Y_{event}$ ) \\
2202: \hline
2203: 4 & saturated star found below event ( $Y_{star} < Y_{event}$ ) \\
2204: \hline
2205: 5 & bright star found in \piname star catalog \\
2206: \hline
2207: 6 & hot pixel \\
2208: \hline
2209: 7 & sky background level > $T_{avg}$ \\
2210: \hline
2211: 8 & anti planets cut \\
2212: \hline
2213: 9 & image quality cut \\
2214: \hline
2215: \end{tabular}
2216: \caption{Rejection flags values}
2217: \label{tab_rejection_codes}
2218: \end{center}
2219: \end{table}
2220:
2221: Events presentation is implemented in form of \texttt{php} script selecting events
2222: from the database. The additional difficulty is the fact that algorithm is
2223: executed on the remote server and results are originally saved to the remote
2224: database. The events data is copied to the local server ( Sec.
2225: \ref{sec_data_synchronization} ) for convenience and also in order not to
2226: overuse the Internet link to LCO by direct accessing of the remote database
2227: multiple times.
2228: The synchronization is executed once and all later analysis and inspections of events are
2229: using events data from local server not from LCO which is much faster.
2230:
2231: \begin{table}[htbp]
2232: \begin{center}
2233: \begin{tabular}{|c|c|c|}
2234: \hline
2235: \textbf{Parameter} & \textbf{Default Value} & \textbf{Script option} \\
2236: \hline
2237: \hline
2238: $N_{obsfield}^{min}$ & 50 & -min\_obs\_field \\
2239: \hline
2240: $mag_{min}$ & 11 & -mag \\
2241: \hline
2242: $mag_{bigstar}^{max}$ & 8 & -near\_bigstar\_max\_mag \\
2243: \hline
2244: $R_{bigstar}^{min}$ & 15*36 [arcsec] & - \\
2245: \hline
2246: $R_{star}^{tycho}$ & 150 [arcsec] & -near\_star\_radius\_arcsec \\
2247: \hline
2248: $mag_{tycho}^{max}$ & 13 & -max\_tycho\_star \\
2249: \hline
2250: $mag_{bigstar}^{above}$ & 5 & -max\_bigstar\_above \\
2251: \hline
2252: $\Delta X_{bigstar}^{max}$ & 10 & -max\_bigstar\_x\_dist \\
2253: \hline
2254: $N_{avgcut}$ & 3 & - \\
2255: \hline
2256: $R_{planet}$ & 7200 [arcsec] & -reject\_planets \\
2257: \hline
2258: \end{tabular}
2259: \caption{Most important parameters of the nova identification algorithm}
2260: \label{tab_nova_algo_params}
2261: \end{center}
2262: \end{table}
2263:
2264: % to mozna sprawdzic :
2265: % find . -name "nova.log" -exec grep "All night new_star" {} \;
2266: The cataloging program stores measurements of $10^4$ - $10^6$ stars per night and flags new objects with special flag.
2267: The nova identification algorithm analysis stars from a given night and
2268: flagged as new objects in the catalog. Large part of the job is performed by the
2269: cataloging program, the algorithm itself analysis $N_{new} \approx 10^4-10^5$ objects, which takes 1-3 hours.
2270: Figure \ref{fig_nova_algo_cuts} shows number of events after subsequent cuts for several nights.
2271:
2272: % heplx40:http://grb.fuw.edu.pl/pi0/user/msok/doc/phd/badania/algo_cuts/nova/20070603/nova_algo_cuts.gif
2273: % ploty zrobione w heplx48:/disk03/results/msok/plots/nova_cuts/all
2274: % przy pomocy skryptu plot_flare_cuts! list1 , lub plot_flare_cuts! list2 etc, etc
2275: % SQL do sprawdzania dlaczego tyle nowych :
2276: % select count(*) from measurements where new_star='t' and id_frm in
2277: % (select id_frm from frame where idaynight=20070603);
2278: % wyniki : 20070603 = 3817 , 20070604 = 9992 , 20070605 = 11953 , 20070606 = 18641 , o dziwo 20070607 = 8655 !!!
2279: % ale moze ona ma najwiecej czesto obserwowanych pol ?
2280: \begin{figure}[!htbp]
2281: \begin{center}
2282: \leavevmode
2283: \ifpdf
2284: \includegraphics[width=6in]{cuts/nova/nova_algo_cuts.gif}
2285: \else
2286: \includegraphics[width=6in]{cuts/nova/nova_algo_cuts.eps}
2287: \fi
2288: \caption{Number of events after subsequent cuts of nova identification algorithm. Results from five nights are shown.}
2289: \label{fig_nova_algo_cuts}
2290: \end{center}
2291: \end{figure}
2292:
2293:
2294: Efficiency of nova identification algorithm was tested. The following
2295: procedure was used, given number of stars ( typically 100 / image ) of given
2296: brightness were added to star catalog database.
2297: Algorithm was executed on data from night for which artificial stars with measurements were
2298: added. The number $N_{gen}^{ident}$ of generated and accepted events was
2299: determined and using this number efficiency was determined as : \\
2300:
2301: \begin{equation}
2302: \epsilon_{nova} = \frac{N_{gen}^{ident}}{N_{gen}}
2303: \label{eq_eff_nova}
2304: \end{equation}
2305:
2306: % heplx48:/disk03/results/msok/nova/eff/nova_eff
2307: % PLOTS : heplx43:/pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS
2308: % SCRIPT : doit! ( calls : plot_nova_eff_multi! plot_novaeff_vs_bkg_multi! )
2309: % AKTUALNIE uzywane min_obs_field=20 i dla niego wyszla mi srednia efektywnosc :
2310: % cd /pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field20
2311: % awk -v n=0 -v sum=0 '{if($3>0.1){n++;sum+=$3;}print n" "sum/n;}' novaeff_vs_night.txt
2312: % wyszlo mi : 25 0.446758 z 25 nocy srednia ~0.44
2313: % skrypt lokalny calc_avg!
2314: \begin{figure}[!htbp]
2315: \begin{center}
2316: \leavevmode
2317: \ifpdf
2318: \includegraphics[width=6in]{nova_eff/novaeff_vs_night.gif}
2319: \else
2320: \includegraphics[width=6in]{nova_eff/novaeff_vs_night.eps}
2321: \fi
2322: \caption{Efficiency of nova identification algorithm for several nights.
2323: Different values of minimum number of field observations requirement were
2324: tested}
2325: \label{fig_novaeff_vs_night}
2326: \end{center}
2327: \end{figure}
2328:
2329: % heplx48:/disk03/results/msok/nova/eff/nova_eff
2330: % PLOTS : heplx43:/pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS
2331: % SCRIPT : plot_novaeff_vs_bkg_multi!
2332: \begin{figure}[!htbp]
2333: \begin{center}
2334: \leavevmode
2335: \ifpdf
2336: \includegraphics[width=6in]{nova_eff/novaeff_vs_bkg.gif}
2337: \else
2338: \includegraphics[width=6in]{nova_eff/novaeff_vs_bkg.eps}
2339: \fi
2340: \caption{Efficiency of nova identification algorithm vs number of
2341: background events from single night}
2342: \label{fig_novaeff_vs_bkg}
2343: \end{center}
2344: \end{figure}
2345:
2346: % hepxl43 :
2347: % /pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field5
2348: % /pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field10
2349: % /pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field20
2350: % /pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field30
2351: % doit!
2352: \begin{figure}[!htbp]
2353: \begin{center}
2354: \leavevmode
2355: \ifpdf
2356: % was 2.7 x 2.7
2357: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field5.gif}
2358: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field10.gif}
2359: \else
2360: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field5.eps}
2361: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field10.eps}
2362: \fi
2363: \caption{Efficiency losses on subsequent cuts of algorithm for different
2364: values of parameter $N_{obsfield}^{min}$=5,10}
2365: \label{fig_novaeff_loses1}
2366: \end{center}
2367: \end{figure}
2368:
2369: \newpage
2370: \begin{figure}[!htbp]
2371: \begin{center}
2372: \leavevmode
2373: \ifpdf
2374: % was 2.7 x 2.7
2375: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field20.gif}
2376: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field30.gif}
2377: \else
2378: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field20.eps}
2379: \includegraphics[width=7.2in,height=4in]{nova_eff/min_obs_field_compare/min_obs_field30.eps}
2380: \fi
2381: \caption{Efficiency losses on subsequent cuts of algorithm for different
2382: values of parameter $N_{obsfield}^{min}$=20,30}
2383: \label{fig_novaeff_loses2}
2384: \end{center}
2385: \end{figure}
2386:
2387:
2388: The Figure \ref{fig_novaeff_vs_night} shows efficiency for several
2389: subsequent nights. The best efficiencies were obtained when only 5 earlier
2390: observations of sky field were required. However, this is very small number
2391: of field observations and many weak objects still remain undetected. This
2392: implies large amount of background events, which can be seen in Figure
2393: \ref{fig_novaeff_vs_bkg}.
2394: The Figures \ref{fig_novaeff_loses1} and \ref{fig_novaeff_loses2} show efficiency losses on subsequent
2395: nights for different values of $N_{obsfield}^{min}$ parameter.
2396:
2397: % novaeff vs mag
2398: % heplx48:/disk03/results/msok/nova/eff/nova_eff/mag_test
2399: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/mag_test
2400: % skrypt glowny : plot_novaeff_vs_mag!
2401: % <eff> = CONST ~= 0.7 ( dla tej nocy 20070520 )
2402: %
2403: % ALL - 5 nocy : 20070520 , 20070522 , 20070525 , 20070604 , 20070621
2404: % heplx48:/disk03/results/msok/nova/eff/nova_eff/mag_test
2405: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/mag_test/ALL/
2406: % root -b -q "novaeff_vs_mag.C(\"many_nights.txt\")"
2407: \begin{figure}[!htbp]
2408: \begin{center}
2409: \leavevmode
2410: \ifpdf
2411: % \includegraphics[width=4in,height=4in]{nova_eff/20070520_novaeff_vs_mag.gif}
2412: \includegraphics[width=4in,height=4in]{nova_eff/many_nights.gif}
2413: \includegraphics[width=4in,height=4in]{nova_eff/many_night_effloss_reason.gif}
2414: \else
2415: % \includegraphics[width=4in,height=4in]{nova_eff/20070520_novaeff_vs_mag.eps}
2416: \includegraphics[width=4in,height=4in]{nova_eff/many_nights.eps}
2417: \includegraphics[width=4in,height=4in]{nova_eff/many_night_effloss_reason.eps}
2418: \fi
2419: \caption{Efficiency of nova identification algorithm for different brightness of object added to star
2420: catalog, tested for different nights (upper plot). The lower plot shows number of
2421: generated events after subsequent cuts.}
2422: \label{fig_novaeff_vs_mag}
2423: \end{center}
2424: \end{figure}
2425:
2426:
2427: The Figure \ref{fig_novaeff_vs_mag} shows efficiency in
2428: function of brightness of generated stars. It may be astonishing that
2429: it is independent of star magnitude, however, it is clear that it should not.
2430: The procedure described above tested only efficiency of subsequent cuts of
2431: the algorithm which does not depend on star brightness.
2432: The efficiency is different for different nights mainly due to differences in Moon phase
2433: and observed fields. Due to cut on number of field observations $N_{obsfield}>N_{obsfield}^{min}$
2434: causing rejection of events generated in images of rarely observed fields.
2435: The best way to eliminate dependency of the efficiency on number of field
2436: observations would be to initialize star catalog with the complete catalog
2437: of objects matching the range of "Pi of the Sky" telescope ( ~ 12-13$^m$ ).
2438: Such a catalog should contain all types of objects ( star, galaxies etc ) in
2439: the range of the telescope.
2440: The overall efficiency of nova determination depends on the star brightness. This
2441: dependency is "hidden" in the photometry procedure. The efficiency of star identification procedure was determined and
2442: described in Section \ref{sec_red_eff}. The result of multiplication of photometry efficiency
2443: on single image ( result of average over 20 ) in
2444: function of star brightness by average algorithm efficiency $\epsilon_{nova-algo} \approx$0.44 is
2445: shown in Figure \ref{fig_overall_nova_eff}.
2446: Comparison of efficiency losses on subsequent cuts for five nights is shown in Figure
2447: \ref{fig_novaeff_night_compare}.
2448:
2449: % heplx43:/pi20/msok/praca/MY/badania/photo_eff/eff_vs_mag/eff_vs_mag/EFF_TOTAL
2450: % awk '{eff=$2*0.44;print $1" "eff;}' eff_vs_mag_9_9.txt > eff_total.txt
2451: %
2452: \begin{figure}[!htbp]
2453: \begin{center}
2454: \leavevmode
2455: \ifpdf
2456: \includegraphics[width=7.2in,height=4in]{nova_eff/eff_on_single_aver20_image_total.gif}
2457: \else
2458: \includegraphics[width=6.0in,height=3.3in]{nova_eff/eff_on_single_aver20_image_total.eps}
2459: \fi
2460: \caption{Total efficiency of nova ( of brightness 0-12$^m$ ) identification algorithm on single aver20 image ( resulting from averaging 20 single images ).}
2461: \label{fig_overall_nova_eff}
2462: \end{center}
2463: \end{figure}
2464:
2465:
2466: % heplx48:/disk03/results/msok/nova/eff/nova_eff/
2467: % heplx43:/pi20/msok/praca/MY/badania/nova_eff/FINAL_PLOTS/min_obs_field20/doit!
2468: \begin{figure}[!htbp]
2469: \begin{center}
2470: \leavevmode
2471: \ifpdf
2472: \includegraphics[width=7.2in,height=4in]{nova_eff/night_compare.gif}
2473: \else
2474: \includegraphics[width=6.0in,height=3.3in]{nova_eff/night_compare.eps}
2475: \fi
2476: \caption{Comparison of efficiency losses on subsequent cuts for 5 nights. During night 2007-05-29/30
2477: the Moon was almost full and cut requiring low level of sky background
2478: rejected most of the generated events}
2479: \label{fig_novaeff_night_compare}
2480: \end{center}
2481: \end{figure}
2482:
2483:
2484: Important result of the tests is determination of the amount of background
2485: events which must be verified daily. The Figure \ref{fig_novaeff_vs_bkg} shows
2486: number of background events vs efficiency of the algorithm for several
2487: different values of $N_{obsfield}^{min}$ parameter.
2488: The final values of the parameters allow to efficiently suppress the
2489: background so that number of events to checked does not exceed twenty events ( during clean sky night ).
2490:
2491: \subsubsection{Flare identification algorithm}
2492: % script : find_flares_all_nights.pl
2493: This algorithm was developed to find outbursts of stars which are already in
2494: the database star catalog, but manifest sudden increase of brightness.
2495: The algorithm is executed on data collected during specified night, it
2496: is started after cataloging of new night is finished. Currently cataloging
2497: is performed on-line during data collection so analyzing script
2498: ( \texttt{find\_flares\_all\_nights.pl} ) can be launched in the morning.
2499: The main steps of the algorithm are the following :\\
2500: \begin{enumerate}
2501: \item Selection of stars brighter then $mag_{min}$, observed during analysed night and having
2502: total number of measurements $N_{obs} \geq N_{obs}^{min}$
2503: % to jednak tak jest tzn ostatecznie analizowane sa tylko gwiazdy obserwowane zadanej nocy
2504: % chociaz tu mozna to ulepszyc juz na poziomie pierwszego SELECT-a
2505: % czy to nie byloby jednak szybsze - dodac tam warunek na zakres klatek ???
2506: \item Variability cut - star brightness range must satisfy condition : $mag_{MAX}-mag_{MIN} \geq T_{magdiff}$.
2507: The number of stars selected at this stage is of the order of $10^4-10^5$. It depends on
2508: quality of the night data and on $T_{magdiff}$ parameter.
2509: This dependency is shown in Fig. \ref{fig_magdiff_histo}.
2510:
2511: % heplx43:/pi20/msok/praca/MY/badania/offline_algo/flare
2512: % histofile_cumul.C
2513: % dane z nocy 20070526/27 : aver20_2006@pi3:/data1/results/msok/mag_diff
2514: %
2515: \begin{figure}[!htbp]
2516: \begin{center}
2517: \leavevmode
2518: \ifpdf
2519: \includegraphics[width=2.7in,height=2.7in]{offline/flare/20070526_magdiff.gif}
2520: \includegraphics[width=2.7in,height=2.7in]{offline/flare/mag_diff_normal.gif}
2521: \else
2522: \includegraphics[width=2.7in,height=2.7in]{offline/flare/20070526_magdiff.eps}
2523: \includegraphics[width=2.7in,height=2.7in]{offline/flare/mag_diff_normal.eps}
2524: \fi
2525: \caption{Number of stars with $mag_{MAX}-mag_{MIN} \leq T_{magdiff}$ (left plot) and distribution of $mag_{MAX}-mag_{MIN}$ (right plot). Only stars satisfying condition on number of measurements are shown ( data from night 2007.05.26/27 ). }
2526: \label{fig_magdiff_histo}
2527: \end{center}
2528: \end{figure}
2529:
2530: For every selected star the following steps and criteria are performed :
2531:
2532: \item Select measurements for given star
2533: \item Check if number of measurements for the given night $N_{obs}^{night} \geq N_{minobs}^{night}$
2534: \item Find upper limit of the magnitude range $mag_{max}$, which is defined
2535: as maximum value of magnitudo in the "last" non empty bin \footnote{in the direction of increasing magnitude} of magnitudo
2536: measurements distribution ( see Fig. \ref{fig_measurements_strip} ).
2537: In most cases the value $mag_{max}$ is the same as maximum magnitude
2538: measurement for given star. However, in some cases (e.g. due to clouds)
2539: real "last" non empty bin can contain few outliers and can be separated
2540: from most of the measurements ( see Fig. \ref{fig_last_non_empty_bin} ). In order to exclude those bad measurements,
2541: "last" non empty bin [M,M$+\Delta$M] is chosen as the one before an empty bin and
2542: satisfying condition :
2543:
2544: \begin{equation}
2545: N( mag \leq M+\Delta M ) \geq 50\% \cdot N_{obs}
2546: \label{eq_flare_range_cond}
2547: \end{equation}
2548:
2549: The maximum magnitude is found in the "last" non empty bin and used as upper limit $mag_{max}$.
2550: \item Find lower limit of magnitude $mag_{min}$ so that at least 85\% of all magnitude
2551: measurements for given star belong to range ($mag_{min}$,$mag_{max}$) ( see Fig. \ref{fig_measurements_strip} ).
2552: Value of flare threshold is determined as $T_{flare}=mag_{min}$, all points brighter then this value
2553: are considered as belonging to an outburst event.
2554: \item Find longest series of measurements with mag < $T_{flare}$
2555: and require this set to have at least $N_{min}^{flare}$ points
2556: \item Check maximum brightness measurement $M_{max}^{flare}$ in the flare series and
2557: determine its time $t^{flare}$
2558: \item Verify if measurements within the series and before and after
2559: have the same chip coordinates (x,y), because due to calibration imperfectness star brightness
2560: measurements can vary with the star position on the chip, causing background events with flare
2561: like signature ( visible as step on the light curve )
2562: \item If the data was collected with permanently opened shutter verify if there is no star brighter then
2563: $mag_{above}$ in the strip of $\Delta X_{bigstar}^{max}$ pixels from the analysed star.
2564: Opened shutter causes that column obtains additional signal from stars with
2565: $Y \leq Y_{0}$ and can generate background events. The rejection condition is the
2566: same as in described earlier nova identification algorithm.
2567: \item Check if there is no bright star nearby which could affect
2568: measurements of flare suspected star. In case a star brighter then
2569: $mag_{bigstar}^{near}$ is closer then $R_{bigstar}$ then event is rejected.
2570: This condition is checked in the "Pi of the Sky" star catalog and also in TYCHO-2 \cite{tycho} star catalog
2571: \item Verify if brightness increase is not due to hotpixel. Chip coordinates
2572: (x,y) are verified. The event is rejected if its (x,y) coordinates belong to a list of known CCD defects stored
2573: in the database table \textbf{HotPixel} ( Fig. \ref{fig_starcat} ).
2574: \item Peak height $\Delta M_{max}^{flare}$ over the average brightness level is determined and flare
2575: event obtains quality flag according to this value, events with
2576: $\Delta M_{max}^{flare}>0.4^m$ obtain quality=1 and those with $\Delta M_{max}^{flare}>1.0^m$ obtain quality=2
2577: \item Finally event is accepted and saved to the database table \textbf{FlareEvent} with
2578: all information describing this event and calculated by the algorithm
2579: \item Sky background level in the image is checked in the same way as in
2580: nova identification algorithm (Sec. \ref{sec_nova_ident_algo}).
2581: \end{enumerate}
2582:
2583: Algorithm parameters are listed in Table \ref{tab_flare_algo_params}.
2584:
2585: % /pi20/msok/data/20070302/cat/corr
2586: % /pi20/msok/praca/MY/badania/flare_algo/
2587: % plot_star.sh 3413787 9.46372 9.59724 - phd_plots - 20061127 050650-2135.0
2588: % plot_star.sh 3413787 9.46372 9.59724 - phd_plots - 20061127
2589: % baza phd_plots
2590: % star = 3413787
2591: %
2592: % HISTO :
2593: % /pi20/msok/praca/MY/badania/flare_algo/HISTO/
2594: % .x
2595: % histofile.C("3413787.txt",0,0,9.02979,9.70025+0.4,(9.70025-9.02979)/0.2,0,"Magnitude","Number of measurements for single star ",1,"050650-2135.0")
2596: \begin{figure}[!htbp]
2597: \begin{center}
2598: \leavevmode
2599: \ifpdf
2600: % was 4/4
2601: \includegraphics[width=2.8in,height=2.8in]{3413787_flare_20061127.gif}
2602: \includegraphics[width=2.8in,height=2.8in]{3413787_measurements_histo.gif}
2603: \else
2604: \includegraphics[width=2.8in,height=2.8in]{3413787_flare_20061127.eps}
2605: \includegraphics[width=2.8in,height=2.8in]{3413787_measurements_histo.eps}
2606: \fi
2607: \caption{Example of determined range of 85\% of measurements points ( area between horizontal lines in the left plot ) and distribution of all measurements for the same star (right plot)}
2608: \label{fig_measurements_strip}
2609: \end{center}
2610: \end{figure}
2611:
2612: % /pi20/msok/praca/MY/badania/flare_algo/HISTO/EXAMPLE
2613: % ./histo_flare.sh 208722300000
2614: % nowy z pokazaniem "Last non empty bin" jest tutaj :
2615: % heplx43:/pi20/msok/praca/MY/badania/flare_algo/HISTO/EXAMPLE/NEW/
2616: % command : root -b -q "histofile_with_cut.C(\"208722300000.txt\",0,0,10,13,3.0/0.2,0,\"Magnitude\",\"Number
2617: % of measurements for single star
2618: % \",1,\"050650-2135.0\",11.1,-1000000,0.4)"
2619: \begin{figure}[!htbp]
2620: \begin{center}
2621: \leavevmode
2622: \ifpdf
2623: \includegraphics[width=4in]{208722300000.txt_histo.gif}
2624: \else
2625: \includegraphics[width=4in]{208722300000.txt_histo.eps}
2626: \fi
2627: \caption{The distribution of magnitude measurements for a single star an argument why not maximum magnitude is used as upper limit $mag_{max}$ in the algorithm}
2628: \label{fig_last_non_empty_bin}
2629: \end{center}
2630: \end{figure}
2631:
2632: \begin{table}[htbp]
2633: \begin{center}
2634: \begin{tabular}{|c|c|c|}
2635: \hline
2636: \textbf{Parameter} & \textbf{Default Value} & \textbf{Script option} \\
2637: \hline
2638: $N_{obs}^{min}$ & 30 & -min\_points \\
2639: \hline
2640: $mag_{min}$ & 12 [mag] & -min\_mag\_star \\
2641: \hline
2642: $T_{magdiff}$ & 0.5 [mag] & -min\_max\_mag\_limit \\
2643: \hline
2644: $N_{minobs}^{night}$ & 20 & -min\_night\_points \\
2645: \hline
2646: $N_{min}^{flare}$ & 3 & -min\_points\_above \\
2647: \hline
2648: $mag_{bigstar}^{strip}$ & 5 [mag] & -max\_bigstar\_above \\
2649: \hline
2650: $W_{strip}$ & 10 [pixels] & -max\_bigstar\_x\_dist \\
2651: \hline
2652: $mag_{bigstar}^{near}$ & 8 [mag] & -near\_bigstar\_max\_mag \\
2653: \hline
2654: $R_{bigstar}$ & 15 [pixels] & -near\_bigstar\_distance \\
2655: \hline
2656: \end{tabular}
2657: \caption{Most important parameters of the flare identification algorithm}
2658: \label{tab_flare_algo_params}
2659: \end{center}
2660: \end{table}
2661:
2662: Depending on the quality of the night, algorithm analysis $N_{star}<10^5$ stars which
2663: takes about 1-2 hours. Event numbers after subsequent cuts are shown in Figure \ref{fig_flare_algo_cuts}.
2664:
2665: % heplx43:/pi20/msok/praca/MY/badania/flare_algo/cuts/FINAL
2666: % doit!
2667: % chodzi tam o to ze zle sie liczyla liczba przypadkow po clouds cut i byla
2668: % wieksza niz poprzednie ciecie , dlatego zmienilem recznie kolejnosc i juz
2669: % na tych logach zapuscilem skrypt do robienia rysunkow
2670: \begin{figure}[!htbp]
2671: \begin{center}
2672: \leavevmode
2673: \ifpdf
2674: \includegraphics[width=7.2in,height=4in]{flare_algo/flare_algo_cuts.gif}
2675: \else
2676: \includegraphics[width=6.0in,height=3.3in]{flare_algo/flare_algo_cuts.eps}
2677: \fi
2678: \caption{Number of events after subsequent cuts of the flare identification algorithm. The data from five nights is shown.}
2679: \label{fig_flare_algo_cuts}
2680: \end{center}
2681: \end{figure}
2682:
2683:
2684: In order to determine efficiency of this algorithm flare events were generated.
2685: Events were generated for specified night and analysed by the same script
2686: for flare identification used for analysis of the real data.
2687: The simulation consists of the following main steps :
2688:
2689: \begin{itemize}
2690: \item Certain number $N_{flare}^{gen}$ of stars in the star catalog was selected ( specified by simulation parameter )
2691: \item Time of flare start was randomly chosen from the series of measurements for a given night
2692: \item Magnitude measurements after the starting time were replaced by values
2693: obtained from flare parametrization
2694: \item Star with generated points added was analysed and accepted or rejected by the algorithm
2695: \end{itemize}
2696:
2697: In order to simulate flare-like outburst simple linear/exponential parametrization was
2698: fitted to real flare detected by the algorithm ( see Section \ref{offline_flashes} ) :
2699:
2700: \begin{equation}
2701: mag = \left\{
2702: \begin{array}{rl}
2703: m_0 - F_{max} \cdot \frac{t}{\Delta t} & \mbox{for }t\leq \Delta t \\
2704: m_0 - F_{max} \cdot exp\left[ -\frac{t-\Delta t}{ \tau } \right] & \mbox{for }t\ge \Delta t
2705: \end{array}\right.
2706: \label{eq_flare_param}
2707: \end{equation}
2708:
2709: These parametrization is presented in Figure \ref{fig_flare_paramterization}.
2710:
2711: % heplx43:/pi20/msok/praca/MY/badania/flare_eff/FLARE_FIT
2712: % fit_flare.C
2713: % 20061127ap0 @ heplx49 star=21878
2714: \begin{figure}[!htbp]
2715: \begin{center}
2716: \leavevmode
2717: \ifpdf
2718: \includegraphics[width=6in,height=2in]{flare_eff/gj_lc_fit.gif}
2719: \else
2720: \includegraphics[width=6in,height=2in]{flare_eff/gj_lc_fit.eps}
2721: \fi
2722: \caption{Linear/Exponential parametrization of the flare-like outburst. Fit was
2723: performed to real flare star GJ 3331A / GJ 3332 outburst which occurred on 2006.11.28 06:03 UT }
2724: \label{fig_flare_paramterization}
2725: \end{center}
2726: \end{figure}
2727:
2728: Examples of generated flare light curves are shown in Figure \ref{fig_gen_flares}.
2729: Flare identification efficiency was tested in function of 4 parameters used
2730: to describe the flare GJ3331A/GJ3332 in fitted parametrization.
2731:
2732: % ALL DATA in heplx48:/disk03/results/msok/simul/flares/TEST/20070616/I0240-10/min_points_above_3/min_points_above_1/
2733: % heplx43:/pi20/msok/praca/MY/badania/flare_eff/GENERATOR/VS_DECAY_TIME_200
2734: % heplx43:/pi20/msok/praca/MY/badania/flare_eff/GENERATOR/VS_MAG_TEST_1.00
2735: % skrypt : show_flare_gen!
2736: % root : plotfile_multi.C("1261271_both.txt",-10000,-10000,0,0,"Time HJD","Magnitude","Original and generated light curve")
2737: \begin{figure}[!htbp]
2738: \begin{center}
2739: \leavevmode
2740: \ifpdf
2741: \includegraphics[width=2.7in,height=2.7in]{flare_eff/gen/1149554_both.gif}
2742: \includegraphics[width=2.7in,height=2.7in]{flare_eff/gen/1261271_both.gif}
2743: \else
2744: \includegraphics[width=2.7in,height=2.7in]{flare_eff/gen/1149554_both.eps}
2745: \includegraphics[width=2.7in,height=2.7in]{flare_eff/gen/1261271_both.eps}
2746: \fi
2747: \caption{Example of light curves with generated flare with outburst peak of
2748: 1$^m$ above average brightness level (left plot) and decay time of 200 sec (right plot), other parameters
2749: of the light curves are the same as those fitted to GJ3331A/GJ3332 outburst}
2750: \label{fig_gen_flares}
2751: \end{center}
2752: \end{figure}
2753:
2754:
2755: The results of these tests are shown in Figure \ref{fig_flare_eff}.
2756:
2757: % heplx48:/disk03/results/msok/simul/flares/TEST/20070616/I0240-10/min_points_above_3/min_points_above_1/MAG_TEST
2758: % heplx43:/pi20/msok/praca/MY/badania/flare_eff/GENERATOR/VS_MAG_TEST_1.00/FINAL
2759: % plotfile.C("flare_eff_vs_flare_mag.txt",NULL,-10000,-10000,0,"Outburst amplitude [mag]","Efficiency of flare identification ","Efficiency of flare identification vs outburst magnitude")
2760: \begin{figure}[!htbp]
2761: \begin{center}
2762: \leavevmode
2763: \ifpdf
2764: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_flare_mag.gif}
2765: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_rise_time.gif}
2766: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_decay_time.gif}
2767: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_mag0.gif}
2768: \else
2769: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_flare_mag.eps}
2770: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_rise_time.eps}
2771: % VS_DECAY_TIME , heplx43:/pi20/msok/praca/MY/badania/flare_eff/GENERATOR/VS_DECAY_TIME
2772: % heplx48:/disk03/results/msok/simul/flares/TEST/20070616/I0240-10/min_points_above_3/min_points_above_1/VS_DECAY_TIME
2773: % root : .x plotfile.C("eff_vs_dt.txt",0,-10000,-10000,0,"Decay Time \\tau [sec]","Efficiency of flare identification ","Efficiency of flare identification vs decay time")
2774: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_decay_time.eps}
2775: % VS_MAGNITUDE ( mag0 ) - normal star brightness
2776: % heplx48:/disk03/results/msok/simul/flares/TEST/20070616/I0240-10/min_points_above_3/min_points_above_1/VS_MAGNITUDE/NEW
2777: % .x plotfile.C("eff_vs_mag.txt",0,0,1.2,0,"Normal star brightness ( not during flare )","Efficiency of flare identification ","Efficiency of flare identification vs star brightness")
2778: % heplx43:/pi20/msok/praca/MY/badania/flare_eff/GENERATOR/VS_MAG0
2779: \includegraphics[width=2.7in,height=2.7in]{flare_eff/flare_eff_vs_mag0.eps}
2780: \fi
2781: \caption{Efficiency of flare identification in function of outburst
2782: amplitude (left upper plot), rise time (right upper), decay time (left
2783: bottom), star average magnitude (right bottom)}
2784: \label{fig_flare_eff}
2785: \end{center}
2786: \end{figure}
2787:
2788: % comments :
2789: % vs rise time - drop of eff at rise time >=1000 sec , poniewaz za wolno
2790: % rosnie i nie zdarzy urasnac na tyle duza zeby zostac zlapana przez
2791: % algorytm, czesto tez wczesniej nastepuje koniec obserwacji
2792:
2793:
2794:
2795: \subsubsection{Data synchronization and presentation}
2796: \label{sec_data_synchronization}
2797:
2798: The cataloging and data analysis are performed on the remote server. In the
2799: prototype version it is performed on pi1, pi2 and pi3 computers at LCO.
2800: In order not to overuse the bandwidth only the final results are once copied to local server in
2801: Warsaw. It is impossible to synchronize the entire database because it is too
2802: large. However, results of algorithms can be synchronized with light curves of
2803: corresponding stars. There is also possibility to specify interesting
2804: objects which light curves should also be synchronized.
2805: After synchronization data is stored in the local database which is
2806: accessible from the command line and script level, but also through the WWW
2807: \texttt{php} interface.
2808: Results of on-line algorithms are synchronized on-line during a night.
2809: Besides database information also subrenders of the sky images containing events
2810: are copied to local server allowing for quick verification of event candidates.
2811: They are also available through the WWW interface.
2812:
2813: \section{Periods when algorithms were working}
2814: \label{sec_algo_usage_periods}
2815:
2816: Before showing the results of the algorithms it is important to mention that
2817: not all algorithms were used during the full time of the prototype work.
2818: Table \ref{tab_proto_periods} shows periods and configurations in which the prototype was working.
2819: Because of technical problems shown in this table coincidence algorithm
2820: could not work for all the time. Also not all the algorithms were ready to be
2821: used since the very beginning. Table \ref{tab_algo_periods} shows which
2822: algorithms were used in what periods. Some algorithms could also be run
2823: off-line, but large amount of events could not be verified.
2824: Currently all off-line algorithms are working and events are systematically checked
2825: after each night. \\
2826:
2827: \begin{table}[htbp]
2828: \begin{center}
2829: % \begin{small}
2830: \begin{tabular}{|c|c|c|}
2831: \hline
2832: \textbf{Algorithm Type} & \textbf{Working Periods} & \textbf{Notes} \\
2833: % & & \\
2834: \hline
2835: & & \\
2836: On-line coincidence & \begin{minipage}{4.5cm}{\mbox{2004.06.25 - 2005.03.01 ,} 2006.05.20 - 2006.08.08}\end{minipage} & two cameras \\
2837: & & \\
2838: \hline
2839: & & \\
2840: On-line confirmation on next image & \begin{minipage}{4.5cm}{2005.03.12 - 2005.08.09 , 2006.05.20 -}\end{minipage} & single camera \\
2841: & & \\
2842: \hline
2843: Off-line nova on aver20 & 2006.07.21 - & \\
2844: \hline
2845: Off-line flare on aver20 & 2006.07.21 - & \\
2846: \hline
2847: \end{tabular}
2848: % \end{small}
2849: \caption{Periods in which different algorithms were used}
2850: \label{tab_algo_periods}
2851: \end{center}
2852: \end{table}
2853:
2854:
2855: %%% ----------------------------------------------------------------------
2856:
2857:
2858: %%% Local Variables:
2859: %%% mode: latex
2860: %%% TeX-master: "../thesis"
2861: %%% End:
2862: