1: % Cosmic Rays Removal
2: % Version 2003.11.12 W. Pych
3:
4: %\documentclass{aastex} % aastex5
5: \documentclass[preprint]{aastex}
6: %\documentclass[preprint2]{aastex}
7:
8: \shorttitle{Cosmic rays removal}
9: \shortauthors{Pych}
10:
11: \begin{document}
12:
13: \title{A Fast Algorithm for Cosmic Rays Removal from Single Images}
14:
15: \author{Wojtek~Pych}
16: \affil
17: {David Dunlap Observatory, University of Toronto \\
18: P.O.~Box 360, Richmond Hill, Ontario, Canada L4C 4Y6 \\
19: and \\
20: Copernicus Astronomical Center \\
21: Bartycka 18, 00--716 Warszawa, Poland}
22:
23: \email{pych@camk.edu.pl}
24:
25: \begin{abstract}
26:
27: We present a method for detecting cosmic rays in single images.
28: The algorithm is based on simple analysis of the histogram of the image
29: data and does not use any modeling of the picture of the object.
30: It does not require a good signal to noise ratio in the image data.
31: Identification of multiple-pixel cosmic-ray hits is realized by running
32: the procedure for detection and replacement iteratively.
33: The tests performed by us, show that the method is very effective, when
34: applied to the images with the spectroscopic data. It is also very fast
35: in comparison with other single image algorithms found in astronomical
36: data processing packages.
37: Practical implementation and examples of application are presented.
38:
39: \end{abstract}
40:
41: \keywords{ astronomical images: reduction methods -- cosmic ray: detection }
42:
43: \section{INTRODUCTION}
44: \label{sec1}
45:
46: Cosmic ray hits cause defects in all astronomical images obtained
47: with CCD detectors.
48: A relatively efficient approach to removing traces of cosmic rays
49: from such images is to use multiple frames of the same object and then
50: combine them with some algorithm for rejection of the outlying data.
51: Methods of this type have been
52: presented among others by \citet{Shaw1992} and \citet{Windhorst1994}.
53: They may be found in most astronomical data processing packages.
54:
55: In many cases we are not able to obtain multiple images of the same
56: object or the required time resolution prevented us from using multiple
57: image methods.
58: A straightforward method that could be used in such cases would be sigma
59: clipping. In practice it is almost impossible to find a good detection
60: threshold for such an algorithm. In fact, it may leave some obvious cosmic
61: rays untouched, while giving numerous spurious detections and rejecting
62: valid data at the same time.
63: Other types of single image algorithms may also be found in most
64: astronomical data processing packages.
65: Most of these methods rely on the sharpness of cosmic rays, relative to
66: the atmospheric smoothing of real images. These methods are
67: based on some interactive learning techniques
68: \citep{Murtagh1992,Salzberg1995} or involve a special fitting
69: of a model representing a real image of a star with a superimposed cosmic
70: ray \citep{Rhoads2000}. These methods were designed to correct direct
71: images of the stellar objects and we find them not suitable for
72: spectral images.
73:
74: The exposure times of the spectra are usually longer than
75: direct images and range from 600~seconds to 1800~seconds.
76: Considerable amounts of cosmic ray events accumulate in the images
77: during such exposures.
78: We have constructed a simple and straightforward algorithm for
79: detecting cosmic rays in single images. Our method does not need any
80: model of the shape of the image features themselves.
81: We analyze the histograms of
82: pixel counts in small sub-frames in order to detect pixels deviating by
83: some factor from the bulk of the pixels under consideration.
84:
85: We would like to note here, that
86: low value bad pixels tend to hide weak cosmic rays by artificially making
87: the local variance larger than would be caused only by usual sources of
88: noise in the image: readout noise, dark current and the Poisson
89: noise associated with background and object signal.
90: The cosmic rays have random distributions of their positions in the
91: images. On the other hand, bad pixels may be found as deviating pixels at
92: the same position in every frame taken with a given instrument.
93: Therefore we suggest to make bad pixel correction as a separate step of
94: the reduction process prior to cosmic rays correction.
95:
96: We describe details of our algorithm in paragraph 2. In paragraph 3. we
97: describe an implementation and the results of our tests.
98: In paragraph 4. we present examples of application to real astronomical data.
99: Paragraph 5 contains the summary.
100:
101: \section{The Algorithm}
102: \label{sec2}
103:
104: The proposed algorithm is based on the idea that cosmic rays deposit a
105: portion of their energy in the pixels they hit, causing some extra
106: signal in these pixels. The signal coming from cosmic rays does not have
107: a Gaussian distribution.
108: This should reflect in the distribution of counts in the image affected
109: by cosmic rays.
110: The image as a whole may have a large range of signal levels in different
111: areas. We analyze relatively small sub-frames, to work with
112: a more concentrated local distribution of counts. In most cases this
113: distribution should be rather compact. The cosmic rays appear as points
114: standing out in the high counts interval.
115:
116:
117: \begin{figure}
118: \figurenum{1}
119: \plotone{fig1.eps}
120: \caption{Histogram used in a typical application of
121: the cosmic-ray detection algorithm.
122: The vertical dashed line marks the lower limit of counts regarded as the
123: cosmic rays. The arrows mark the first gap in the histogram of the width
124: larger than the threshold.}
125: \label{fig1}
126: \end{figure}
127:
128: The proposed algorithm of cosmic rays detection generally consists of
129: the following steps:
130: %
131: \begin{enumerate}
132: \item Select small size sub-frames, which cover the whole frame with
133: substantial overlapping.
134: \end{enumerate}
135: In each sub-frame:
136: \begin{enumerate}
137: \setcounter{enumi}{1}
138: \item Calculate standard deviation of the distribution of counts:
139: \begin{equation}
140: \sigma = \sqrt { {\sum{c_i^2} - {(\sum{c_i})^2 \over n}} \over {n} }
141: \end{equation}
142: \item Apply a single sigma clipping step to correct the estimate of
143: standard deviation for outlying pixels.
144: \item Construct a histogram of the distribution of counts.
145: \item Find the mode of the distribution of counts, i.e. the peak of the
146: histogram.
147: \item In the interval of counts higher than the mode,
148: find gaps in the histogram i.e. bins with zero data points.
149: \item Find the first gap which is wider than a threshold, which is the
150: standard deviation multiplied by an arbitrary number (usually 3.0).
151: \item If such gap exists, flag pixels with counts lying above the gap as
152: affected by cosmic rays.
153: \end{enumerate}
154:
155: Figure \ref{fig1} illustrates an example of a histogram
156: with a few pixels identified as affected by cosmic ray events.
157:
158: The next step consists of a replacement of the count numbers in the
159: flagged pixels. If we consider a single image, the information about the
160: real signal in the affected pixels is lost.
161: In many cases however, the characteristic scale of spacial
162: variations of signal are of the order of at least a few pixels. In such
163: cases, one may use some interpolation to replace missing pixel counts.
164: In our implementation, we decided to substitute the cosmic rays by the
165: average of the counts in the neighboring pixels.
166:
167: Cosmic rays are often multiple-pixel events. High signal pixels may
168: screen neighboring pixels from detection. For this reason
169: we run the procedure of detection and cleaning cosmic rays iteratively.
170: The process is rapidly converging and usually there are no new detections
171: after 2 or 3 iterations. We have also introduced a parameter called
172: "growing radius", which tells the procedure to clean the pixels closer
173: than this radius to the detected one, even when they do not possess
174: enough signal to be flagged as affected by cosmic rays.
175:
176: \section{Implementation and Test}
177: \label{sec3}
178:
179: \begin{figure}
180: \figurenum{2}
181: \plotone{fig2s.eps}
182: \caption{Distribution of counts associated with cosmic rays. Panel A:
183: distribution of counts associated with high energy cosmic rays;
184: Panel B: distribution of counts associated with low energy cosmic rays;
185: Panel C: distribution of counts associated with undetected cosmic rays;
186: Panel D: distribution of residual counts after cosmic ray cleaning. See
187: text for details.}
188: \label{fig2}
189: \end{figure}
190:
191: We have written a computer program, which implements the above described
192: algorithm\footnote{The C-source of our program may be downloaded
193: from the following Web-page: \url{http://www.camk.edu.pl/\~{ }pych/}}.
194: This program has several input parameters which allow the user to control
195: the process of detecting and cleaning cosmic rays.
196: These parameters define:
197: %
198: \begin{itemize}
199: \item
200: the size of the sub-frame box: X and Y dimensions of the rectangle,
201: \item
202: the threshold: the number by which the local variance is multiplied,
203: \item
204: the dispersion axis: the interpolation for bad pixel substitution is
205: calculated along this axis; if no axis selected, it is calculated
206: in the annulus around that pixel,
207: \item
208: the lower and upper radii of the annulus for the interpolation of data,
209: \item
210: the growing radius: the maximum distance of the pixels which are
211: to be corrected in the neighborhood of the flagged pixel.
212: \end{itemize}
213:
214: Our program also creates a map of detected cosmic rays.
215: This map can be examined to check for any correlation with the real
216: data. The existence of such correlation indicates that the
217: detection threshold has been set too low and the real data has been
218: modified.
219:
220: To check the capabilities of our method,
221: we have generated an artificial two-dimensional Echelle spectrum with the
222: size of \mbox{2048 x 4096 pixels}. For this purpose, we have used a procedure
223: from IRAF package: {\it noao.artdata.mkechelle}, with its default parameters.
224: We have also used the procedure {\it noao.artdata.mknoise}
225: to obtain two noise images. The first one consists of readout noise
226: (5 electrons) and a background of 500~counts with Poisson noise. The
227: second image consists of 1000 cosmic rays with maximum energy
228: 30000~electrons.
229:
230: Our program finished with zero detections on the noiseless image of the
231: Echelle spectrum. The result for the spectrum with the readout and Poisson
232: noise added was one detection over the whole frame.
233: Finally, we tested the procedure on a
234: spectrum with Poisson noise and cosmic rays added to the original
235: image of a spectrum. Our program has found all but one cosmic
236: ray event of peak greater than 200 counts.
237: The only event left, and with a peak of 430 counts, was located
238: in the vicinity of a trace of the spectrum and was concealed by this
239: signal.
240:
241: Figure \ref{fig2} presents histograms of the counts distribution.
242: The upper-left panel (A) presents the original distribution of counts
243: from cosmic rays. The Y axis is cut at a value of 10 pixels to highlight
244: the distribution of high energy hits.
245: The upper-right panel (B) presents the distribution of
246: cosmic rays in the interval of small counts.
247: The lower panels present the distribution of counts in the image obtained
248: as a difference between the spectrum image after cosmic rays cleaning
249: and the image without cosmic rays added.
250: The Y-axis in the lower-left panel (C) is cut at a value of 10 pixels,
251: to show the distribution of pixels with undetected cosmic rays. The
252: features visible above 200 counts are the traces of a single undetected
253: cosmic ray event.
254: The lower-right panel (D) presents the distribution of the residual
255: counts. It has the shape of a Gaussian with a small 'high-end' tail,
256: produced by undetected cosmic rays.
257: The standard deviation of the residuals, $\sigma = 26.0$, is similar to
258: the variance of the original image without cosmic rays; $\sigma = 27.8$.
259: This similarity reflects the nature of the adopted substitution method
260: for the pixels affected by cosmic rays, which is simply a mean value
261: of the surrounding pixels.
262:
263: \begin{figure}
264: \figurenum{3}
265: \plotone{fig3s.eps}
266: \caption{Central fragments of our test images.
267: The upper-left panel
268: shows the artificial Echelle spectrum with the Poisson noise
269: and cosmic rays added.
270: The upper-right panel
271: shows the difference between the image cleaned by our program
272: and the original spectrum (without cosmic rays).
273: The lower-left panel
274: presents the same image as in the panel above, but after being
275: processed by the IRAF procedure {\it noao.imred.crutil.cosmicrays}.
276: The lower-right panel
277: shows the difference between the image obtained from
278: the IRAF routine {\it noao.imred.crutil.cosmicrays}.
279: and the original spectrum (without cosmic rays).}
280: \label{fig3}
281: \end{figure}
282:
283: Our tests were conducted on a PC with a 500~MHz Intel Celeron processor.
284: The program was compiled using a GNU C Compiler
285: (gcc version 3.2.2) under Red Hat Linux 9.0.
286: The CPU time needed to process a single frame: 2048~x~4096, 32~bit pixels,
287: was about 40~seconds. The CPU time required to process an image
288: depends linearly on the number of the image pixels.
289:
290: We have also run the IRAF procedure {\it noao.imred.crutil.cosmicrays}.
291: The CPU time needed to process the same image was over 900~seconds.
292: In comparison with the above, our algorithm may be classified as a fast
293: one. The relative speed
294: results from the fact that the whole detection process does not require
295: extensive calculations.
296: Our test demonstrated that the IRAF routine was not able to remove
297: multiple pixel cosmic ray events. They were slightly modified, but
298: most of them remained in the frame.
299: At the same time, a large number of image pixels unaffected by cosmic
300: rays were modified.
301: We would like to stress here that our algorithm does remove cosmic rays
302: effectively while leaving almost all of the image data untouched.
303:
304: Figure \ref{fig3} presents central fragments of our test images.
305: The upper-left panel
306: shows the artificial Echelle spectrum with the Poisson noise and cosmic
307: rays added.
308: The upper-right panel
309: presents the residual signal remaining after subtraction of the original
310: spectrum without cosmic rays from the image cleaned by our program.
311: The lower-left panel
312: presents the image after being processed by the IRAF procedure
313: {\it noao.imred.crutil.cosmicrays}. Note that the cosmic rays were not
314: removed.
315: The lower-right panel
316: shows the residual signal remaining after subtraction of the original
317: spectrum without cosmic rays from the image obtained
318: from the IRAF routine {\it noao.imred.crutil.cosmicrays}.
319: The scales of the images of the residual signals are the same.
320:
321: \section{Examples of practical application}
322: \label{sec4}
323:
324: \begin{figure}
325: \figurenum{4}
326: \epsscale{1.0}
327: \plotone{fig4s.eps}
328: \figcaption{Example of a frame with a long slit spectrum:
329: before cosmic ray cleaning -- left panel;
330: after cosmic ray cleaning -- right panel.
331: The stellar signal is in the white band along each panel.}
332: \label{fig4}
333: \end{figure}
334:
335: \begin{figure}
336: \figurenum{5}
337: \epsscale{1.0}
338: \plotone{fig5s.eps}
339: \figcaption{Example of the application of our method on an Echelle
340: spectrum of a faint star.
341: The upper-left panel
342: shows a fragment of the spectrum of a 17~mag. star,
343: obtained in an 1800~second exposure with MIKE spectrograph
344: at the Las Campanas Observatory.
345: The upper-right panel
346: presents the map of cosmic rays detected in the image. A sky emission
347: line is visible in the center.
348: The lower-left panel
349: presents the same fragment of the spectrum as shown in the upper-left
350: panel, but after subtraction of the map of cosmic rays modified to
351: remove the sky emission line.
352: The lower-right panel
353: shows the map of cosmic rays with the sky emission line removed.}
354: \label{fig5}
355: \end{figure}
356:
357: We have applied the above algorithm to several data sets.
358: Our general experience is that the method works very well on images of
359: both long-slit and Echelle spectra.
360: The algorithm was originally designed to work with the spectroscopic data.
361: Our method could be used to clean images of stellar
362: fields with a very wide Point Spread Function (PSF).
363: It works well in a sense that it does not produce too many
364: spurious detections. The limitation of the presented method for these
365: types of images comes from the fact that direct images usually have
366: large count variations within small scales. This produces a large
367: standard deviation of the counts and prevents cosmic rays in the
368: neighborhood of bright objects from being detected.
369: We suggest using the method proposed by \citet{Rhoads2000} in such
370: cases.
371:
372: Figure \ref{fig4} illustrates an example result of cosmic ray removal in a
373: long-slit spectrum. A fragment of the
374: spectrum of a 9.2~mag. variable star, obtained in a 1200~second
375: exposure with the 1.88-m telescope
376: at the David Dunlap Observatory, University of Toronto, is shown.
377: The left panel
378: presents the original image (after bias subtraction and flat-field
379: correction), while the right panel shows the cleaned image. No evident
380: cosmic rays are identified by eye in the cleaned frame.
381:
382: Our method works best on low signal images.
383: For the faint stars however, we encounter another problem.
384: The night-sky emission lines, which are sharp features perpendicular to
385: the dispersion axis, may sometimes be strong enough to be identified
386: as cosmic rays. Our solution to this problem is to edit the map of cosmic
387: rays, erase features identified as sky emission lines (replace the
388: counts with zeros) and then subtract a modified cosmic ray map from
389: the original image.
390:
391: Figure \ref{fig5} illustrates an example of the application on an Echelle
392: spectrum of a faint star.
393: The upper-left panel
394: shows a fragment of the Echelle spectrum of a 17~mag. star,
395: obtained in an 1800~second exposure with MIKE spectrograph
396: (Magellan Inamori Kyocera Echelle) at the Las Campanas Observatory.
397: The upper-right panel
398: presents the map of cosmic rays detected in the image. A sky emission
399: line is visible in the center.
400: The lower-left panel
401: presents the same fragment of the spectrum as shown in the upper-left
402: panel, but after subtraction of the modified map of cosmic rays. The map
403: was edited to remove the sky emission line.
404: The lower-right panel
405: shows the map of cosmic rays with the sky emission line removed.
406:
407: \section{Summary}
408: \label{sec5}
409:
410: We have presented a cosmic-ray rejection algorithm based on a simple
411: analysis of the histogram of the image data.
412: The most important advantage of our
413: method is that it does not require modeling of the image data and
414: may {\it apriori} be applied to any type of well sampled image data.
415: We have checked that for the spectroscopic images, it is very effective
416: in detecting cosmic rays, while avoiding numerous spurious detections.
417: Our method does not require advanced and extensive computations, so it is
418: relatively fast.
419:
420: The weak point of this approach, is that bright objects may shield cosmic
421: rays in their neighborhood from detection. This is caused by two factors.
422: First: the sensitivity to cosmic ray events is reduced at the locations of
423: bright objects because of the Poisson noise associated with the image
424: photons. Second: we look for bright spots and the legitimate object
425: may be brighter than a nearby comic ray trace.
426:
427: Our suggestion is to use the algorithm presented above for spectroscopic
428: data whenever multiple image methods cannot be employed for cosmic rays
429: removal.
430:
431: Future improvement to the presented method could be an introduction of
432: better interpolation for the replacement of pixels affected by cosmic
433: rays.
434:
435: \acknowledgements
436:
437: The author would like to thank Prof. S. Rucinski, J. R. Thomson and H.
438: DeBond for help in preparing this article.
439: This article has been prepared when W. Pych held the NATO Post-Doctoral
440: Fellowship administered by the Natural Sciences and Engineering Council
441: of Canada (NSERC).
442: The author acknowledges also the support from the Polish Grant KBN
443: 2~P03D~029~23 and the NSERC research grant to S. Rucinski.
444:
445: %
446:
447: \begin{thebibliography}{}
448: \bibitem[Murtagh(1992)]{Murtagh1992}
449: Murtagh, F. D.
450: 1992, Astronomical Data Analysis Software and Systems I, A.S.P. Conference
451: Series, Vol. 25, eds., Diana M. Worrall, Chris Biemesderfer, and
452: Jeannette Barnes, p. 265
453: \bibitem[Rhoads(2000)]{Rhoads2000}
454: Rhoads, J. E.
455: 2000, \pasp, 112, 703
456: \bibitem[Salzberg et~al.(1995)]{Salzberg1995}
457: Salzberg, S., Chandar, R., Ford, H., Murthy, S. K.; White, R.
458: 1995, \pasp, 107, 279
459: \bibitem[Shaw \& Horne(1992)]{Shaw1992}
460: Shaw, R. A., Horne, K.
461: 1992, Astronomical Data Analysis Software and Systems I, A.S.P. Conference
462: Series, Vol. 25, eds. Diana M. Worrall, Chris Biemesderfer, and
463: Jeannette Barnes, p. 311
464: \bibitem[Windhorst, Franklin, \& Neuschaefer(1994)]{Windhorst1994}
465: Windhorst, R. A., Franklin, B. E., Neuschaefer, L. W.
466: 1994, \pasp, 106, 798
467: \end{thebibliography}
468:
469: \end{document}
470: