Data Reduction Procedures Python Data reduction can be done in a number of ways using different software packages and tools. Python is being used more and more frequently for astronomical data reduction, so some basic tools for doing basic CCD Direct Imaging data reduction. Below are some general notes before getting started with detailed directions on how to reduce Lick Observatory data with python. 1. Back up raw data. Often one makes mistakes in data reduction or wants to try a different method or procedure, so you should make sure the original data are preserved. 2. To start python type python at the command prompt (in linux this will be an xterm window; in MacOS this will be a terminal window in XQuartz or Terminal; in Windows use Cygwin or similar). 3. We will use DS9 for image display rather than python image display tools, as that is designed for astronomical images and used by most astronomers in the USA. Is it advisable to take a quick look at the images after each step to make sure they look as expected. The tutorial below will walk you through reducing direct imaging data from the Nickel telescope at Lick Observatory. The procedures are generally applicable to typical CCD images. Overscan subtraction For Lick data we have a script, overscanlickobs.py (can be downloaded from http://mtham.ucolick.org/techdocs/instruments/nickel_direct/obs_hints/overscanlickobs.py) that reads the FITS headers of a list of images to get the image and overscan regions of the detector and performs the fit and subtraction of the overscan from each image and writes out a new overscan subtracted image. You can look at the script to see the details of what python is doing and as a guide to writing custom python scripts for data from other observatories. You may have to alter the first line of the script with the location of your python executable file. First a list of image files must be created. This is easily done with the following command in a terminal window: ls *.fits > allfiles.list This will be the list of input FITS files to overscanlickobs.py. Next a list of output files should be created so you don t overwrite the original files. For example, cp allfiles.list allfiles_os.list
The os indicates overscan subtracted file names, and you should edit allfiles_os.list with your preferred text editor to change the output file names, e.g. d100.fits to d100os.fits. To perform the overscan subtraction simply type the command: overscanlickobs.py f i allfiles.list o allfiles_os.list You can look at Figures 3 and 4 to see sample raw and overscan subtracted data images. Starting python For the rest of this tutorial, we ll work within the python environment using individual python commands with occasional commands done in at the terminal prompt. To start python type the following at the command prompt python Next we ll want to import the routines that will be used during the data reduction: from astropy.io import fits This command will import the fits file reading procedures from the astropy python package. import numpy as np will import all the numpy package procedures, which are handy procedures such as median and you can call them by typing np. before the procedure name (e.g. np.median). Combine Bias Frames Again it is easiest to make a list of bias file names rather than reading in every one individually at the command line in python. For example, if the overscan subtracted bias frame file names are d100os.fits through d109os.fits you can do at the terminal prompt ls d10?os.fits > bias.list or use a text editor to create a list of bias file names. In python you can read the file names into a list: biasfiles = [line.rstrip( \n ) for line in open( bias.list )] And then read the images into a data stack (note that indentation is important in python when doing for loops and similar operations). data_stack = [] for file in biasfiles: data_stack.append(fits.getdata(file))
Next the bias images need to be median combined to create a final bias frame to subtract from the overscan subtracted flat fields and data. The following command will median combine the image stack along the appropriate axis (in this case axis=0): medianbias = np.median(data_stack,axis=0) It is good practice to put information into the header concerning the actions taken on the data. Hence we want to add a HISTORY keyword to the combined bias frame FITS header. To create a header for the combined bias frame, let s first read in the header for one of the overscan subtracted bias frames header = fits.getheader(biasfiles[0]) then add a history comment to the header header[ HISTORY ] = Median combined Finally, to write the combined bias frame to a file use the following command fits.writeto( bias.fits,medianbias,header) Bias Subtraction Next the combined bias frame needs to be subtracted from the flat field and data images. Again it is easiest to make a list of all the data frames, and editing an existing data file list is expedient, e.g. cp allfiles_os.list datafiles_os.list then use a text editor to remove the bias frames from the list (they have been combined and there is no need to do any further processing on them). Next make a list of output file names for the bias subtracted data, e.g. datafiles_bs.list. Read the lists into python: datafilesin = [line.rstrip( \n ) for line in open( datafiles_os.list )] datafilesout = [line.rstrip( \n ) for line in open( datafiles_bs.list )] n = len(datafilesin) for i in range(0,n): data,header = fits.getdata(datafilesin[i],header=true) dataout = data - medianbias header['history'] = 'Bias subtracted' fits.writeto(datafilesout[i],dataout,header) Combine Flat Fields Using a text editor, make lists of bias subtracted flat field frames for each filter, e.g. bflat.list, vflat.list, etc.
Read in each list of files: bflatfiles = [line.rstrip( \n ) for line in open( bflat.list )] vflatfiles = [line.rstrip( \n ) for line in open( vflat.list )] rflatfiles = [line.rstrip( \n ) for line in open( rflat.list )] Next make an image stack for each filter s flat field frames and scale each image by its median. We need to scale each image by its median because twilight flats have different illumination levels for each frame as the sky changes brightness, but to median combine them together to beat down noise, they need to have similar count levels between images. bflat_stack = [] for file in bflatfiles: data,header = fits.getdata(file,header=true) data = data / np.median(data) bflat_stack,append(data) The image stack can then be median combined and normalized by the mean so we can measure the pixel to pixel variation in sensitivity of the detector. bflat = np.median(bflat_stack,axis=0) m = np.mean(bflat) bflat = bflat/m Finally, write out the combined, normalized flat field frame: header[ HISTORY ] = Combined and normalized flat field fits.writeto( bflat.fits,bflat,header) Repeat the above for the other filters. Flat Field Data Make a list of all bias subtracted data frames for each filter in a text editor, e.g. bdata.list, vdata.list, etc. Make similar lists with output file names, e.g. bdata_out.list, vdata_out.list, etc. In python read in the lists: bdatain = [line.rstrip( \n ) for line in open( bdata.list )] bdataout = [line.rstrip( \n ) for line in open( bdata_out.list )] Now divide each image by the appropriate filter s normalized flat field: n=len(bdatain) for i in range(0,n):
data,header = fits.getdata(bdatain[i],header=true) dataout = data / bflat header[ HISTORY ] = Flat Fielded fits.writeto(bdataout[i],dataout,header) Repeat above for the other filters. Fix Bad Columns and Pixels Many detectors have known bad columns or pixels. Replacing bad pixels with the mean of the surrounding good pixels is a typical technique to correct these flaws. In this example we ll create a bad pixel mask and fill in those pixels with the median of the surrounding good pixels. In Nickel telescope data the known bad columns are 255, 256, 783, 784, 1001, 1002 (note, python starts counting at 0, whereas DS9 and IRAF start counting at 1, so in DS9 the bad columns will show as being +1 of what is listed here). Create the mask with the following command mask = np.ma.make_mask(data,copy=true,shrink=true, dtype=np.bool) and set all pixels to unmasked: mask[:,:] = False Now set the bad columns to be masked. Note that we can indicate ranges of columns and that in python the first number of the range is inclusive, whereas the last number is exclusive. mask[:,255:257] = True mask[:,783:785] = True mask[:,1001:1003] = True Now read in a data file and header, for example data,header = fits.getdata( d143ff.fits,header=true) Next we mask the data and set the bad pixels to NaN: mdata = np.ma.masked_array(data,mask=mask,fill_value=np.nan) Now we can loop through the masked data to identify NaN values and replace with the mean of the surrounding pixels. This is a slow way to do it, by going one pixel at a time and there are likely more efficient ways to do this. Copy data to new array datafixed = data.copy() and set box size for area around bad pixel to be averaged
s=2 The following will loop through each pixel and account for the edges of the array for i in range(0,mdata.shape[0]): for j in range(0,mdata.shape[1]): if np.math.isnan(mdata[i,j]): x1 = i-s x2 = i+s+1 y1 = j-s y2 = j+s+1 if x1<0: x1 = 0 if x2>mdata.shape[0]: x2=mdata.shape[0] if y1<0: y1 = 0 if y2>mdata.shape[1]: y2 = mdata.shape[1] datafixed[i,j] = np.mean(mdata[x1:x2,y1:y2]) Now that the bad pixels are replaced, write it to disk, for example: header[ HISTORY ] = Bad pixels replaced fits.writeto( d143fix.fits,datafixed,header) As an exercise for the student, write a procedure to do this for every data image. Cosmic Ray Removal There is a package known as astroscrappy that does cosmic ray identification and removal. You can install it from the command line with pip install astroscrappy Note that on Macs you might need to use sudo pip install astroscrappy and on linux you might need root permission to install python packages with pip. To use it in python type import astroscrappy The function detect_cosmics will accept a bad pixel mask so you can replace bad pixels and cosmic rays at the same time, rather than doing the loop above to replace the bad pixels first. Use
the following command (which assumes many defaults about the detector and noise properties, but those defaults are usually fine for Nickel telescope data): crmask,datacr = astroscrappy.detect_cosmics(datafixed,inmask=mask,cleantype= medmask ) And write out the data to a file: header[ HISTORY ] = Cosmic rays replaced fits.writeto( d143final.fits,datacr,header) As an exercise for the student, write a script to do this for all the data images. Finally, to exit python, type exit() or Ctrl-D. Python Resources and Tutorials Python4Astronomers http://python4astronomers.github.io/intro/intro.html AstroPython Tutorials http://www.astropython.org/tutorials/ astropy Tutorials http://www.astropy.org/astropy-tutorials/ Python for Astronomers http://www.iac.es/sieinvens/siepedia/pmwiki.php?n=howtos.empezandopython
IRAF and IDL Data reduction can be done in a number of ways using different software packages and tools. For the sake of examples, IRAF is predominantly shown, as this package is free and widely used. IDL is also frequently used, but can be expensive to purchase. The examples given are not meant to describe the only way to do things, but are just one example of how to process data. CCD Direct Imaging 1. Back up raw data. Often one make mistakes in data reduction or wants to try a different method or procedure, so you should make sure the original data are preserved. 2. Overscan subtraction For Lick data you can use IDL script ccd2bias.pro (or similar, instrument dependent) e.g. ccd2bias, d100.fits, d100new.fits IRAF: colbias e.g. colbias d100 d100new bias=[1026:1052,*] trim=[1:1025,*] function=spline3 Things get more complicated if two or more amplifiers for the CCD. The IDL script mentioned above are specific to Lick instruments, but can be easily modified for other CCDs. 3. Bias subtraction Median combine bias images IRAF: imcomb e.g. imcomb bias1,bias2,bias3 BIAS combine=median Note: for near IR or high dark current detectors, bias subtraction is replaced with dark subtraction. Subtract bias from data IRAF: imarith e.g. imarith @datalist - BIAS @datalist 4. Create Flat Fields Twilight flats generally have different count levels in each frame, so it is typical to scale each frame by the mode (can also scale by median)
Median (or mean) combine flats, rejecting high and low values if you have enough flat field frames. IRAF: imcomb e.g. imcomb flat1,flat2,flat3,flat4,flat5,flat6,flat7 FLAT combine=average scale=mode reject=minmax Normalize combined flat field by mean. IRAF: imstat & imarith Find mean of FLAT: e.g. imstat FLAT # IMAGE NPIX MEAN STDDEV MIN MAX FLAT.fits 38656 24516.7 100.4 20431. 27106. Divide FLAT by mean: e.g. imarith FLAT / 24516.7 FLAT 5. Flat field data frames IRAF: imarith e.g. imarith @datalist / FLAT @datalist 6. Fix bad pixels This step is optional, but often a CCD will have bad columns or rows that need to be corrected. IRAF: fixpix e.g. fixpix sd128 badpix where badpix is a text file containing the regions of bad pixel coordinates in the following format: x1 x2 y1 y2. For CCD2 at the Nickel a typical badpix file has the following contents for the two bad columns: 257 257 1 1024 784 785 1 977 7. Cosmic ray rejection This step is optional, but highly recommended, particularly for long exposure time images. A common way to reject cosmic rays is to take multiple exposures of the same field and combine them. IRAF: crrej
e.g. crrej @imagelist combinedimage 4,4,3,2 If only have a single frame, use IRAF: cosmicrays e.g. cosmicrays d100new d100cr fluxratio=2. threshold=25. interactive+ 8. Analyze data At this point your data images are ready for analysis. Depending on what you are doing will determine what data packages or tools you will need. Handy tools for quick photometry and astrometry are: IRAF: imexam IDL: atv Spectroscopy 1. Backup raw data (as above) 2. Overscan subtraction (as above) 3. Bias subtraction (as above) 4. Create flat fields Median combine flat fields (as above) Spectral flat fields are combination of CCD and lamp response. IRAF: response (in twodspec, longslit package) e.g. response bflat bflat BFLAT order=6 5. Flat field data frames (as above) 6. Wavelength calibration Identify arc lamp lines IRAF: identify e.g. identify r100 coordli= linelists$idhenear.dat e.g. identify b100 coordli=/data/gemini/spectra/cdhgar.dat NB: identify is an interactive procedure with many options which are too extensive to cover here. Reidentify lines for 2D spectra: IRAF: reidentify e.g. reidentify r100 r100 step=10 nsum=10 nlost=2 verbose=yes
Fit coordinates: IRAF: fitcoords e.g. fitcoords r100 functio=chebyshev xorder=1 yorder=1 NB: you will probably want to up xorder and yorder as necessary in interactive mode. Note, you can use the same procedures to trace bright star spectrum: IRAF: identify, reidentify, and fitcoords Use identify, etc. to trace middle column instead of middle line e.g. identify r134 nsum=10 section= middle column coordli= e.g. reidentify r134 r134 step=10 nsum=10 nlost=2 verbose=yes section= middle column e.g. fitcoords r134 function=chebyshev xorder=2 yorder=1 7. Apply wavelength solution to data IRAF: transform e.g. transform @datalist @datalist fitnames=r100 You can choose to not do this step to apply the wavelength solution after the spectra are extracted (step 7) using the IRAF task dispcor (when using dispcor, you do not need to use fitcoords in step 5). 8. Extract spectra IRAF: apall (in twodspec, apextract package) e.g. apall r145 NB: Like identify this is an interactive program with many options which are too extensive to go over here. 9. Examine spectra IRAF: splot (onedspec package) e.g. splot r145.ms 10. Flux calibrate There are many different ways to do this, so I won t go in to detail, but the IRAF tasks standard and fluxcalib are often used. 11. Remove telluric absorption As with flux calibration, there are many tasks to do this. IRAF procedures telluric and skytweak are useful.
12. Analyze data At this point the spectra are ready to be analyzed, which won t be covered here. Possible analysis would be measuring redshift, equivalent widths, de-blending lines, etc. depending on the science objectives. IRAF s splot can do many of these funtions Examples of Data at various stages of data reduction Direct Imaging Figure 1: Raw Bias Figure 2: Raw Flat Field Raw frames show overscan region (rightmost columns), bad columns, dust motes, cosmic rays, vignetting, and other chip defects (Figures 1 thru 3). Overscan and bias subtraction remove the baseline count level and structure of the CCD. Other defects still remain (Figure 4).
Figure 3: Raw Data - M106 Figure 4: Overscan and Bias Subtracted Flat fielding adjusts for pixel by pixel differences in sensitivity, either intrinsic to the device or caused by dust on the detector, dewar window, filters, or other optics in the light path. Figure 5 shows the flat fielded data frame. Figure 5: Flat Fielded Data Figure 6: Final Data After flat fielding, images still have problems such as cosmic rays and bad columns. Using IRAF routines such as cosmicrays, fixpix, and imedit can remove the remaining bad pixels or regions and yield a clean final data image ready for analysis.
Spectroscopy Figure 7: Raw Arc Spectrum Figure 8: Raw Spectral Fla Figure 9: Raw Spectral Data Figure 10: Final Spectral Flatfield
Figure 11: Flat Fielded Data Figure 12: Identified arc lines for wavelength solution Figure 13: Extracted Spectrum
Figure 14: Final Flux Calibrated and Telluric Absorption Corrected Spectrum Other data reduction packages IDL MIDAS Starlink (GAIA, ORAC-DR, etc) Dr. Prochaska (UCSC) has written an IDL package called XIDL to do reductions of spectral data for a number of instruments, including Lick s Kast Double Spectrograph and Keck s HIRES and LRES. http://www.ucolick.org/~xavier/idl/