Table of contents

• Introduction
• Important keywords
• Running the programs
  • Direct execution
  • Using shell scripts
  • Multiple execution with RUN
• Handling error images
• General reduction process
  • Data files involved in the reduction process
  • Reading the tape
  • Reduction of BIAS frames
  • Reduction of DARK frames
  • Reduction of FLATFIELD frames
  • Reduction of ARC frames
  • Reduction of STANDARD frames
  • Reduction of OBJECT frames
• Image format
• Auxiliary libraries

Introduction

Important keywords

Running the programs

Direct execution

*******************************************************************************
REDUCEMEv3.0                   Welcome to istat        Version: 7-December-1996
-------------------------------------------------------------------------------
 
Work with error images (y/n) [n] ? y
Input file name? file001.u
>>> NSCAN :    218
>>> NCHAN :   1024
>>> STWV  : 3740.
>>> DISP  : 2.9000001
>>> OBJECT: X1+56-12
Input error file name [file001e.u] ? file001e.u
>>> NSCAN :    218
>>> NCHAN :   1024
>>> STWV  : 3740.
>>> DISP  : 2.9000001
>>> OBJECT: X1+56-12 @ERROR@
Valid region is: 1,218
1st & Last Scan (0,0=EXIT) [0,0] ? 130,200
1st & Last Scan (0,0=EXIT) [0,0] ? 0,0
Valid region is: 1,1024
1st & Last Channel (0,0=EXIT) [0,0] ? 1,1024
1st & Last Channel (0,0=EXIT) [0,0] ? 0,0
Thinking... ...OK!
 
* Number of pixels employed/Total number: 72704/223232
> Maximum.........................: 5725.73096     in pixel (x,y): 635,130
> Minimum.........................: 0.             in pixel (x,y): 1,130
> Mean............................: 941.285767
> Sigma...........................: 440.506714
> Median..........................: 872.831665
> Mean (removing pixels > 3 sigma): 890.916565
 
* Signal-to-noise ratio:
> Mean............................: 27.616251
> Sigma...........................: 6.31950378
NOTE: this last statistic is uncertain (there are pixels with errors <= 0)
 
(1) change file
(2) change limits
(0) STOP
Option [0] ? 0

Using shell scripts

It is also possible to use shell scripts to run any program. A simple script file will be of the form:

program_name<<string
parameter1
parameter2
..........
..........
string

For instance, the following script file will run the program istat in the same way than in the above example:

istat<<endistat
y
file001.u
file001e.u
130,200
0,0
1,1024
0,0
0
endistat

% source istat.com
...

You can also give execute access to the script file and run it directly:

% chmod +x istat.com
% ./istat.com

istat<<endistat
y
$1

130,200
0,0
1,1024
0,0
0
endistat

% istat.com file001.u

istat<<endistat
y
$1

$2
0,0
$3
0,0
0
endistat

% istat.com file001.u 200,203 1,1024
...
% istat.com file013.u 11,80 1,1024
...
% istat.com file034.u 54,119 101,101
...

However, there is an important problem related to the use of shell scripts in the running of programs. This is the fact that all the program inputs are invisible to the user (they are not printed out in the screen). We have been able to handle this problem in a subtle way. All the inputs in REDUCEME programs can be printed out immediately (like an echo) if there is a file in the current directory called .running_RUN. You only need to create a file with this name (no matter its content), and all the programs inputs will be displayed during the execution of any shell script. In the following tables we summarized the special forms of a variable (in the C shell), filename modifiers, comparison operators and file comparison operators, which are very useful in the building of shell scripts:

Special forms of the C shell variables

$?name	returns 0 if variable name is not defined, 1 if defined
$#name	returns integer specifying the number of elements of variable name
$name[m-n]	returns elements m through n of variable name, where m and n are integers
$name[m-]	returns elements m through the last element of variable name, where m is an integer
$name[*]	returns all elements of variable name
$name[$]	returns last element of variable name
${0}	returns name of script being executed
$?0	returns 0 if name of script being executed is known, 1 if not
$a ($argv[a])	returns ath variable passed to script
$*	returns all variables passed to script
$$	returns process number of current shell
$<	takes input from terminal keyboard

Mathematical operators (Integers) + add - subtract * multiply / divide ++ increment by 1 -- decrement by 1 % modulo

Filename modifiers :r returns portion of filename preceding a period :e returns portion of filename following a period :h returns head (path) of file specification :t returns tail of file specification, excluding path

Mathematical operators (Bits) >> right bit shift << left bit shift ~ 1's complement ! logical negation | inclusive or ^ exclusive or & and

Comparison operators = assign value to variable == equal to ! boolean not, for example not equal to && boolean and || boolean or > greater than < less than >= greater than or equal to <= less than or equal to

File comparison operators -d true if file is a directory -e true if file exists -f true if file contains text -o true if executor of file is owner -r true if file is readable by executor -w true if file is writable by executor -x true if file is executable by executor -z true if file is empty

Multiple execution with RUN

With the aim of simplifying the reduction process, specially when dealing with a program (or set of programs) which is going to be used repeatedly over many images, we have created a special shell script call RUN to run any script file with all the selected images of a fixed directory.

In order to use the script RUN, it is necessary to create, previously, the script file that is going to be executed many times. For example, we could use the file istat.com already described:

istat<<endistat
y
$1

130,200
0,0
1,1024
0,0
0
endistat

% istat.com file001
...
% istat.com file002
...
...
...
% istat.com file0??
...

% RUN istat.com
#> Script file ./istat.com found. OK.
#> Enter file specification.............: file0??
#> Verify before running ([y]/n)........: n
...

...
#> End of script /usr/local/reduceme/bin/RUN

...
#> Verify before running ([y]/n)........: y

Number of files found with current specifications: 14

File No. 1 out of 14
--> FILE FOUND ------------------------> ? file001
Image size (NSCAN,NCHAN): 218,1024
Object  : X1+56-12
#> Run program istat.com with this file ([y]/n/x/g)? y
#> Work in progress...
...

As you probably can imagine, this is not the end of the story. The script file RUN accepts more than one single argument in the command line. For example

% RUN istat.com 1,200 1,1024

% istat.com file??? 1,200 1,1024

istat>>endistat
y
$1

$2
0,0
$3
0,0
0
endistat

Handling error images

Anyway, it is important to note that this convention is not mandatory, and error file names can be chosen arbitrarily (although in this last case, the default values for the error files will not be of any use).

General reduction process

Data files involved in the reduction process

The reduction of spectroscopic data requires, apart from images of the objects of astronomical interest, the utilization of calibration frames. Basically, we can summarize the different types of images (commonly employed in the reduction process) in the following list:

• BIAS: frames with no illumination (exposure time = 0 sec), to determine a possible stationary, two-dimensional, additive background pattern (which is repeated in each CCD readout).
• DARKS: frames with no illumination but with exposure times > 0 sec, to measure the background level resulting from thermally created electrons.
• FLATFIELDS: frames obtained by observing a uniform light source (typically a tungsten lamp or the telescope dome). However, since the illumination of the slit by a lamp might no be uniform, sky frames (usually twilights or, preferably, although time demanding, night-sky exposures) are also needed. Lamp and dome flatfields are employed to correct pixel-to-pixel differential response, whereas twilight and night-sky flatfields are used to compensate for two-dimensional low-frequency scale sensitivity variations on the chip.
• ARCS: lamp spectra for wavelength calibration and correction of C-distortion.
• STANDARDS: observations of spectrophotometric standards to perform the absolute/relative flux calibrarion (this type of observation is not always necessary, depending on the science requirements of the observations).
• OBJECTS: observations of science objects, aim of the observations.

Download the gzipped postscript version of this figure from here.

Reading the tape

The first step in the reduction process is usually the reading of FITS files from a tape. For this purpose, REDUCEME includes a shell script, called read_tape_fits.com, to perform easily this action:

% read_tape_fits.com
Number of files to be skipped...................[0]:
First file number (0 if first file is not an image): 0
Enter maximum number (max. 999)....................: 300
Enter tape device................(e.g. /dev/nrmt0a): /dev/nrmt0a
Enter root name for file names...............[file]:
Rewind tape before reading......................[y]: n
Dismount tape after reading.....................[n]: y
...

If the total number of FITS files stored in the tape is less than the maximum number introduced in the script at running time, the last files (from the real maximum number+1 to the expected maximum number) will also be created, but with a null size.

If the number of FITS files to be read are greater than 999, you must change the script file (it's easy!) or run the script file several times with different root names (i.e. with a prefix equal to file12, you will obtain file names like file12001, file12002,..., file12999).

Once all the required files have been loaded, it is interesting to run the program fitshead which creates an ASCII output file containing the information of some important FITS keywords.

In addition, the program outfitshead outputs the full header of a FITS image into a text file.

Finally, before starting the reduction process, it is very useful to create different directories (e.g. ../bias/, ../flats/, ../cuar/, ../stand/, ../templ/, ../gal/, ...) and move each image into its corresponding directory.

Reduction of BIAS frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Remove cosmic rays:
   cleanest or autocos: file000.u --> file000.uc
4. Determine underscan and overscan regions. For this purpose, it is convenient to use the program plotsplus with flatfield images.
5. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), and extract the useful region of the images:
   basicred: file000.uc --> file000.ucb
6. If the total number of BIAS frames obtained for each night is large, obtain the normalized added frame (addnf). If a two-dimensional structure is apparent, fit it with a smooth surface by using a filter (ifilter), a polynomial surface (fit2dpol) or a composite polynomial/spline surface (fit2dspl).
7. Obtain the residual frame (using imath if necessary), measure the standard deviation (istat) around the mean (that must be zero!) and obtain the readout noise as this standard deviation times the square root of the number of indiviual BIAS frames added.

Reduction of DARK frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Remove cosmic rays:
   cleanest or autocos: file000.u --> file000.uc
4. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), the fitted two-dimensional BIAS surface, and extract the useful region of the images:
   basicred: file000.uc --> file000.ucb
5. If a two-dimensional structure is apparent, fit it (for each night?) with a smooth surface by using a filter (ifilter), a polynomial surface (fit2dpol) or a composite polynomial/spline surface (fit2dspl).
6. Determine the dark current (constant value, with imath, or two-dimensional surface, with ifilter, fit2dpol or fit2dspl).

Reduction of FLATFIELD frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Remove cosmic rays:
   cleanest or autocos: file000.u --> file000.uc
4. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), the fitted two-dimensional BIAS surface, the dark current (constant or two-dimensional surface), and extract the useful region of the images (if desired, error frames for lamp/dome flatfields can also be generated --gain and readout noise must be known--):
   basicred: file000.uc --> file000.ucb
5. Average flatfields for each observing night:
   addnf: file000.ucb, file000e.ucb,..., file???.ucb, file???e.ucb --> flatn?, flatn?e
   where the question mark in the flat file name indicates the night number.
6. Normalize the averaged lamp/dome flatfields of each night by dividing the original frame by the averaged spectrum:
   adnsc: flatn?, flatn?e --> flatn?.sx, flatn?e.sx
   imath: flatn?, flatn?e, flatn?.sx, flatn?e.sx --> flatn?.nor, flatn?e.nor
7. Divide twilight/sky flatfields by the normalized lamp flatfields.
8. Twilight flatfields can be normalized using the same procedure than with lamp flatfields. However, if C-distortion is apparent, the normalization can be obtained using nortwi.
9. Obtain a smooth two-dimensional surface by fitting the normalized twilight flatfields with ifilter, fit2dpol or fit2dspl).

Reduction of ARC frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), the fitted two-dimensional BIAS surface, the dark current (constant or two-dimensional surface), divide by the corrsponding lamp flatfield, reverse the spectra in the wavelength direction (if required), and extract the useful region of the images (error frames are not used in the reduction of arc frames):
   basicred: file000.u --> file000.ub
4. Remove cosmic rays:
   cleanest: file000.ub --> file000.ubc
5. Correct C-distortion:
   • Obtain one averaged arc spectrum:
     adnsc: file000.ubc --> file000.sx
   • Search line peaks automatically:
     findmax: file000.sx --> lincdis.dat
   • Run fitcdis and remove from the file lincdis.dat the arc lines which are not well suited for the fit.
   • Obtain one averaged arc spectrum for each available arc frame:
     adnsc: file000.ubc --> lincdis.sx
   • Determine the relative offsets between the first averaged arc spectrum (from which one have obtained the file lincdis.dat) and all the rest, with the help of the program corrfft. Note that the offsets measured with this program must be rounded to an integer and change their sign before introducing them in fitcdis.
   • Running fitcdis and cdisc, create the polynomial for the C-distortion correction:
     fitcdis: lincdis.dat, file000.ubc --> file000.cdis1
     cdisc: file000.ubc, file000.cdis1 --> file000.c1
     fitcdis: lincdis.dat, file000.c1 --> file000.cdis2
     cdisc: file000.c1, file000.cdis2 --> file000.c2
     fitcdis: lincdis.dat, file000.c2 --> file000.cdis3
     cdisc: file000.c2, file000.cdis3 --> file000.c3
     ...
6. After the correction of the C-distortion, all the spectra of each corrected frame can be added:
   adnsc: file000.c? --> sp000.sx
7. Wavelength calibration:
   • Identify all the visible lines:
     findarc: sp000.sx --> fitlin.dat
   • Perform the wavelength calibration:
     fitlin: fitlin.dat, sp000.sx > pol000
8. Determine the spectral resolution by fitting gaussians (with plotsplus) to different, well-isolated arc lines.

Reduction of STANDARD frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), the fitted two-dimensional BIAS surface, the dark current (constant or two-dimensional surface), divide by the corresponding lamp/dome and twilight/sky flatfields, reverse the spectra in the wavelength direction (if required), and extract the useful region of the images (error frames are not used in the reduction of standard star frames):
   basicred: file000.u --> file000.ub
4. Remove cosmic rays:
   cleanest: file000.ub --> file000.ubc
5. Wavelength calibration, C-distortion and radial velocity correction:
   rebincw: file000.ubc, file000.cdis?, pol000,, radial velocity --> file000.w
   If arc frames are not available, one can use wcnoarc (note that in this case, an intermediate sky subtraction could be required if sky lines are present): wcnoarc: file000.ubc, pol???, radial velocity --> file000.w
6. S-distortion correction:
   sdistor: file000.w --> file000.wsd
7. Atmospheric (and interstellar?) extinction correction:
   corrext: file000.wsd, airmass --> file000.wsde
8. Sky subtraction:
   skysubm: file000.wsde --> file000.wsdek
9. Obtain the flux calibration curve for each standard (the tabulated flux calibration curve must be known):
   fcalspl: file000.wsdek, table --> file000.res
10. If there are more than one calibration curve, it is useful to create an image with the average and all the individual flux calibration curves:
    prfcal: file000.res,...,file???.res --> curvresf

Reduction of OBJECT frames

1. Create a log file with fitshead and, if necessary, a LaTeX table with fitstex.
2. Transform FITS images to REDUCEME format:
   readfits: file000 --> file000.u
3. Subtract the BIAS value (mean, median,...) measured in the underscan and/or overscan region(s), the fitted two-dimensional BIAS surface, the dark current (constant or two-dimensional surface), divide by the corresponding lamp/dome and twilight/sky flatfields, reverse the spectra in the wavelength direction (if required), and extract the useful region of the images (if desired, error frames for objects can also be generated --gain and readout-noise must be known--):
   basicred: file000.u --> file000.ub
4. Remove cosmic rays:
   cleanest: file000.ub --> file000.ubc
5. Wavelength calibration, C-distortion and radial velocity correction:
   rebincw: file000.ubc, file000.cdis?, pol000,, radial velocity --> file000.w
   If arc frames are not available, one can use wcnoarc (note that in this case, an intermediate sky subtraction could be required if sky lines are present): wcnoarc: file000.ubc, pol???, radial velocity --> file000.w
6. S-distortion correction:
   sdistor: file000.w --> file000.wsd
7. Atmospheric (and interstellar?) extinction correction:
   corrext: file000.wsd, airmass --> file000.wsde
8. Sky subtraction:
   skysubm: file000.wsde --> file000.wsdek
9. Extract final spectra (snratio, gluesc), and measure indices (and errors?) with index, midelines.

Image format

• Program simple_read: read a data file with REDUCEME format.
• Program simple_write: write a data file with REDUCEME format.

	PROGRAM SIMPLE_READ
	IMPLICIT NONE
C
	INTEGER NCHAR
	INTEGER NSCAN,NCHAN
	REAL STWV,DISP
	REAL AIRMASS,TIMEXPOS
	REAL SPECTRA(1124,1124)
	CHARACTER*12 IDENTIFICATION
	CHARACTER*255 OBJECT
	CHARACTER*255 FITSFILE
	CHARACTER*255 COMMENT
C
C open file
	OPEN(10,FILE='file000.dat',STATUS='OLD',FORM='UNFORMATTED')
C read header information
	READ(10) IDENTIFICATION
	READ(10) NSCAN,NCHAN
	READ(10) STWV,DISP
	READ(10) AIRMASS
	READ(10) TIMEXPOS
	READ(10) NCHAR
	IF(NCHAR.GT.0) READ(10) OBJECT(1:NCHAR)
	READ(10) NCHAR
	IF(NCHAR.GT.0) READ(10) FITSFILE(1:NCHAR)
	READ(10) NCHAR
	IF(NCHAR.GT.0) READ(10) COMMENT(1:NCHAR)
C read data frame
	DO I=1,NSCAN
	  READ(10) (SPECTRA(J,I),J=1,NCHAN)
	END DO
	CLOSE(10)
C end of program
	STOP
	END

It is important to note that the data array SPECTRA must be dimensioned at least to the maximum expected NCHAN and NSCAN.

In order to distinguish data frames from their associated error frames, we have decided to add the extra character string ' @ERROR@' (CHARACTER*8) to the OBJECT header keyword. For example:
data file:file000.dat, with OBJECT='NGC 9999' --> error file:file000e.dat, with OBJECT='NGC 9999 @ERROR@'

Auxiliary libraries