There are two main file format for IFS final data products, and one further prospective format. The first of these file formats is a MOS style multi-extension FITS file, and is being put forward by the GEMINI group (see Section 4.1). The other main format is an —data cube which is used by the Durham group (i.e. for SMIRFS and TEIFU data) (see Section 4.3).
Of these two formats the more natural for data analysis is the TEIFU style data cube, it has therefore been adopted as the standard format by the major IFS groups in the UK, Durham (SMIRFS and TEIFU), Cambridge (CIRPASS) and the ATC (UIST). Conversion between the GMOS/CIRPASS MEF and UK standard data cube formats(see Section 4.4) will therefore need to be implemented before these instruments are brought into general use. The EOS VIMOS instrument will provide its final data product in both MEF and data-cube formats.
The final IFS format is still in draft, and is the new IRAF spectroscopic file format (see Section 4.2). The specification is intended to provide a general description for two-dimensional spectroscopic image data, and should be able to represent long-slit, multi-object (MOS), integral-field unit (IFU) and slitless spectroscopy. Conversion between this format and the standard data-cube format will be implemented if the standard is adopted.
Currently the final data product of the GMOS and CIRPASS data-reduction software is a multi-extension FITS (MEF) file. However, this format may be replaced by the new IRAF spectral format (see Section 4.2) which is currently in development by the IRAF group at NOAO. The MEF is similar to the standard NIRI format now used with GEMINI, and has a binary FITS table with separate data, variance and quality planes.
|1||BINTABLE||TAB||num. of fibres||8|
|2||IMAGE||SCI||num. of fibres||32||F|
|3||IMAGE||VAR||num. of fibres||32||F|
|4||IMAGE||DQ||num. of fibres||16||F|
The first extension is a binary FITS table with columns: ID, RA, DEC, and SKY. This table would hold information specific to individual lenslets/fibres like relative fibre positions on the sky (RA, DEC), whether the fibre is a sky or object spectrum (SKY), etc.
The three image planes are like the IRAF multispec format; each row is a separate spectrum. This is a compact and efficient way of storing the extracted spectra, avoiding having multiple extensions for each individual spectra. From IRAF, ONEDSPEC tasks like splot can be used on individual planes, and ldisplay should be able to work directly on the MEF.
A draft document has been written by the NOAO describing the new IRAF spectroscopic file format. This may, eventually, replace the GMOS working format as the final science data product for GMOS and CIRPASS observations. An example header block is shown below for the CIRPASS instrument.
The aperture identification,
APERTURE, specifies the IFU. The aperture type
APTYPE, aperture diameter
APERDIA, and aperture position angle
APERPA are the same for each spectrum. This information can be
used to construct a data cube (see Section 4.4) or spatial/dispersion displays in conjunction with the
aperture centres. The position angle is for one of the hexagonal edges and would orient the hexagons
(IFU lenslets) when reconstructing a spatial display or data cube. For a purely fibre IFU (such as
SAURON) much the same description would be used except the position angle would be
The centre of each spectrum in world co-ordinates is given by the
CRVAL keywords. In this example each spectrum is
centred at about 1.1 m
in the dispersion direction (
CRVAL1) and zero pixels in the cross-dispersion direction (
cross-dispersion co-ordinates are defined as pixels from the centre of the fibre profile, since
there is no real spatial information. The region the spectra cover in world co-ordinates are
given by the
CMAX keywords. In this example the spectra cover the range 0.9 to
1.3 m along the
dispersion and 1.5
to 1.5 pixels relative to the fibre profile centre.
CD keywords define the conversion between world co-ordinates and pixels on the detector. They
also define any possible tilt of the dispersion path relative to the detector pixels. In this example the
dispersion is 0.22 nm per pixel along the first image axis (detector rows) and there is no
CRP keywords override the
CRPIX keyword and provide the positions of the fibre spectra on the
The fibre full width at half maximum (
SPECFWHM) gives the fibre profile FWHM at the detector in the
units of the spatial WCS, in this case pixels. This is used to guide the tracing and extraction of blended
ADEC keywords give the centre positions of each lenslet element or fibre. While it is
desirable for the absolute co-ordinates to be accurate it is more important that the relative positions be
fairly precise. It is these keywords that determine the reconstructed field and gives the IFU sampling
pattern and orientation. The relative positions of the lenslets or fibres on the sky is something that
should be well-known for each IFU instrument.
The MOS-style MEF format, which is the end product of the GMOS and CIRPASS data-reduction software, is not particularly natural way of handling IFS data. Indeed, under the paradigm (used to reduce TEIFU data) these files cannot be generated. The TEIFU-style data cube format has therefore been adopted as the standard UK IFS file format and will be used in the analysis stages for both CIRPASS and UIST. This adoption allows the use of many of the generic applications within the SSC, which due to the adoption of NDF as the standard file interchange format for Starlink applications, has many tasks that can process -dimensional data.
A conversion program for GMOS and CIRPASS data to a more easily analysed data cube, which will involve re-binning the input spectra on to a rectangular array, is therefore desirable (see Section 4.4).
|1||IMAGE||SCI||32||3-D science array|
|2||IMAGE||VAR||32||3-D variance array|
|3||IMAGE||DQ||16||3-D data-quality array|
In the case of this format the IFU geometry information is no longer needed, as the input spectra have already been rebinned, but it is likely that (in the finalised file format) such information will be included as a FITS binary table.
An example of the FITS header block from a TEIFU data cube is shown below.
Here the the number and size of the cube dimensions is specified by the
and as with the IRAF spectral format the
CD keywords define the conversion between
world co-ordinates and pixels on the detector, along with the tilt of the dispersion path
relative to the detector pixels. While the
CRVAL keywords defines the central value
of each axis in world co-ordinates, e.g. in the case the spectral axis is centred on
7302 Å (
A full dictionary defining FITS header keywords which can be generated by the data-acquisition system is provided on the web by the National Optical Astronomy Observatories (NOAO) at http://iraf.noao.edu/projects/ccdmosaic/imagedef/fitsdic.html.
In the Gemini IRAF package the command gfcube converts the GMOS working format, which is currently being used as the science end product file format for the GMOS and CIRPASS instruments, to a UK standard ,, data cube in FITS format.
The Starlink CONVERT package (see SUN/55) can be used to convert to and from the Starlink NDF format. On-the-fly conversion of supported file formats (such as FITS and IRAF) can also be done by most Starlink applications if the CONVERT package has been initialised.
The CONVERT package handles the GMOS/CIRPASS MEF working format without complaint, as in the following example.
Converting the MEF to a Starlink standard NDF, the FITS binary table is converted into a normal NDF extension. An example of a resulting NDF is illustrated below. The indicate the data type of a component. Those beginning with an underscore are primitive types; others are structures.
Here the right ascension and declination position of each fibre is preserved in the
extension along with and array indicating whether the fibre is ‘on sky’.
The CONVERT package also handles the UK standard data cube format (i.e. TEIFU style data) without complaint, such as in the example below.
The FITS file is be converted it into a three-dimensional NDF which, as discussed earlier (see Section 4.3) can be read by many existing applications in the software collection.
While both GMOS (MOS style) and TEIFU (data cube) representations of IFS data are perfectly valid, there are several advantages to using the data-cube format in preference to other options. First, and perhaps most importantly, for ‘longslit’ paradigm instruments such as TEIFU (and perhaps also CIRPASS) a MOS style data reduction is not possible and therefore it is impossible to produce the first file type without major problems. Additionally a data-cube format is considered, by most people, to be intrinsically easier to visualise. Both these reasons were taken under consideration when the data-cube format was adopted, with consultation of Durham, Cambridge and the ATC, as the UK standard format for this data.
Due to the still developing nature of the IFU file formats it is possible that your data may have missing FITS header keywords, or keywords which contain incorrect information. If this is the case you may need to manually edit your FITS file headers.
A good package to use for FITS header (and data) manipulation is the FTOOLS software, which is released along with XANADU, as part of the HEASOFT package from GSFC. Further information about HEASOFT, along with detailed installation instructions, user manuals and a development guide, can be found at http://heasarc.gsfc.nasa.gov/docs/software/lheasoft/.
When a FITS file is converted to an NDF a FITS extension—sometimes called the ‘airlock’ to avoid confusion with extensions within FITS files—is created. This comprises a one-dimensional array of character strings containing the imported FITS header information. On exporting a file from NDF format back to FITS using ndf2fits the airlock contents will be propagated back to the FITS file. However, since the FITS extension is not updated when an NDF is manipulated, any information that can be derived directly from the NDF structure such as dimensionality, units and axis information will replace any equivalent information held in the FITS extension when it is exported.
The (SUN/95) package provides tools that allow you to read from, and write to, an NDF FITS extension. Example code using some of these tools is shown later in this document (see Section 6.5), and detailed documentation on these tasks is available in SUN/95.
FITS I/O with IDL can be accomplished using the IDL Astronomy Library from the GSFC. The IDL
Astronomy Library contains four different sets of procedures for reading, writing, and modifying
FITS files. The reason for having four different methods of FITS I/O with IDL is partly historical, as
different groups developed the software independently. However, each method also has its own
strengths and weakness for any particular task. For example, the procedure
read a FITS table into an IDL structure—is the easiest procedure for analyzing FITS files at the IDL
prompt level (provided that one is comfortable with IDL structures). But mapping a table
into an IDL structure includes extra overhead, so that when performing FITS I/O at the
procedure level, it may be desirable to use more efficient procedures such as
For example a data cube can be read into an IDL array using the
As can be seen, various values contained within the FITS header of the original file can be obtained
MRDFITS procedure can be used, as in this example.
In both examples the image data is read into an IDL array called
DATA, while the FITS header
information is read into another array, of TYPE
In addition FITS files can be read into IDL using the CONVERT package’s on-the-fly file conversion
ability (see SUN/55 for more details) and the
READ_NDF IDL function.
There are several methods for reading an NDF file into IDL. First the NDF can be converted to a FITS file using the ndf2fits application in the CONVERT package,
and then read into IDL using the IDL Astronomy Library (as in Section 4.8). However, there are several other approaches that can be taken.
The easiest approach is to use the
READ_NDF IDL procedure available with the CONVERT package.
When CONVERT is installed, both the IDL procedures
WRITE_NDF are placed in
$CONVERT_DIR so, to make them available to IDL, that directory must be added to the IDL search path.
This will be done if the environment variable
IDL_PATH has been set.
For example assuming we have a data cube called
file.sdf which is of type
creates an IDL floating array, data_array, with the same dimensions as the NDF and containing the
values from its
As above except that any occurrence of a bad value (
VAL__BADR as defined by the Starlink
PRIMDAT package) in the NDF will be replaced by
NaN in the IDL array.
creates an IDL byte array from the VARIANCE component of the same NDF. Output of an IDL array is
achieved using the corresponding
WRITE_NDF procedure, for example assuming
data_array is an IDL
floating array then,
creates the NDF
file.sdf with the same dimensions as the IDL array
data_array, and writes the
array to its DATA component (of type
_REAL). No checks on bad values are made by default, such
checks can be carried out, e.g.
Here any occurrence of the value
NaN in the array will be replaced by the
VAL__BADR value as defined
by the Starlink PRIMDAT package. While
writes the IDL array
var_array to the
VARIANCE component of the NDF created above. A check is
made that the size of the array corresponds with the size of the NDF.
There is yet another approach to read NDF data into IDL. Again we make use of the CONVERT package,
this time we use the ndf2ascii application to convert the NDF to a ASCII text file so that
we may use the IDL
read_ascii procedure, after generating an associated data template
ascii_template GUI. For instance reading the file
file.dat in the subdirectory
we create an associated data template using
ascii_template GUI and read the data into the IDL
We can similarly use the ndf2unf application to create a sequential unformatted binary file
and use the
read_binary and assocaited
binary_template GUI to read the data into IDL,
will return an IDL structure variable
The alternative more-low-level approach to reading either the ASCII or binary unformated files can be taken, allowing you to bypass the template GUIs. More details can be found in CONVERT documentation (see SUN/55).