7 Cheat sheet for checking data quality

This section is a cheat sheet for a number of simple operations that can be used to do quick checks of the data quality.

7.1 Using sc2clean to help inspect time-series

sc2clean can be used to do two basic tasks in one go: concatenate data (with or without applying a flatfield); and cleaning (fix up steps and spikes, remove the means, filter, remove common-mode etc.). It uses the same configuration files as the iterative map-maker (though ignoring the map-making specific items).

In this first basic example, we just want to clean up some data enough to be able to see whether the bolometers have been flat-fielded correctly, and more-or-less exhibit the same behaviour over time:

Here $FILES can just be a single file from a subarray, or a subset, e.g. s8a20110417_00051_0003.sdf (the first file containing science data), s8a20110417_00051_000"[1234]" (file 1 is a noise observation with shutter closed that gets ignored, file 2 is a flatfield observation that will be used to override the flatfield stored in the subsequent files 3 and 4 which are concatenated together, the .sdf is optional), s8a20110417_00051_000\? (files 1 through 9), s8a20110417_00051_\* (the whole observation). If you inspect the resulting clean.sdf in Gaia (Section 3.3) and flip through the data cube you should see all of the bolometers signals go up and down together with about the same amplitude: the hope is that for a well-behaved instrument you are mostly seeing sky noise variations that are seen with roughly the same amplitude by all bolometers. Another common feature, if the scans are particularly long and/or fast (e.g. 1 deg across), is strong periodic signals that are correlated with the scan pattern. See Section 3.2 – in particular you will want to plot az and el (the absolute azimuth and elevation), and also daz and del (the azimuth and elevation offsets from the map centre). This signal is usually azimuth-correlated due to magnetic field pickup. It only shows up in azimuth, because the instrument is on a Nasmyth platform and therefore does not move in elevation.

Part of the reason the signals look the same is because they have been flatfielded. You can turn off flatfielding using the noflat option to sc2clean, and you should then see that all of the detector amplitudes vary.

Another very useful option is to remove the common signal observed by all of the bolometers. This may be accomplished by

The residual of this signal will exhibit second-order time-varying correlated signals across the focal plane. Usually these are not very large, but in some cases some very large localized signals have been detected, particularly in the 850 $\mu$m arrays in early 2011.

Another variation on this is to accentuate the residual low-frequency noise by low-pass filtering the result. This can again be accomplished by simply adding a filter command in the config parameter, which in this case low-pass filters with a cutoff at 10 Hz:

Finally, in some cases you might just want to fit and remove polynomial baselines from the bolometers (by default only the mean is removed). This example will remove a line, but you can increase the value of order to remove higher-order polynomials

7.2 Looking at calibration data

Currently all science observations are preceded by two calibrations: a short shutter-closed integration (file 1), followed by a “sky flat” to determine the response of all the bolometers to a current ramp through their heater resistors with the shutter open (file 2). The science observations are all subsequent files, except for the last which is also a sky flat.

The shutter-closed integrations can be useful to track the array sensitivity independent of sky conditions. You can run these files (or the science data files as well) through calcnoise (the NEP map for the subarray will be stored in the .MORE.SMURF.NEP extension of the output NDF s8a_noise.sdf):

The sky flats can be used to calculate responsivity images using calcflat:

The resulting s8a_flat.sdf contains a data cube of the data used to fit the responsivity (stacking several heater ramps together – it is easy to see bizarre/non-linear detectors that will subsequently be flagged and removed), and the actual responsivity image is stored in s8a_resp.sdf.