There are two ways to clean time-series data:
= 1to your configuration (see Section 5.4.3).
The rest of this appendix gives more details on using sc2clean.
sc2clean can be used to do two basic tasks in one go: concatenate data (with or without applying a flatfield); and cleaning (fix up steps and spikes, remove the means, filter, remove common-mode etc.). It uses the same configuration files as the iterative map-maker (though ignoring the map-making specific items).
In this first basic example, we just want to clean up some data enough to see whether the bolometers have been
flat-fielded correctly, and more-or-less exhibit the same behaviour over time. The pre-processing or cleaning
steps used by default (i.e. if “
config=def” is included on the command line) are summarised in Section I.3.
Note, whilst it is not recommended to run makemap in this way (i.e. without a configuration file), it is not so
critical when running sc2clean.
files.lis can just contain a single file from a sub-array, or a subset, e.g.
(the first file containing science data),
s8a20110417_00051_000"" (File 1 is a noise observation with
shutter closed that gets ignored, File 2 is a flatfield observation that will be used to override the
flatfield stored in the subsequent Files 3 and 4 which are concatenated together, the
.sdf is optional),
s8a20110417_00051_000\? (Files 1 through 9),
s8a20110417_00051_\* (the whole observation).
If you inspect the resulting
clean.sdf in Gaia (Section 9.4) and flip through the data cube you should see all of
the bolometers signals go up and down together with about the same amplitude: the hope is that for a
well-behaved instrument you are mostly seeing sky noise variations that are seen with roughly the same
amplitude by all bolometers.
Another common feature, if the scans are particularly long and/or fast (e.g. 1 degree across), is strong periodic
signals that are correlated with the scan pattern. See Section 9.3—in particular you will want to plot
(the absolute azimuth and elevation), and also
del (the azimuth and elevation offsets from the
map centre). This signal is usually azimuth-correlated due to magnetic-field pickup. It only shows
up in azimuth, because the instrument is on a Nasmyth platform and therefore does not move in
Part of the reason the signals look the same is because they have been flatfielded. You can turn off flatfielding
noflat option to sc2clean, and you should then see that all of the detector amplitudes
Another very useful option is to remove the common signal observed by all of the bolometers. This may be accomplished by
config setting causes the default values to be used for all configuration parameters except
compreprocess, which is set to 1 (the default is 0). The residual signal left by this command will exhibit
second-order time-varying correlated signals across the focal plane. Usually these are not very
large, but in some cases some very large localized signals have been detected, particularly in the
arrays in early 2011.
Another variation on this is to accentuate the residual low-frequency noise by low-pass filtering the result. This
can again be accomplished by simply adding a filter command in the
config parameter, which in this case
low-pass filters with a cutoff at 10 Hz:
Finally, in some cases you might just want to fit and remove polynomial baselines from the bolometers (by
default only the mean is removed). This example will remove a line, but you can increase the value of
remove higher-order polynomials
Non-default values for any of the cleaning parameter can be specified like so:
Or you can create your own customised configuration file. For instance:
The more interesting pre-processing options that may be specified are listed and described in Appendix H.