This guide is designed to instruct SCUBA-2 users on the best ways to reduce and visualise their data using Starlink packages: Smurf, Kappa, Gaia, ORAC-DR  and Picard.
This guide covers the following topics.
Throughout this document, a percent sign (
%) is used to represent the Unix shell prompt. What follows each
will be the text that you should type to initiate the described action.
Before reducing SCUBA-2 data using the Dynamic Iterative Map-Maker you should confirm you have sufficient computing resources for your type of map.
We recommend the following:
|Large maps (pong)||96 GB|
|Small maps (daisy)||32–64 GB|
|Blank fields||32–64 GB|
For large-area maps it is important to process a full observation in a single chunk. See the text box on Chunking for an explanation of chunking. For large maps, using normal map-maker parameters a machine having 96 GB is acceptable. It is important that the memory is as fast as can be afforded, as RAM speed has a direct linear effect on processing time given that the time-series data are continually being funnelled through the CPU.
For blank-field surveys or smaller regions of the sky you can usefully run the map-maker with less memory and 32 to 64 GB is reasonable depending on the specifics of your data set. smurf is multi-threaded so multiple cores do help although above eight cores the price/performance gains tend to drop off.
If you have a very large machine (128 GB and 24 cores) you may be able to run two instances of
the map-maker in parallel without chunking, depending on the nature of the data. Use the
environment variable to restrict each map-maker to half the available cores.
This manual uses software from the Starlink packages: Smurf , Kappa , Gaia , ORAC-DR  and Picard . Starlink software must be installed on your system, and Starlink aliases and environment variables must be defined before attempting to reduce any SCUBA-2 data.
Data files for SCUBA-2 use the Starlink N-dimensional Data Format (NDF, see Jenness et al. 2014), a hierarchical format which allows additional data and metadata to be stored within a single file. Kappa contains many commands for examining and manipulating NDF structures. The introductory sections of the Kappa document (SUN/95) contain much useful information on the contents of an NDF structure and how to manipulate them.
A single NDF structure describes a single data array with associated meta-data. NDFs are usually stored within files of type “.sdf”. In most cases (but not all), a single .sdf file will contain just one top-level NDF structure, and the NDF can be referred to simply by giving the name of the file (with or without the “.sdf” prefix). In many cases, a top-level NDF containing JCMT data will contain other “extension” NDFs buried inside them at a lower level. For instance, raw files contain a number of NDF components which store observation-specific data necessary for subsequent processing. The contents of these (and other NDF) files may be listed with Hdstrace. Each file holding raw JCMT data on disk is also known as a ‘sub-scan’.
The main components of any NDF structure are:
The Convert package contains commands fits2ndf and ndf2fits that allow interchange between FITS and NDF format.
The commands and environment variables needed to start up the required Starlink packages (Smurf, Kappa, etc.) must first be defined. For C shells (csh, tcsh), do:
before using any Starlink commands. For Bourne shells (sh, bash, zsh), do:
The Sub-Millimetre User Reduction Facility, or Smurf, contains the Dynamic Iterative Map-Maker, which will process raw SCUBA-2 data into images (see SUN/258). Kappa meanwhile is an application package comprising general-purpose commands mostly for manipulating and visualising NDF data (see SUN/95). Before starting any data reduction you will want to initiate both Smurf and Kappa.
After entering the above commands, you can access the help information for either package by typing
kaphelp respectively in a terminal, or by using the showme facility to access the hypertext
documentation. See Section 1.3.7 for more information.
Image visualisation can be done with Gaia (see SUN/214). Gaia is a GUI-driven image and data-cube display and analysis tool, which incorporates facilities such as source detection, three-dimensional visualisation, photometry and the ability to query and overlay on-line or local catalogues.
Alternatively, the Kappa package includes many visualisation commands that can be run from the shell comand-line or incorporated easily into your own scripts—see Appendix “Classified KAPPA commands” in SUN/95. These tools are particularly useful for creating more complex composite plots including multiple images, line-plots, etc, such as the multi-image plots in Section 5.3.2.
The ORAC-DR Data Reduction Pipeline  (hereafter just Orac-dr) is an automated reduction pipeline. Orac-dr uses Smurf and Kappa (along with other Starlink tools) to perform an automated reduction of the raw data following pre-defined recipes to produce calibrated maps. The following commands initialise Orac-dr ready to process 850 m and 450 m data respectively.
For more information on available recipes and intructions for running the pipeline see Chapter 4.
Picard uses a pipeline system similar to ORAC-DR for post-processing and analysis of reduced data. Picard documentation can be found at Orac-dr web page, or at SUN/265. All Picard recipes follow the same structure and are run like so:
<recipe_param_file> is a text file containing the relevant recipe parameters.
RECIPE is the name of the
recipe to be run (note the caps). The list of files to be processed is given by
<input_files>. These must be in the
current directory or a directory defined by the environment variable
ORAC_DATA_IN. A number of Picard recipes
will be demonstrated in Chapter 8.
Other command-line options include
-log xsf where the log file is written to any combination of the screen [
a file [
f] or an X-window [
sf is recommended as the recipes are short and the X-window automatically
closes upon completion.
You do not specify an output filename for Picard, instead the output is generated by adding a recipe depending suffix to the input filename. If there is more than one input file then the name of the last file is used.
You can create a file which lists the input files to be passed to Picard for processing. this file is read by Picard via the Linux/Unix cat command. For example:
To execute the cat command you must enclose it in back quotes. You must also include the
.sdf extension on
any files passed to Picard.
ORAC_DATA_OUTis defined, any files created by Picard will be written in that location. Check there if new files are expected but do not appear in your working directory.
If you know the name of the Starlink document you want to view, use showme. When run, it launches a new webpage or tab displaying the hypertext version of the document.
findme searches Starlink documents for a keyword. When run, it launches a new webpage or tab listing the results.
docfind searches the internal list files for keywords. It then searches the document titles. The result is displayed using the Unix more command.
Run routines with prompts
You can run any routine with the option
| || |
A simple Google search such as “
You have two options for processing your data:
The pipeline approach is simpler and works well if you have a lot of data to process. Performing each step by hand allows more fine-tuned control of certain processing and analysis steps, and is especially useful for refining the parameters used by the map-maker. However, once the optimal parameters have been determined, it is possible to pass them to the pipeline to process other observations using the same configuration. Chapter 3 and Chapter 5 discuss the manual approach; to use the science pipeline, skip straight to Section 4.
The JCMT will produce pipeline-reduced files for each observation and group of repeat observations for each night. These are reduced using the ORAC-DR pipeline with the recipe specified in the MSB. Chapter 4 gives instruction on retrieving reduced data from the JCMT Science Archive at CADC.
SMURF_THREADS should be set to an integer value indicating the number of threads to be used by each process.