Firehose stores all information about which flat, arc, and telluric files are paired with which science frames in an IDL structure for portability. The next step of the reduction process is to generate this structure, and then edit it to ensure it reflects the pairings you want (e.g. eliminating saturated tellurics, etc.). This procedure is done by a largely automated, and fairly complicated procedure. It performs the following procedures without user intervention:
- Read in the headers for all objects, and record telescope RA, DEC
- Check telescope pointing against the input Magellan object catalog, and assign each science target a unique object ID and common object name
- Cross-check pointing against a list of known A0V telluric calibrators; flag standards accordingly and record B and V mags if available.
- Assign ThAr arc frames to science and telluric exposures by checking for matches in telescope pointing, and asserting that lamps are on and mirrors in.
- Match telluric frames to science targets by searching for the closest match to each science target in UT obs, airmass, and sky angle
- Match flat field frames to each object or standard
These operations are all performed when the "Generate Structure" button is selected. You may select the "verbose" or "Loud" (= very verbose) buttons to produce more output, but these are mostly only useful for debugging purposes.
Once you have created the structure, it is advisable to inspect and (if appropriate) edit it by hand to ensure there are no errors. To do this, click "Edit Structure." You will be presented with a window that looks like the following:
The first column is an index number in the FIRE structure, used for bookkeeping. The second and third columns show the fits file and object name as determined by matching with the catalog, or, failing a match, form the OBJECT field in the header. The third column shows how firehose has classified the frame. All files classified as SCIENCE frames are tagged with an object ID, and grouped with telluric and arc calibrators, shown in the rightmost fields. Actual telluric frames are paired with an arc image.
You should check to make sure that the intended tellurics and arcs are matched with the right science frames, and that classifications are correct. By highlighting a line in the list and clicking a button at the top, you can change the relevant fields for an element of the FIRE structure. This is useful for example if you wish to delete a file from the reduction list (change exptype to TRASH or object ID to -1). Or, if name matching failed you can force objects to be grouped together by matching their object IDs and names. Or, you can split a single object pointing into individual subsets by assigning extra object IDs (e.g. by rotator angle, which is not considered by the automated object matching routines).
When you are happy with the structure, click "Done." Your changes will be committed, and the entire structure will be written to file for future use. You can save or reload the structure from disk at any time using the save and load buttons.
We are working on a utility to dump the contents of the structure to a text file that may be edited and read back in as a scriptable version of the pipeline; this feature will be available soon.
10 Comments
Unknown User (dosip_1@touchstonenetwork.net)
The standard catalog allows a header which causes Generate Structure to fail
Unknown User (gblancm_1@touchstonenetwork.net)
This error is still around. I get the following error when trying to use the Magellan catalog to build the structure:
% Type conversion error: Unable to convert given STRING to Double.
% Detected at: FIRE_STR2DEG 9 /Users/gblancm/idl/FIRE/Utils/fire_str2deg
.pro
% Attempt to subscript SPLIT with <INT ( 1)> is out of range.
% Execution halted at: FIRE_STR2DEG 13
/Users/gblancm/idl/FIRE/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52
/Users/gblancm/idl/FIRE/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236
/Users/gblancm/idl/FIRE/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1115
/Users/gblancm/idl/FIRE/Utils/fire_mkstrct.pro
% FIRE_EVENT 671
/Users/gblancm/idl/FIRE/firehose.pro
% WIDGET_PROCESS_EVENTS
% $MAIN$
Unknown User (jane.rigby_1@touchstonenetwork.net)
The firestrct contains many absolute paths, which is preventing me from copying a reduction from one computer to a second and restarting (or improving on a colleague's reduction). Is there a simple way to do a global replace for these paths? I tried sed on firestrct.fits, but since it's a binary file it's twitchy about having its text edited.
Robert A Simcoe
It is fairly straightforward to edit the structure, but requires some idl fluency.
First, read it in as follows:
IDL> tmp = mrdfits("firestrct.fits",1)
This is an array of data structures, one per frame, containing useful bookkeeping info for reductions. To see the innards, type
IDL> help, tmp, /str
To access or change a variable, use structure notation. For example, to change the rawpath, type
IDL> tmp.rawpath = '/new/path/'
and then if you like, you can re-save the structure using
IDL> mwrfits, tmp, "firestrct_new.fits", /create
To swap machines on reductions, you may need to write a small .pro script to change the paths holding the calibration data as well, e.g. tmp.pixelflatfile.
Unknown User (gblancm_1@touchstonenetwork.net)
When associating frames with an object the pipeline won't consider two frames with the same object ID but different "Object" name to belong to the same object. For example in my case the "Object" field says "Target1-A" and "Target1-B" because I specified the dither in this field when I was at the telescope. Even though I set the same object ID to both frames, only frames called "Target1-A" are considered latter and frames called "Target1-B" are ignored.
The workaround is to edit the "Object" field in the structure and give the same name to all frames.
Unknown User (phil.massey_1@touchstonenetwork.net)
deleted.
Unknown User (chakrabarti_1@touchstonenetwork.net)
In the edit structure GUI, how do I delete an entry in the telluric column? i.e. I want to get rid of the association between the internal flat and a telluric. When I click "change telluric" it requires that I enter something.
Unknown User (jteske_2@touchstonenetwork.net)
I am trying to supply a non-FIREHOSE-generated pixel flat to use in the FIREHOSE extraction. I noticed that FIREHOSE rotates all of the images versus their raw form. I tried using IRAF's "rotate" on my IRAF-generated pixel flat, but I still get the following errors when I update the firestrct accordingly with the new IRAF_generated, rotated pixel flat:
MRDFITS: Binary table. 54 columns by 151 rows.
% READU: End of file encountered. Unit: 105, File: Structs/firestrct_trappist.fits
% Execution halted at: XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 786 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
% XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 756 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
% XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 756 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
% XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 786 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
% XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 756 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
% XMRDFITS 82 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/XIDL/xidl/Gene
ral/xmrdfits.pro
% FIRE_EVENT 756 /Users/jteske/Documents/Arizona/Exo_work/Observing/FireHose/FireHose_Pipel
ine/firehose.pro
% WIDGET_PROCESS_EVENTS
Has anyone else tried to use a pixel flat that was no generated by FIREHOSE in the extraction that is done by FIREHOSE? Thanks!
Unknown User (nicolas.godoy_1@touchstonenetwork.net)
I get the following error when trying to build the structure, I see in the earlier post that the same error ocurred. Can you please let me know how you solve it?
LOUD: 0
fire_mkstrct.pro: Data directory input as /home/postgrado/Escritorio/FIRE_data/datos/todos_fits/
fire_mkstrct.pro: Retrieving files in this directory...
fire_mkstrct.pro: ...329 retrieved
fire_guess_exptype() Current epoch approximately 2017.0500
fire_guess_exptype(): Reading in science catalogs...
fire_read_cat(): Reading in catalog /home/postgrado/Escritorio/FIRE_data/datos/todos_fits/manual_log.txt
% Type conversion error: Unable to convert given STRING to Double.
% Detected at: FIRE_STR2DEG 9 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% Attempt to subscript SPLIT with <INT ( 1)> is out of range.
% Execution halted at: FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_COMBINE_FLATS 25 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Flat/fire_combine_flats.pro
% FIRE_SUPERFAST_SUPERFLAT 58 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Flat/fire_superfast_superflat.pro
% FIRE_MAKEFLAT 202 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Flat/fire_makeflat.pro
% FIRE_EVENT 601 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% FIRE_STR2DEG 13 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_str2deg.pro
% FIRE_READ_CAT 52 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_read_cat.pro
% FIRE_GUESS_EXPTYPE 236 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_MKSTRCT 1155 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/Utils/fire_mkstrct.pro
% FIRE_EVENT 684 /usr/local/exelis/firehose/firehose_v2/FireHose_Pipeline/firehose.pro
% WIDGET_PROCESS_EVENTS
% $MAIN$
% Program caused arithmetic error: Floating illegal operand
Robert A Simcoe
This looks like a formatting error in the catalog file - it shouldn't be a log file, but rather should be the catalog file that you brought to the Magellan Telescopes in the format used by the Telescope Control System. If there is a question about formatting, you can post the first few lines of your catalog file and we can see why the error is occurring.
The anticipated format should look something like this:
002 Feige110 23:19:58.39 -05:09:55.8 2000.0 0.0 0.0 -0.2 HRZ 00:00:00 00:00:00 2000.0 00:00:00 00:00:00 2000.0 Std