KNOWN PROBLEMS WITH INSTALLING XIDL AND/OR FIREHOSE

We are aware that some users have encountered difficulties installing xidl, but we have not been able to verify this on every possible platform.  If you have encountered a problem or found a solution for a particular configuration, please add a comment to this wiki so that we can pass along your information to future users.

  • Problem: On some architectures IDL reports problems loading shared execuable libraries using call_external, possibly from 32 versus 64 bit conflicts.
    • Solution: The SDSS idlutils package only runs in 32 bit mode, so you need to start idl using the "idl -32" command.  Also, xidl will compile by default into 64 bit mode unless you tell it otherwise.  To force xidl to compile in 32 bit mode, go into $XIDL_DIR/src/Makefile.  Add option "-m32" to both the CFLAGS line and also right after the $(X_LD_FLAGS) field in the line starting with $(LD).
KNOWN BUGS OR DEFICIENCIES IN THE FIREHOSE SUITE

1. The wavelength calibration and sky subtraction are not yet perfected, resulting in some residuals in sky subtracted images and mismatch of sky lines in wavelength space for adjacent orders.  It is *pretty* good, but we'd like residuals at the 0.1 pixel level and are continuing to work on this problem.  Current RMS in the wavelength fit is ~0.3-0.5 pixels.

2. The code currently crashes when attempting to combine the reddest order into the final 1d spectrum.  For this reason we have restricted the combine operation to one order below this.  It appears to be a bookkeeping error and will be fixed (we hope) in the next release.

  • No labels

24 Comments

  1. Adam Burgasser

    OK, I had a problem right off the bat installing IDLUTILS

    I'm running 64-bit OS 10.5 with IDL version 7.1.1 (darwin x86_64 m64).

    Following the instructions on the installation site, I got the latest version of idlutils with svn.  Then, when I run bin/evilmake clean:

    bash-3.2$ bin/evilmake
    OS type detected: Darwin
    Darwin/i386; requires a native (x86) IDL and gfortran
    bin/evilmake: line 71: idl: command not found
    bin/evilmake: line 82: [: =: unary operator expected
    bin/evilmake: line 84: [: =: unary operator expected
    unsupported architecture for Darwin/Intel:

    TO FIX:

    I commented out the following lines from the idlutils/bin/evilmake file:

    LINE 77: export MACOSX_DEPLOYMENT_TARGET=10.4
    LINE 82: if [ $idlarch = "x86_64" ]; then
    LINE 84: elif [ $idlarch = "i386" ]; then
    LINE 85: BITS_FLAGS=-m32
    LINE 86: else
    LINE 87: echo "unsupported architecture for Darwin/Intel: $idlarch" >&2
    LINE 88: exit 1
    LINE 89: fi

    This seemed to do the trick.

    1. Unknown User (kendrew_1@touchstonenetwork.net)

      I also had problems with IDLUTILS. Running 64-bit OS 10.8.3 with IDL version 8.0 (darwin x86_64 m64)

      On running bin/evilmake, I kept getting the error that the export.h file couldn't be located, even though the $IDL_DIR was pointing to the right directory and the file is there. 

      Following a lead from the IDLUTILS page on sdss3.org I created a new directory called ~/foo, and placed a copy of external/export.h in there. I modified the $IDL_DIR environment variable in .bash_profile to point to ~/foo instead of the actual IDL directory. Works fine now.

      No idea what caused the initial problems, but if anyone has a similar issue, give it a try.... Just be careful of any other environment variables that use $IDL_DIR as they will now be pointing to the wrong place.

      OK this fix is a really bad idea, as idl tries to locate its binary in the $IDL_DIR directory. So: does anyone know why my export.h file can't be detected in the $IDL_DIR/external directory?

      1. Unknown User (kendrew_1@touchstonenetwork.net)

        For info: All my installation problems were fixed with using absolute paths in .bash_profile. Don't use ~/, write out paths in full every time.

  2. Unknown User (fbauer_1@touchstonenetwork.net)

    Finally got a chance to test out the FIRE software some this week. Not sure this thread is still being monitored, but I will try here first. I am encountering a problem or two:

    When I trace a deep image (or an internal ARC) for sky lines, the trace seems to skew on some orders (bad7 is particularly odd):

    http://www.astro.puc.cl/~fbauer/bad1.tiff

    http://www.astro.puc.cl/~fbauer/bad6.tiff

    http://www.astro.puc.cl/~fbauer/bad7.tiff

    http://www.astro.puc.cl/~fbauer/bad14.tiff

    Also, when I arrive at my final flat field, I seem to be getting some odd linear structures. Their location is kind of random depending on the files used and on a few occasions have even disappeared when I switched on the interactive option (even though this has no interactivity!). Very strange. And some orders seem to have significant portions missing?

    http://www.astro.puc.cl/~fbauer/finalflat.tiff

  3. Unknown User (jane.rigby_1@touchstonenetwork.net)

    Does anyone have a simple tabulation (or image) that shows the approx wavelength range of each order?

  4. danmasters1_1@touchstonenetwork.net

    I had problems with installing idlutils and xidl on an OS X Lion machine. The evilmake files that are used to compile the code can be replaced by evil_64, which solved the problem (thanks to Jason Prochaska for providing this). I'm having a problem attaching the file so email me if you want it, dmasters@obs.carnegiescience.edu.

  5. Unknown User (burninghamster_1@touchstonenetwork.net)

    Hi,

    I'm having trouble with firehose_ld making the flat. It falls over with the following error. Any seen this before? suggestions?  (running idl 7.1.1, on snowleopard 10.6.8)

      FIRE_SUPERFLAT_WORK: Storing pixel dependence in temppixel-flat_composite.fits
    % FXADDPAR: Required NAXIS keyword not found
    % Error occurred at: FXADDPAR          414
       /Applications/itt/idl/lib/fxaddpar.pro
    %                    CHK_AND_UPD       282
       /Applications/itt/idl/lib/mwrfits.pro
    %                    MWR_IMAGE        1452
       /Applications/itt/idl/lib/mwrfits.pro
    %                    MWRFITS          1636
       /Applications/itt/idl/lib/mwrfits.pro
    %                    FIRE_SUPERFLAT_WORK  286
       /Applications/itt/idl/lib/FIREHOSE/LowDispersion/fire_superflat_work.pro
    %                    FIRE_SUPERFLAT_LD   72
       /Applications/itt/idl/lib/FIREHOSE/LowDispersion/fire_superflat_ld.pro
    %                    FIRE_EVENT        210
       /Applications/itt/idl/lib/FIREHOSE/firehose_ld.pro
    %                    WIDGET_PROCESS_EVENTS
    %                    $MAIN$          
    % Execution halted at: CHK_AND_UPD       282

    1. Unknown User (burninghamster_1@touchstonenetwork.net)

      OK found it.. it was old versions of mwrfits and fxaddpar lurking in my idl path.. Lesson: ensure the goddard/pro folder in idlutils is preferred in idl path..

  6. Unknown User (turner_1@touchstonenetwork.net)

    Hello, 

    I'm running Mac OS X 10.7.4 with IDL80. These are my specific path to the folders:

    setenv, 'FIRE_DIR=/Applications/itt/idl/idl80/FIRE'

    setenv, 'IDLSPEC2D_DIR=/Applications/itt/idl/idl80/idlspec2d'

    setenv, 'IDLUTILS_DIR=/Applications/itt/idl/idl80/idlutils'

    And here is my path in idl: 

    !PATH = Expand_Path('/Applications/itt/idl:/Applications/itt/idl/idl80/xidl/Spec/General/x_specplot.pro:/Applications/itt/library/coyoteprograms:/Applications/itt/library/catalyst:/Applications/itt/idl/idl80/idlutils/src/math:/Applications/itt/idl/idl80/lib:/Applications/itt/idl/idl80/idlutils/bin:/Applications/itt/idl/idl80/idlutils/goddard/pro:+/Applications/itt/idl/idl80/idlutils/pro:+/Applications/itt/idl/idl80/idlspec2d/bin') + ':' + !PATH

    I'm testing Firehose on the test data download from Fire Date Reduction. I run IDL in 32 bit mode. I'm able to get through the Trace, Flats, and Structure steps with out a problem. When doing the Extract step after several minutes I run into the following problem:
    "
    ------------------REDUCING-------------

       FIRE_LOCALSKYSUB: 

    Finding profile for obj # 1 of  1

       FIRE_LOCALSKYSUB: 

    At  x= 1707.49 on slit # 11

       FIRE_LOCALSKYSUB: 

    This is Object# 21 of total  21

       FIRE_LOCALSKYSUB: 

    -----------------------------------------

    % Compiled module: EXTRACT_BOXCAR.

    % CALL_EXTERNAL: Error loading sharable executable.

                     Symbol: extract_boxcar, File = /Applications/itt/idl/idl80/idlspec2d/lib/libspec2d.dylib

                     dlopen(/Applications/itt/idl/idl80/idlspec2d/lib/libspec2d.dylib, 1): image not found

    % Execution halted at: EXTRACT_BOXCAR     69 /Applications/itt/idl/idl/idlspec2d/pro/spec2d/extract_boxcar.pro

    %                      FIRE_LOCALSKYSUB  141 /Applications/itt/idl/idl/FIRE/Extract/fire_localskysub.pro

    %                      FIRE_ECHEXTOBJ    553 /Applications/itt/idl/idl/FIRE/Extract/fire_echextobj.pro

    %                      FIRE_PIPE         488 /Applications/itt/idl/idl/FIRE/Utils/fire_pipe.pro

    %                      FIRE_EVENT       1063 /Applications/itt/idl/idl80/FIRE/firehose.pro

    %                      WIDGET_PROCESS_EVENTS

    %                      $MAIN$          
     "
    Any help would be appreciated! Thank you for your time in advance.

  7. Unknown User (jane.rigby_1@touchstonenetwork.net)

    The installation instructions, and some of these comments, are obsolete, in that they advise that idlutils only works in 32-bit.  Happily, IDLUTILS and XIDL each have lines in their Makefiles to compile in 64-bit.  So you can do everything in 64 bit.  Your 64-bit architecture should be automatically recognized by bin/evilmake.  If not, there's bin/evil_64.

  8. Unknown User (jane.rigby_1@touchstonenetwork.net)

    Deleted.

     

    1. Robert A Simcoe

      Hi,

      Thanks for pointing this out.  It stems from a change in the way object structures are now allocated in xidl, which was causing conflicts with the FIREHOSE definition.  I've checked in the fix for this, and a few other things via svn.  The particular fix for this is in the file fire_findobj.pro.

      -Rob

  9. Unknown User (jane.rigby_1@touchstonenetwork.net)

    I added a bug report a few weeks ago on the Object Extraction page, reporting a crash when attempting to use a separate sky image.  Anybody got any ideas?

    1. Unknown User (jane.rigby_1@touchstonenetwork.net)

      Deleted.

    2. Unknown User (kendrew_1@touchstonenetwork.net)

      I keep encountering the same bug reported by Jane when trying to do off source sky subtraction (with separate file) - would be very grateful for ideas.

      1. Unknown User (kendrew_1@touchstonenetwork.net)

        Update: I got the off-source sky subtraction to work. I've added a note to the Source extraction page with info on how I set it up.

        1. Robert A Simcoe

          This bug has been fixed and off-source sky frame capability is now much more integrated into the pipeline (although the user interface for it is still a bit awkward).  See this page for instructions on how to use this feature.

  10. Unknown User (jane.rigby_1@touchstonenetwork.net)

    How should I cite FIRE and FIREHOSE in publications?  I looked for the "How to cite FIRE" section of the Manual, but couldn't find it.  I believe the correct citation is 2008SPIE.7014E..27S.  As an observer, I always try to correctly cite the lovely instruments I'm using.  :)

    1. Robert A Simcoe

      Hi, this issue is a little confusing since there were old SPIE papers out there that were cited in early work.  There is now a paper in press 2013PASP..125..270S that we encourage people to cite because it is a refereed publication describing design through commissioning.  No more changes to this though, I promise.

  11. Unknown User (jane.rigby_1@touchstonenetwork.net)

    Deleted.

     

  12. Unknown User (phil.massey_1@touchstonenetwork.net)

    deleted

  13. Unknown User (jane.rigby_1@touchstonenetwork.net)

    I've found a bizaare bug.  I'm re-reducing spectra that previously firehose handled fine.  Now, for multiple objects, I encounter the same problem.  Inexplicably, for one frame, optimal extraction appears to  proceed normally, but the extracted output is bogus:  in Object/Obj_0093.fits, the counts structure (.fx) is almost all zeros.   The other 3 frames extract fine. One can see that the extraction is bad by doing this:

    tmp = mrdfits("Obj_0093.fits",1) 

    plot, tmp[10].wave, tmp[10].fx   ; plot the counts for order 10

    Screenshot at https://www.dropbox.com/s/a96qgj8hkhyxh1y/debug-screenshot.tiff?dl=0  (bad frame 93 in red, good frame 91 in white).

    I can't figure out why frame 93 fails at extract, while neighbor frames 91, 92, 94 extract fine.  

    This failure ONLY HAPPENS when the spectral extraction mode is optimal, fitting the spatial profile from an emission line.  Boxcar extraction, and regular optimal extraction, both work.

    Suggestions?  Help?  I've spent all day debugging this.  A tidy directory of the Raw & reduced data is at
    https://dl.dropboxusercontent.com/u/23479449/fire_fails2.tar.gz
    I would be so grateful if someone could take a look.

    I've reproduced this bug on two different machines. 

    Help, please?

    1. Unknown User (jane.rigby_1@touchstonenetwork.net)

      Two years later, w latest version of firehose, this is still happening.

  14. Unknown User (eg17abk_1@touchstonenetwork.net)

    Hi,

    When solving the arc image in firehose low dispersion I get the following error after I do the fit (it happens both with my data and with the test one). I'm running idl_8.4 in Scientific Linux 7.3. Thank you.

     

    % XMANAGER: Caught unexpected error from client application. Message follows...

    % Invalid pointer: <POINTER  (<PtrHeapVar620>)>.

    % Execution halted at: X_CALCFIT          67 /home/egonzalez/idl/xidl/FIT/x_calcfit.pro

    %                      LONG_WAVEQA        45 /home/egonzalez/idl/xidl/Spec/Longslit/pro/QA/long_waveqa.pro

    %                      FIRE_WAVEIMG_LD   161 /home/egonzalez/idl/FIRE/LowDispersion/fire_waveimg_ld.pro

    %                      FIRE_WAVESOLVE_LD   70 /home/egonzalez/idl/FIRE/LowDispersion/fire_wavesolve_ld.pro

    %                      FIRE_EVENT        241 /home/egonzalez/idl/FIRE/firehose_ld.pro

    %                      WIDGET_PROCESS_EVENTS

    %                      X_CALCFIT          67 /home/egonzalez/idl/xidl/FIT/x_calcfit.pro

    %                      LONG_WAVEQA        45 /home/egonzalez/idl/xidl/Spec/Longslit/pro/QA/long_waveqa.pro

    %                      FIRE_WAVEIMG_LD   161 /home/egonzalez/idl/FIRE/LowDispersion/fire_waveimg_ld.pro

    %                      FIRE_WAVESOLVE_LD   70 /home/egonzalez/idl/FIRE/LowDispersion/fire_wavesolve_ld.pro

    %                      FIRE_EVENT        241 /home/egonzalez/idl/FIRE/firehose_ld.pro

    %                      WIDGET_PROCESS_EVENTS

    %                      X_CALCFIT          67 /home/egonzalez/idl/xidl/FIT/x_calcfit.pro

    %                      LONG_WAVEQA        45 /home/egonzalez/idl/xidl/Spec/Longslit/pro/QA/long_waveqa.pro

    %                      FIRE_WAVEIMG_LD   161 /home/egonzalez/idl/FIRE/LowDispersion/fire_waveimg_ld.pro

    %                      FIRE_WAVESOLVE_LD   70 /home/egonzalez/idl/FIRE/LowDispersion/fire_wavesolve_ld.pro

    %                      FIRE_EVENT        241 /home/egonzalez/idl/FIRE/firehose_ld.pro

    %                      WIDGET_PROCESS_EVENTS

    %                      X_CALCFIT          67 /home/egonzalez/idl/xidl/FIT/x_calcfit.pro

    %                      LONG_WAVEQA        45 /home/egonzalez/idl/xidl/Spec/Longslit/pro/QA/long_waveqa.pro

    %                      FIRE_WAVEIMG_LD   161 /home/egonzalez/idl/FIRE/LowDispersion/fire_waveimg_ld.pro

    %                      FIRE_WAVESOLVE_LD   70 /home/egonzalez/idl/FIRE/LowDispersion/fire_wavesolve_ld.pro

    %                      FIRE_EVENT        241 /home/egonzalez/idl/FIRE/firehose_ld.pro

    %                      WIDGET_PROCESS_EVENTS

    %                      $MAIN$