Madrigal database exercises - overview

Each group will do the following exercises:

  1. Go through the web interface exercises below

  2. Install one (or more) remote APIs (python, Matlab, or IDL)

  3. Run and then modify the example script in the language of choice that gets the latest Poker Flat data on Madrigal

  4. Run and then modify the example script in the language of choice that runs the globalIsprint example

Links and scripts to download (will be used below)

 

Python script to plot the latest Poker Flat data

Matlab script to plot the latest Poker Flat data

IDL script to plot the latest Poker Flat data

Python script to run globalIsprint example

Matlab script to run globalIsprint example

IDL script to run globalIsprint example

 

Step 1: Web exercises

Simple Local Data Access

  1. Start at the SRI or the CEDAR Madrigal site, and choose Simple Local Data Access. We will be looking at Sondrestrom data, and those are the only two sites that have local copies. As the name suggests, Simple Local Data Access is only for local data.

    1. Choose the Sondrestrom radar. If you are using the CEDAR site, it will be easier if you choose "Incoherent Scatter Radars" as the instrument type. This option does not appear on the SRI site because it only has data from a few instruments.

    2. Choose 2012, then January, and then the 12th.

    3. Sondrestrom ran two experiments on this day. Choose the StratWarm (for Stratospheric warming) experiment.

    4. Sondrestrom creates a number of different types of files. Choose ACPORT - FITTED GATE DATA FROM DWELLS.

    5. Use the Download data to download the file in both ascii and Hdf5 format.

    6. Use the Print data to display the file in ascii format in the browser.

    7. Use the View info, Show plots, and More parameters buttons. The More parameters button is a link to the full UI that allows you to choose parameters and set filters.

Browse for Individual Madrigal Experiments

  1. Start at any Madrigal site above (SRI, CEDAR, Millstone, or EISCAT).

  2. Use Full Access Data and then Browse for Individual Madrigal Experiments. Search for all instruments that were running on 2007-03-27. Use the default All Madrigal Sites option. Go back and repeat the search so only Incoherent Scatter Radars are shown.

  3. Select the Poker Flat World Day experiment that was running on 2007-03-27, and do the following with that experiment:

    1. Determine how many data files there are for that experiment, and how they differ.

    2. For one of the files, choose "View description from the catalog and/or header records", and read the summary information stored there. Ask questions if anything is unclear.

    3. Download one of the files is ascii format by using the Download file link, and sticking with the default Simple column-formated ascii option. Remember that this option does not filter the data, and no derived parameters will be included. Open the downloaded file with a text editor to make sure its easy to understand and parse.

    4. Madrigal administrators can add plots and links to each Madrigal experiment. Click on the Geometry Plot link to see the beams that were used. Examine other plots from that experiment.

    5. For the default file for Alternating Code (AC16-30), choose "Print file as ascii (isprint)". This link allows you print both measured and derived parameters. For this file, choose time parameters (year, month, day, hour, min, sec), geographic parameters (elm, azm, gdlat, glon, gdalt), geophysical parameter (kp), and I. S. Radar Basic Parameters (ne, dne, ti, dti, te, dte). Which of these parameters are in the file, and which are derived?

    6. Repeat the above with headers off and missing data replaced with the string NaN.

    7. Save the result in a file using the Save text to file button.

    8. Determine what the Autocorrelation parameter NUMTXAEU means.

    9. Next, we'll try to filter the data. There are some standards filter at the top of the web page, such as elevation or altitude. Just under them are free-form filters that allow filtering using any parameter. First, apply a filter to eliminate elevations under 75 degrees. Look at the resulting data to be sure all data with elevation less than 75 degrees is eliminated.

    10. Add a filter so that only data where te/dte is greater than 10.0 is printed (that is, the error in te is less than one-tenth the measured value).

Run Models

  1. Use the Madrigal page Run Models->Calculate any Madrigal parameter for a given time and range of lat, lon, and alt to calculate the shadow height (SDWHT) and magnetic field vector (BN,BE,BD) 1000 km directly above PFISR (lat 65.130, lon -147.471) at 2007-03-27 12:00:00 UT.

Step 2: Download Python, Matlab, and/or IDL Remote APIs

Go to the OpenMadrigal download page, and download one or more version. Available as *.tar.Z or *zip files. The Matlab and IDL installations assumes you have these commercial products already installed; the python version is open-source and can be installed on any computer.

Matlab installation: The Matlab version is simply *.m files, so they need to be put in a standard Matlab directory.

IDL installation: The IDL version is simply *.pro files, so they need to to put in a standard IDL directory.

To install the python version:

  1. Make sure you have python installed. If not, install it from python.org.

  2. To run the plotting test script in the next exercise, run the next two steps. The non-plotting parts of the Madrigal python API will run without those third-party modules.

  3. Install the package numpy from numpy.scipy.org if you don't already have it. This package does numerical arrays and linear algebra.

  4. Install the ploting package matplotlib from matplotlib.sourceforge.net if you don't already have it.

  5. To install the remote python madrigal api from www.openmadrigal.org, you will need to uncompress the file. Then you will need to run python setupMadrigalWeb.py install. If you are installing on Windows, you will need to open a command prompt, and then change directory to where you uncompressed the file. From windows the python command may be something like c:\python25\python.exe setupMadrigalWeb.py install. All the command line scripts (globalIsprint.py, madrigalPColor.py, and madrigalScatter.py) will be put into your PATH and so can be called directly from the command line.

Step 3: Use and modify the script to plot latest Poker Flat data (plotLatestPoker.*)

  1. Download either the python, Matlab, or IDL version of the script to plot the last hour of Poker Flat data (see top of page).

  2. Verify the script runs successfully as is.

  3. This version just plots Nel (log of electron density) for the last hour of Poker Flat data. Modify the script to plot Ti (ion temperature) or Te (electron temperature). You will need to change the requested parameters, the Y label string, and for the Matlab and IDL versions, the minimum and maximum parameters values.

  4. Modify the filter string to append " filter=elm,80,90 " to only include high elevation data.

Step 4: Create and run a globalIsprint script

  • Download either the python, Matlab, or IDL version of the script to run the globalIsprint example (see top of page).

  • Go to the SRI or the CEDAR Madrigal site, and choose Full data access then Global Madrigal search.

  • Choose the Poker Flat radar and the date range of March 10-20, 2007.

  • Select Show advanced filters.

  • Choose Alternating code for kind of data.

  • Add three filters:

 

Mnemonic

Lower limit

Upper limit

Kp

4.0

leave blank

gdalt

240

260

ne

2e11

leave blank

 

  • Hit the Select parameters button.

  • Choose the parameters year,month,day,hour,min,sec,elm,azm,gdalt,gdlat,glon,kp,ne,dne and hit Done.

  • Choose Continue at the bottom.

  • Select the language you are using, and hit OK.

  • Verify the script runs successfully as is. If there is a problem, compare to the scripts at the top.

  • Modify the script as suggested below. You can either directly edit your script, or regenerate it using the web interface.

  • Instead of using Kp as a filter, filter on Ap3 > 15.

  • Add Te and Ti as parameters

  • Filter also on Te > 1000

  • No labels