Madrigal 2019 Group Exercise 1: Using the web interface

Part 1: Madrigal 3 Web interface

Selected Madrigal 3 site:

Data Access -> List Experiments
In the List Experiments, you can list experiments to choose from.  This interface allows you to look either at ALL Madrigal sites, or just the local one. 
  1. Choose Data Access -> List experiments from either of the two Madrigal sites listed above.
  2. If this is the first time you have used Madrigal, you will be redirected to a page where you set up a cookie with your name, email, and affiliation.  Madrigal does not require passwords, but your data downloading is logged based on your cookie.  When you have finished with that form, again choose Data Access -> List experiments.
  3. Leave "Use all Madrigal sites" selected so you search over all Madrigal sites.
  4. For "Instrument category" choose "Incoherent Scatter Radars".
  5. Briefly look through the list of incoherent scatter radars for which there is data on Madrigal.
  6. Select the "Millstone Hill IS Radar", and for a time period choose February 2018. Then hit "List experiments".
  7. You will notice that every time period is repeated four times.  There are copies on the same experiment both on the CEDAR site and the Millstone site.  For each experiment, there is also the standard processing, and alternate processing that you can ignore for now.
  8. Choose the standard version of the Feb 6, 2018 experiment from the CEDAR site with experiment name "Wide Coverage East Azimuth Scan and Regional Vector Experiment".  If the background blue color disappears, you have accidentally selected the Millstone Madrigal 2 site - hit the back button and try again.
  9. Look at the top line.  This lists the principle investigator of this data set.  Contact them as soon as you consider using their data in a paper!  If you are really interested in an experiment or an instrument, you can sign up to be notified when that experiment or instrument is updated.
  10.  Click on the "Select file" pull-down menu.  You will notice there are numerous different files associated with this experiment.  This is explained in detail in the Millstone Hill overview document.  Basically there is a combined file, and then smaller files with subsets of that combined files, along with derived velocity files and a gridded file where complex measurements are reduced to a standard grid for use by modelers.  For now choose the "Combined basic parameters file" listed first.
  11. Choose the "Show Plots" button.  This will list all the plots and other documentation the instrument principal investigator added to Madrigal to help users understand the data.  For this experiment, choose "Summary plots - electron density".
  12. In this experiment, the Millstone Hill was looking in a number of directions 1) zenith, 2) 45 elevation, azimuth 135, 3) 45 elevation, azimuth 45.  Finally, the radar was also scanning across a range of azimuths with a constant elevation of 6 degrees.  If you are familiar with Google Earth or install the Google Earth plug in on your browser, you can see this scan's data displayed in a Google Earth environment.
  13. Select the "View File Info" button.  This will list the descriptive text embedded in the Hdf5 or netCDF4 versions of this file.
  14. Select the "Cite this file" button.  This will create a full citation with a permanent w3id.org based url which can be used as a data citation in a publication that uses this data file.
  15. Click the "Download File" button.  You will see two options "As is" and "Select parameters/filters".  For now select "As is".  This means you are downloading the file as created the instrument principal investigator.  This offers less flexibility than "Select parameters/filters" but is often much faster.
  16. Choose any of the three formats, and download that file.

 

Data Access -> Select Single Experiment

In select single experiment, you will be guided to a single experiment to examine, rather than seeing a list of experiments.  This interface allows you to look either at ALL Madrigal sites, or just the local one. 

  1. Choose Data Access -> Select single experiment from either of the two Madrigal sites listed above.

  2. Leave "Use all Madrigal sites" selected so you search over all Madrigal sites.
  3. For "Instrument category" choose choose "Fabry-Perots".
  4. Select the "Arecibo Imaging Doppler Fabry-Perot".
  5. Select year 2018, month January, and day 15.
  6. Note that from this point on, the options are similar to the List experiment user interface.  
  7. This time we will print the file with and without filters.  Select the one file listed, then choose "Print file" and "As is".
  8. Next we will print the file, but filter by data quality.  To learn about data quality, choose "Show plots" and "Description of data quality (FPI_DATAQUAL) parameter".
  9. Next choose "Print file" and "Select parameters/filters".
  10. Open the "Select Parameters to include" section by clicking on it.
  11. Click or hover on any of the parameters to see its definition.
  12. You could select any parameters to print, including derived ones in grey.  For now select just the parameters in the file by clicking "Select all parms in original file" at the top.
  13. Close  "Select Parameters to include" section by clicking on it, and open "Select filters to use (optional)".
  14. In the bottom section select or type "FPI_DATAQUAL" as the filter parameter.  Leave the lower limit blank, and set the upper limit to 0.  This will remove all data with values above 0.
  15. Verify the resultant output only has values of 0 for FPI_DATAQUAL.
Metadata Access

All the metadata in Madrigal can be browsed in this section.  Briefly glance at each option.  The Filter String metadata is information only needed when users are writing advanced scripts to download data.

Run Models
  1. Choose Run Models->Run Madrigal derivation engine. This page allows user to directly run the Madrigal derivation engine.  Use this page to calculate the shadow height (SDWHT) in the Geographic Coordinate parameters section, and magnetic field vector (BN,BE,BD) parameters from the Magnetic Coordinate parameters section.  Look at a single point 1000 km directly above PFISR (lat 65.130, lon -147.471) at 2007-03-27 12:00:00 UT.
  2. The Run models -> looker section is optional.  This section derivers parameters that depend on a look direction, such as would be relevant to radar measurements.
  3. The Run models -> ISR empirical models section allows you to run an empirical model to predict the conditions around a number of ISR radars.  Try it by clicking Run under Millstone Hill Radar Models. Leave all defaults, and select RUN.  Select "Local time Variation" and click "generate plots".

Part 2: Using older Madrigal 2 Web Interface for EISCAT data

Simple Local Data Access

  1. Start at the Eiscat site, and choose Simple Local Data Access. We will be looking at the EISCAT Svalbard radar data, and those are the only two sites that have local copies. As the name suggests, Simple Local Data Access is only for local data.

    1. Choose Incoherent Scatter Radar as the instrument type, and then choose the EISCAT Svalbard radar. 

    2. Choose 2014, then January, and then the 22th.

    3. Two experiments show up because the experiment form 2014-01-21 ran 29 seconds into this UT day. Choose the second one that starts at 2014-01-22 00:00:29.

    4. There are two files - one for each of the two Svalbard antenna. Choose the fixed 42 meter antenna that looks up B.

    5. Use the Download data to download the file in both ascii and Hdf5 format.

    6. Use the View info, Show plots, and More parameters buttons. The More parameters button is a link to the full UI that allows you to choose parameters and set filters.

Browse for Individual Madrigal Experiments

  1. Start at the Eiscat Madrigal site.

  2. Use Full Access Data and then Browse for Individual Madrigal Experiments. Search for all instruments that were running on 2013-01-11. Use the default All Madrigal Sites option. Note that data from many different Madrigal sites appears, not just from the site you started with.

  3. Select the EISCAT Svalbard experiment that was running on  2013-01-11, and do the following with that experiment:

    1. Determine how many data files there are for that experiment, and how they differ.

    2. Download one of the files is ascii format by using the Download file link, and sticking with the default Simple column-formated ascii option. Remember that this option does not filter the data, and no derived parameters will be included. Open the downloaded file with a text editor to make sure its easy to understand and parse.

    3. Madrigal administrators can add plots and links to each Madrigal experiment. Click on the plots from that experiment.

    4. For the default file for the 42 meter fixed antenna, choose "Print file as ascii (isprint)". This link allows you print both measured and derived parameters. For this file, choose time parameters (year, month, day, hour, min, sec), geographic parameters (elm, azm, gdlat, glon, gdalt), geophysical parameter (kp), and I. S. Radar Basic Parameters (ne, dne, ti, dti, te, dte). Which of these parameters are in the file, and which are derived?

    5. Repeat the above with headers off and missing data replaced with the string NaN.

    6. Save the result in a file using the Save text to file button.

    7. Next, we'll try to filter the data. There are some standards filter at the top of the web page, such as elevation or altitude. Just under them are free-form filters that allow filtering using any parameter. Add a filter so that only data where te / dte is greater than 10.0 is printed (that is, the error in te is less than one-tenth the measured value).

 

  • No labels