Page History

Exercise 2: Scripting interface

This exercise has three parts, and an optional addition for those familiar with python jupyter-notebooks:

1. Installing the Madrigal API of your choice (python, Matlab, or IDL)

2. Running the globalDownload script

3. Generate and run the globalIsprint script

4. Optional: Run python Jupyter notebook

Links and scripts to download

Madrigal

...

site

• CEDAR Madrigal archive site

  Millstone Madrigal

• OpenMadrigal web site

  OpenMadrigal web site

Step 1: Download Python, Matlab, and/or IDL Remote APIs

Go to the OpenMadrigal download page, and download one or more version. Available as *.tar.gz or *zip files. The Matlab and IDL installations assumes you have these commercial products already installed; the python version is open-source and can be installed on any computer.

Matlab installation: The Matlab version is simply *.m files, so they need to be put in a standard Matlab directory.

IDL installation: The IDL version is simply *.pro files, so they need to to put in a standard IDL directory.

To install the python version:

1. Make sure you have python installed. If not, install it from python.org.

2. The easiest way to install is: pip install madrigalWeb (you may need to be sudo on unix)

3. If that doesn't work, To install the remote python madrigal api from from www.openmadrigal.org, and you will need to uncompress the file. Then you will need to run run python setupMadrigalWeb.py install. If you are installing on Windows, you will need to open a command prompt, and then change directory to where you uncompressed the file. From windows the python command may be something like like c:\python25\python.exe setupMadrigalWeb.py install. All the command line scripts (globalIsprint.py, madrigalPColor.py, and madrigalScatter.py) will be put into your PATH and so can be called directly from the command line.

For Matlab or IDL, go to the OpenMadrigal download page, and download one or more version. Available as *.tar.gz or *zip files. The Matlab and IDL installations assumes you have these commercial products already installed.

Matlab installation: The Matlab version is simply *.m files, so they need to be put in a standard Matlab directory.

IDL installation: The IDL version is simply *.pro files, so they need to to put in a standard IDL directory.

Step 2: Create and run a globalDownload script

globalDownload is a simple script that will allow you to download a whole series of files, in any format - ascii, Hdf5, or the old Cedar format. Unlike globalIsprint, there is not a web interface to generate these commands, but in general the commands are simplernetCDF4. Generate the script using the CEDAR Madrigal archive web site at http://cedar.openmadrigal.org and choose Access Data, then Create a command to download multiple exps, then Download a group of files as is. You specify the time range, the instruments, the format, the programming language, and, optionally, the kinds of data .

• Documention for the python version

• Documentation for the Matlab version

• Documentation for the IDL version

and other filters.

Create a script to download all Millstone Hill ISR data (kinst=30) of data type "Gridded data fitted to a uniform spatial and temporal grid - useful for modellers" (kindat=13300) for the entire year 2010 in Hdf5 format.  Note that the script by default downloads files to the /tmp directory - change this to something else if you wishUse one the three scripts above to download all Arecibo linefeed (kinst=20) with MRACF (kindat=2010) for 2010 in Hdf5 format. If you run into problems, see the answers at the bottom of the page for all three languages.

Step 3: Create and run a globalIsprint script

• Go to the CEDAR Madrigal archive site, and choose Access Data, then Create a command to download multiple exps, then Download data with selected parms/filters.

• Choose Instrument type Distributed Ground Based Satellite Receivers, then instrument World-wide GPS Receiver Network.

• Choose any date range that covers two days between 2001 and three days in the past.
• Choose and file format you wish (Hdf5, ascii, or netCDF4), along with programming language.
• If you are saving the output as ascii, you can choose whether to save all data to one file, or to a directory with one file for each file analyzed.

• Expand the Select parameters section.

• Choose the parameters year,month,day,hour,min,sec,gdlat,glon,tec,dtec.

• Expand the Optional filters section.

• Choose Minimum Scallop TEC Processing as the data type.  The line of sight data is too large to analyze dynamically

• Download either the python, Matlab, or IDL version of the script to run the globalIsprint example (see top of page).

• Go to the CEDAR Madrigal or Arecibo site, and choose Full data access then Global Madrigal search.

• Choose the Arecibo Gregorian radar and the date range of all of 2012.

• Select Show advanced filters.

Choose MRACF for kind of data

• .

• Add three filters:

• Hit the the Select parameters button button.

• Choose the parameters year,month,day,hour,min,sec,gdalt,gdlat,glon,netec,dne dtec and hit hit Done.

• Choose Choose Continue at  at the bottom.

• Select the language you are using, and hit hit OK.

• Verify the script runs successfully as is. If there is a problem, compare to the scripts at the top.

globalDownload answers

Python command

globalDownload.py --url=http://madrigal.naic.edu/madrigal --outputDir=/tmp --user_fullname="your name" --user_email=your@email --user_affiliation=your_affiliation --format=hdf5 --inst=20 --kindat=2010 --startDate=1/1/2012 --endDate=12/31/2012 --verbose

Matlab command

url = 'http://madrigal.naic.edu/madrigal'; 
outputDir = '/tmp';
user_fullname = 'your name';
user_email = 'your email';
user_affiliation = 'your affiliation';
format = 'hdf5';
startTime = datenum('1-Jan-2012 00:00:00');
endTime = datenum('31-Dec-2012 23:59:59');
inst = 20;
kindats = [2010];


globalDownload(url, ...
    outputDir, ...
    user_fullname, ...
    user_email, ...
    user_affiliation, ...
    format, ...
    startTime, ...
    endTime, ...
    inst, ...
    kindats);

IDL command

...

Optional: Run python Jupyter notebook

For those familiar with python Jupyter notebooks, download the following notebook:

Madrigal_python_demo.ipynb

This notebook will walk you though all the calls available as part of the Madrigal API.

...