Exercise 2: Scripting interface
This exercise has three parts:
Installing the Madrigal API of your choice (python, Matlab, or IDL)
Running the globalDownload script
Generate and run the globalIsprint script
Links and scripts to download
Madrigal sites
- CEDAR Madrigal archive site: http://cedar.openmadrigal.org/
- EISCAT Madrigal site: http://www.eiscat.se/madrigal/
- Millstone Madrigal site: http://madrigal.haystack.mit.edu/madrigal/
- Madrigal 3 beta site: http://madrigal3.haystack.mit.edu/
OpenMadrigal web site: http://www.openmadrigal.org
Step 1: Download Python, Matlab, and/or IDL Remote APIs
For Matlab or IDL, go to the OpenMadrigal download page, and download one or more version. Available as *.tar.gz or *zip files. The Matlab and IDL installations assumes you have these commercial products already installed; the python version is open-source and can be installed on any computer. The python module may also be installed via the command: pip install madrigalWeb (sudo may need to be prepended depending on python installation)
Matlab installation: The Matlab version is simply *.m files, so they need to be put in a standard Matlab directory.
IDL installation: The IDL version is simply *.pro files, so they need to to put in a standard IDL directory.
To install the python version:
Make sure you have python installed. If not, install it from python.org.
Use the command pip install madrigalWeb, or (sudo may need to be prepended depending on python installation) or to install the remote python madrigal api from www.openmadrigal.org, you will need to uncompress the file. Then you will need to run
python setupMadrigalWeb.py install. If you are installing on Windows, you will need to open a command prompt, and then change directory to where you uncompressed the file. From windows the python command may be something likec:\python25\python.exe setupMadrigalWeb.py install. All the command line scripts (globalIsprint.py, globalDownload.py) will be put into your PATH and so can be called directly from the command line.
Step 2: Create and run a globalDownload script
globalDownload is a simple script that will allow you to download a whole series of files, in any format - ascii, Hdf5, or the old Cedar format. Unlike globalIsprint, there is not a web interface in Madrigal 3 to generate these commands, but in general the commands are simpler. You specify the time range, the instruments, and, optionally, the kinds of data. There is a web interface to generate the command in Madrigal 3 (http://madrigal3.haystack.mit.edu -> Access Data -> Create a command to download multiple exps)
Documention for the python version
Documentation for the Matlab version
Documentation for the IDL version
Use one the three scripts above to download all EISCAT Tromso VHF IS radar (kinst=74) for January 2016 in Hdf5 format. If you run into problems, see the answers at the bottom of the page for all three languages. The data can be downloaded from the CEDAR Madrigal site, the EISCAT site, or the Madrigal 3 Beta site. Metadata values such as kinit id codes for each instrument are available in Madrigal 3 (http://madrigal3.haystack.mit.edu -> Access Metadata -> Instrument metadata)
Step 3: Create and run a globalIsprint script
Download either the python, Matlab, or IDL version of the script to run the globalIsprint example (see top of page).
Go to the CEDAR Madrigal or
Arecibo EISCAT site, and choose Full data access then Global Madrigal search.
Choose the Arecibo Gregorian EISCAT Svalbard radar and the date range of all of 2012Jan 20-24, 2014.
Select Show advanced filters.
Choose MRACF for GUISDAP Fitted Parameters for kind of data.
Add three two filters:
Mnemonic | Lower limit | Upper limit |
gdalt | 240200 | 260300 |
nel | 12 | leave blank |
Hit the Select parameters button.
Choose the parameters year,month,day,hour,min,sec,gdalt,gdlat,glon,ne,dne and hit Done.
Choose Continue at the bottom.
Select the language you are using, and hit OK.
Verify the script runs successfully as is. If there is a problem, compare to the scripts at the top.
globalDownload answers
Python command
globalDownload.py --url=http://www.eiscat.se/madrigal/ --outputDir=/tmp --user_fullname="your name" --user_email=your@email --user_affiliation=your_affiliation --format=hdf5 --inst=74 --startDate=1/1/2016 --endDate=01/31/2012 2016 --verbose |
Matlab command
outputDir = '/tmp';user_fullname = 'your name';user_email = 'your email';user_affiliation = 'your affiliation';format = 'hdf5';startTime = datenum('1-Jan-2016 00:00:00');endTime = datenum('31-Jan-2016 23:59:59');inst = 74;globalDownload(url, ... outputDir, ... user_fullname, ... user_email, ... user_affiliation, ... format, ... startTime, ... endTime, ... inst); |
% (url may also be http://cedar.openmadrigal.org/ or http://madrigal3.haystack.mit.edu/) |
IDL command
'/tmp', 'your name', 'your@email', 'your_affliliation', $ julday(1,1,20122016,0,0,0), julday(1,31,20122016,23,59,59), 74, $PTR_NEW() , 'hdf5' |
globalIsprint answers
Python command
globalIsprint.py --verbose --url=http://www.eiscat.se/madrigal --parms=MIN,GDALT,GDLAT,HOUR,NE,SEC,DAY,DNE,YEAR,GLON,MONTH --output=example.txt --user_fullname="your name" --user_email=your@email --user_affiliation=your_affiliation --startDate="01/20/2014" --endDate="01/24/2014" --inst=95 --filter=gdalt,200.0,300.0 --filter=nel,12.0, --kindat=6800 |
Matlab command
globalIsprint('http://www.eiscat.se/madrigal', ... % (url may also be http://cedar.openmadrigal.org/ or http://madrigal3.haystack.mit.edu/) |
IDL command
madglobalprint, 'http://www.eiscat.se/madrigal', $ |
2014,23,59,59), $ |
2010, 'hdf5'