Google Earth Interface to Madrigal Database

This project's goal is to allow a user searching for Madrigal data to use a Google Earth interface to find it.

Here's an example of viewing atmospheric data with Google Earth (you'll need the Google Earth plugin) : Millstone ISR data

The project has two parts:

Write a python script that queries Madrigal, and produces a text file that describes where and when there is data.
Write a python script that reads that text file, and creates a Google Earth kml file.

Task list

Learn python
Learn Eclipse/Subversion
Learn Google Earth and kml
Learn Madrigal

The url to cvs using eclipse is:

extssh:<username>@jupiter.haystack.mit.edu:/opt/local/MillstoneSoftwareArchive/MillstoneCVS

Learning about Google Earth

One part of this project is to take a text file that describes radar data, and create a kml file that can be opened by Google Earth to show where the data is. Here are the steps to learn how to do that:

Read one of the Xml tutorials (for example: http://www.freewebmasterhelp.com/tutorials/xml/)
Read the Google Earth Kml tutorial
Dowload the Madrigal kml source code to your home directory.
Log on to jupiter ssh -X jupiter and run tar -xf madrigalKml.tar
Run the python code /opt/solaris/madrigal/bin/python madrigalKml.py from your home directory.
The code should produce some *.kml files in your home directory. Open up Google Earth, choose Open, and then one the kml files. Look at the result.

Google Earth Exercises

Exercise 1

In this exercise you will add a new class to madrigalKml.py. Call the class RadarWedge. The init argument will take 8 arguments 1) radar latitude, 2) radar longitude, 3) radar altitude, 4) starting elevation, 5) ending elevation, 6) starting azimuth, 7) ending azimuth, and 8) maximum range. Add a str method to the class that describes the inputs. Then write a separate test script that creates a RadarWedge object, and then prints it.

Exercise 2

In this exercise, we will expand the RadarWedge class to add four DataTriangle objects that describe a radar wedge. See the madrigalKml.py file for what a DataTriangle object is. A DataTriangle object requires 3 points in space to form. The 3 points in space for each of the four DataTriangle objects are:

(radar location, (starting el, starting az, range), (starting el, ending az, range))
(radar location, (starting el, ending az, range), (ending el, ending az, range))
(radar location, (ending el, ending az, range), (ending el, starting az, range))
(radar location, (ending el, starting az, range), (starting el, starting az, range))

Note that I have listed some of these points in terms of elevation, azimuth, and range, but you will need to convert everything to latitude, longitude, and altitude. To do this you will need import geopy and then call the method:

altitude,latitude,longitude = geopy.radarToGeodetic(radarLat, radarLong, radarAlt, az,el,range)

So in the RadarWedge init method, you want to create one new attribute called dataTriangleList, which will contain a list of 4 DataTriangle objects in exactly the order listed above.

Exercise 3

In this exercise, we will expand the RadarWedge class to add a outputKml method. This method will take a single argument, the output kml file name. It should end with a *.kml extension. The goal of this exercise is to create an actual kml file that you can view in Google Earth.

Your outputKml method should do the following steps:

Create a madrigalKml.Kml object.
Call Kml._createKmlText with the dataTriangleList you created in exercise 2 as an input argument. This method returns a string, which is the entire kml file.
Save that string as a file, with the name of the file given by the argument to the outputKml method.

Then open that file in Google Earth to see if you can see it. Use reasonable arguments to make the wedge visible. Suggestions: radar latitude = 42.0, radar longitude = -71.0, radar altitude = 0.0, starting el = 40, ending el = 50, starting az = -100, ending az = -90, maximum range = 800 km.

Exercise 4

In this exercise, we will add another class to your module called RadarExperiment. The idea is that this class will hold a list of RadarWedge objects, depending on all the directions the radar pointed in a given experiment.

For now, your new class's init method should take two arguments, startTime and endTime. These argument should be python datetime objects. To learn about datetime ojects, google "python datetime" These two arguments should be stored as class attributes. Your init method should also create self.radarWedgeList as an empty list.

Then add a method to your new class called addRadarWedge. This method takes as an argument a RadarWedge object. The method simply appends the input RadarWedge object to self.radarWedgeList.

Finally, add a str method that prints out the startTime and endTime, and the list of RadarWedge objects.

Exercise 5

Read the section at the bottom of this page called File interface between Madrigal output and Google Earth input. Run the example code - first writeGoogleEarthIni.py and then readGoogleEarthIni.py. Be sure you understand every line in readGoogleEarthIni.py, because your code will have to also read that kind of file.

Exercise 6

Modify the RadarExperiment init method to take three arguments: startTime, endTime, and url. Store all three as attributes.

Add one final class to your module, called RadarExperimentCollection. This class is designed to hold a collection of RadarExperiment objects. The init method will take just one argument: the filename of an ini file. The init method's job is to create the list self.radarExperimentList, and then add RadarExperiment objects to it using the information in the ini file. Using the read ini code as an example, have the init method read the ini file, create RadarExperiment objects and then add them to self.radarExperimentList.

Write a test script that uses the ini file below to create a RadarExperimentCollection object.

Exercise 7

Modify the RadarExperiment init method to take seven arguments: startTime, endTime, url, radarLatitude, radarLongitude, and radarAltitude. Store all seven as attributes.

Modify RadarExperimentCollection so that it uses all seven arguments when creating each RadarExperiment

Write a test script that uses the ini file below to create a RadarExperimentCollection object. Note that the new ini file below does have the extra line 'location'.

Exercise 8

Rename the module you are working on to madrigalUIKml.py. Remove all the classes you did not write - your code is now big enough to be its own module.

From now on you will be using source control to control development. You will be checking in both your main module into CVS. When you can run the test script from exercise 7 with your new module, we will check your code into CVS. Work with Bill to set this up.

Exercise 9

In exercise 9, we are going to start producing kml output. However, to make things simpler, we are not going to produce the kml all in one method. Instead, we are going to add a method "createKmlText" to the RadarExperiment class. This method will produce only the part of the kml file related to the one experiment. Later we will write a "createKmlFile" method in the RadarExperimentCollection object that will create the whole kml file, in part by calling createKmlText for each experiment.

To do this:

Copy the entire method _getKmlRGB from the old file madrigalKml.py.
Copy the entire method _createKmlText from the old file madrigalKml.py. Rename it createKmlText without the underscore.
Remove the argument dataTriangleList, and replace it in the method with self.radarWedgeList.
Delete the documentTemplate variable, and remove all references to it in the method. The document part of the kml file will be added by createKmlFile later.

Call your new method from a test script.

Exercise 10

In exercise 9, we produce a full kml file. You will create a new method createKmlFile in the RadarExperimentCollection class. The one input argument will be the kml filename to save. This method will add the text from the documentTemplate string in the old madrigalKml.py _createKmlText method. To do this:

Replace %s %s in the documentTemplate string with a single %s.
Create the string you are going to insert in the middle of documentTemplate by creating a blank string. Then loop through each RadarExperiment in radarExperimentList, call createKmlText, and then add it to the string.
Use the % operator to insert that newly created string into documentTemplate, and write that that string to the ini file.

Test the kml file by viewing in Google Earth.

Learning about the Madrigal database of upper atmospheric data

The other part of this project is to create the text file mentioned above that describes where and when there is radar data. To do this, you will need to learn about the Madrigal database, and how to get data from it using the python API. Here are the steps to learn how to do that:

Go through the Madrigal web tutorial
Go through the Madrigal Python API tutorial

Madrigal Exercises

Exercise 1

Start with the program exampleMadrigalWebServices.py and modify it. Remove the parts that call getAllInstruments, downloadFile, simplePrint, getExperimentFileParameters, and everything after the isprint command. Change the script to only print two things: the filename you use to call isprint, and the output of isprint. For isprint, use the parameters elevation (elm), azimuth (azm) and range (range). Also, use a blank filter.

Exercise 2

Modify your code above to do exactly the same thing, except this time using a class file and a test script. Call the class file ExperimentSummary.py, and call the class in it RadarExperimentSummary. Write documentation saying that this class summarizes the spatial coverage of a radar experiment. We'll add more details later. The init method takes as arguments only the experiment id, just like madrigalWeb.MadrigalData.getExperimentFiles. It then calls getExperimentFiles and isprint just like your script above does.

In your test script, call getExperiments just like in exercise 1 to get the experiment id. Then use that experiment id to create a RadarExperimentSummary object, which should print out the same thing as exercise 1.

Exercise 3

In exercise 3, you will be creating a histogram of "radar wedges". That means you will be creating a matrix of elevation and azimuth ranges, and determining how much data there is in each range. Elevation goes from 0 to 90 degrees, and azimuth from -180 to 180. For this exercise, we will use 30 degree spacing for the grid size in both elevation and azimuth. Later on, we will replace that hard-coded number with variables for elevation spacing and azimuth spacing.

We will use numpy arrays to keep track of this histogram. Here's some example code:

>>> import numpy 
>>> histogram = numpy.zeros((90/30, 360/30), dtype=numpy.int)
>>> histogram
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
>>> histogram[2,5] += 1
>>> histogram
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]])

Note that the number of rows is calculated dynamically (that is, 90/30 instead of 3). This will make it easier to switch from 30 degrees spacing to a variable.

The goal of this exercise is to go through each data line, figure out which elevation and azimuth range it belongs in, and then add one to that histogram element. Store the histogram as a class attribute (that is, self.histogram).

Exercise 4

In exercise 4, you will repeat exercise 3, except now you will convert any azimuth outside of -180 to 180 to be within that range. The basic idea is that adding or subtracting 360 degrees to any azimuth spins you around to exactly where you started, so you can always add or subtract 360 to any azimuth without changing it. So the algorithm will be:

If you azimuth is over 180, keep subtracting 360 until its under 180.
If you azimuth is under -180, keep adding 360 until its over -180.

Exercise 5

In exercise 5, you will repeat exercise 4, except you will make the elevation and azimuth range inputs to your init method. So instead of using 30 degrees for your step, you take in elevStep and azimStep as inputs. So now your init method should be:

    def __init__(self, expID, elevStep, azimStep)

Here's some example code of how this might be done:

        elIndex = 0
        azIndex = 0
        for thisIndex, elevation in enumerate(range(0, 90, self.elevStep)):
            # test if this line's elevation is below elevation, if true:
                elIndex = thisIndex
                break
        # do the same for azimuth
        
        # now that you found elIndex and azIndex, add one to radarObj.histogram[elIndex,azIndex]

Exercise 6

Read the section at the bottom of this page called File interface between Madrigal output and Google Earth input. Run the example code - first writeGoogleEarthIni.py and then readGoogleEarthIni.py. Be sure you understand every line in writeGoogleEarthIni.py, because your code will have to also write that kind of file.

Exercise 7

In exercise 7, we will make a change to self.histogram. Now instead of storing the count of wedges in that array, we will instead be storing the range. That means we now need to make the array use floats instead of integers, so change dtype=numpy.int to dtype=numpy.float. When you want to add a data line to self.histogram, only add it if the range in the line you have is bigger than the range already there. The point of this exercise is that self.histogram will now contain all the information you need to create wedges.

Exercise 8

In this exercise, we will convert the information in the numpy array into a string in the form used for the wedges part of the ini. So you want to add a new method to your class called createWedgesString(). The wedges line list all the wedges in an experiment. Each wedge is made up of five numbers, and is separated by a comma. The file numbers are:

Starting elevation
Ending elevation
Starting azimuth
Ending azimuth
Maximum range

So the idea is you want to loop through the whole numpy array. Whenever the range is > 0.0, you will add the five numbers above to the output string, plus a comma.

Here's some suggested code to get you started:

outputStr = ''

for elIndex in range(self.histogram.shape[0]):
    for azIndex in range(self.histogram.shape[1]):
        # is range greater than zero?
            # yes - get startingEl, endingEl from elIndex and elStep
            # get startingAz, endingAz from azIndex and azStep
            # create a new string with the five numbers with spaces in between , then a comma
            # add it to outputStr

return(outputStr)

Write a test script to output the string.

Exercise 9

In this exercise, we will add four arguments to the RadarExperimentSummary init method. These are url, radarLat, radarLong, and radarAlt. Store these as class attributes.

Exercise 10

In this exercise, we will add a whole new class to your module, called RadarExperimentCollection. The init method will have three arguments: a startDatetime, and endDatetime, and an instrumentList. See the python datetime module for what a date time is. The instrumentList is just of list of integers, each of which represents a radar. The init method should also create a class attribute called self.radarExperimentList as an empty list.

Write a test script that calls your new class with the following arguments:

datetime.datetime(1998,1,19)
datetime.datetime(1998,1,24)
[30, 95]

Exercise 11

In this exercise, we will expand the RadarExperimentCollection init method so that it fills out the self.radarExperimentList by appending a bunch of RadarExperimentSummary objects. Here's how:

Create a madrigalWeb.MadrigalData as before to help get information. But this time use http://cedar.openmadrigal.org/ instead of http://madrigal.haystack.mit.edu/madrigal/.
Loop through each instrument code in self.instrumentList.
Find the lat, long, and alt of that radar.
Call getExperiments for that instrument code, and use startDatetime and endDateTime to get the time arguments.
For each experiment you get back, create a new RadarExperimentSummary object.
Append it to self.radarExperimentList

Exercise 12

In this exercise, we will add a new method to RadarExperimentCollection called createIniFile. It will take as an argument the ini file name to save. Use the example code below to figure our how to write an ini file. You will need to loop through each radar experiment in self.radarExperimentList, and for each one create a new section.

Exercise 13

In this exercise, we will start to solve the following problem: sometimes there are ranges that don't have any real data. Right now your code adds any range it finds in the file. We will change your code so that it only adds ranges with real data. Here's how:

Define a new method in the class RadarExperimentSummary called getParameters(filename). It will return a list of parameters you will check for. Filename is the file you will be analyzing. Right now, hard code this method to always return ['POPL', 'NE', 'TE', 'TI'].
Add these parameters to elm, azm, range that are already in your isprint command.
Ignore any line where none of the values of the new parameters can be converted into a float. That is, if they are all strings (such as 'missing') you should ignore that elm, azm, and range.

Exercise 14

In this exercise, we will improve the method getParameters(filename). You will examine the file, and determine which parameters will be in the list you return. Here's how:

Call madrigalWeb.MadrigalData.getExperimentFileParameters to get a list of MadrigalParameter objects.
Loop through each MadrigalParameter object. Include its mnemonic if and only if the following are true:
1. isError == 0
2. isMeasured == 1
3. category is one of the following: 'I. S. Radar Basic Parameter', 'Neutral Atmosphere Parameter', 'Harmonic Analysis Parameter', 'Unit Vector Definition', or 'Vector Quantity'
Return the list of accepted mnemonics

File interface between Madrigal output and Google Earth input

We will use and ini file as the interface between the two parts of this project. That is, the Madrigal part will create an ini file describing all the radar wedges in a series of experiments, and the Google Earth part will read that ini file and output a kml file. Sounds easy, huh? to learn more about ini files and python, read the ConfigParser python documentation.

Here's an example ini file the Madrigal code will produce:

[Experiment61234]
url = http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madExperiment.cgi?exp=experiments/2003/mlh/29oct03&displayLevel=0&expTitle=Local%20Storm
wedges = 30 35 -90 -85 600, 30 35 -85 -80 630
endtime = 2003-11-04 23:30:00
location = 42.619,-71.49,0.146
starttime = 2003-10-29 03:00:00

[Experiment61235] url = http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madExperiment.cgi?exp=experiments/2003/mlh/20nov03&displayLevel=0&expTitle=Storm wedges = 60 70 -90 -85 600, 60 70 -85 -80 630 endtime = 2003-11-21 21:30:00 location = 42.619,-71.49,0.146 starttime = 2003-11-20 05:00:00

The wedges line list all the wedges in an experiment. Each wedge is made up of five numbers, and is separated by a comma. The file numbers are:

Starting elevation
Ending elevation
Starting azimuth
Ending azimuth
Maximum range

The times are both in the format YYYY-MM-DD HH:MM:SS. The location is the radar location in latitude, longitude, and altitude in km.

Here's example code that creates the above file:

"""simple example code to write a GoggleEarth ini file

$Id: writeGoogleEarthIni.py,v 1.1 2012/07/06 20:46:21 brideout Exp $
"""
import ConfigParser

config = ConfigParser.RawConfigParser()

config.add_section('Experiment61234')
config.set('Experiment61234', 'startTime', '2003-10-29 03:00:00')
config.set('Experiment61234', 'endTime', '2003-11-04 23:30:00')
config.set('Experiment61234', 'wedges', '30 35 -90 -85 600, 30 35 -85 -80 630')
config.set('Experiment61234', 'url', 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madExperiment.cgi?exp=experiments/2003/mlh/29oct03&displayLevel=0&expTitle=Local%20Storm')
config.set('Experiment61234', 'location', '42.619,-71.49,0.146')

config.add_section('Experiment61235')
config.set('Experiment61235', 'startTime', '2003-11-20 05:00:00')
config.set('Experiment61235', 'endTime', '2003-11-21 21:30:00')
config.set('Experiment61235', 'wedges', '60 70 -90 -85 600, 60 70 -85 -80 630')
config.set('Experiment61235', 'url', 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madExperiment.cgi?exp=experiments/2003/mlh/20nov03&displayLevel=0&expTitle=Storm')
config.set('Experiment61235', 'location', '42.619,-71.49,0.146')

f = open('exGoogleEarth.ini', 'w')
config.write(f)
f.close()

Here's example code that reads and prints everything in the above ini file:

"""simple example code to read a GoggleEarth ini file

$Id: readGoogleEarthIni.py,v 1.1 2012/07/06 20:46:21 brideout Exp $
"""
import ConfigParser

config = ConfigParser.RawConfigParser()
config.read('exGoogleEarth.ini')

# get a list of all the experiment sections
experimentList = config.sections()

for experiment in experimentList:
    print(experiment)
    print(config.get(experiment, 'startTime'))
    print(config.get(experiment, 'endTime'))
    print(config.get(experiment, 'wedges'))
    print(config.get(experiment, 'url'))
    print(config.get(experiment, 'location'))
    print

Child pages

Google Earth Interface to Madrigal Database

Learning about Google Earth

Google Earth Exercises

Exercise 1

Exercise 2

Exercise 3

Exercise 4

Exercise 5

Exercise 6

Exercise 7

Exercise 8

Exercise 9

Exercise 10

Learning about the Madrigal database of upper atmospheric data

Madrigal Exercises

Exercise 1

Exercise 2

Exercise 3

Exercise 4

Exercise 5

Exercise 6

Exercise 7

Exercise 8

Exercise 9

Exercise 10

Exercise 11

Exercise 12

Exercise 13

Exercise 14

File interface between Madrigal output and Google Earth input