Week of June 1 This week I familiarized myself with the fields of upper atmospheric studies and space weather. I learned about the mechanism of incoherent scatter radar and what physical quantities can be observed and derived using this technique. Several sources including the articles by Evans (1969), Rishbeth (1998), the book by Hargreaves (1992), and the EISCAT webpage proved especially helpful in providing this background information. The Rishbeth article also revealed the more or less current state of ionospheric studies, with an emphasis on the F2 region anomalies. Some other topics covered in my readings were ionospheric chemistry or reaction rates and the numerous observed ionosopheric variations (i.e. diurnal, seasonal, semi-annual, annual, and solar cycle) and some of their potential causes.
I also became acquainted with the tools necessary for my project. I learned the basics of writing MATLAB scripts as well as some important UNIX commands. I learned the method of accessing data from the Madrigal Database and downloaded data from the Millstone Hill Radar as well as the Poker Flat Radar in Alaska. I used this data to investigate basic properties of the ionosphere over both sites. Using MATLAB I plotted various physical quantities over the 2007-2008 observation period, including electron density, electron temperature, and ion temperature. A large spread was evident in the Poker Flat data and following Dr. Zhang's recommendations I attempted to reduce the spread to make annual trends more apparent. I tried to achieve this by looking at the electron density over time in specific ranges of azimuth, elevation, local time, and altitude. However, none of these techniques yielded much success. This is because most of the Poker Flat observations were confined to specific angles of azimuth and elevations so the spread was largely due to those conditions under which a majority of the observations were conducted. As an example, all of the Poker Flat data for azimuth angles corresponding to more southern latitudes were collected at elevations less than 80 so filtering out higher elevations had no effect.
Nevertheless, some interesting information could be gained from these preliminary plots of the Poker Flat data. Most notably, for southward observations the electron density begins to increase in the spring of 2008, when the data set ends. However, this feature is not present in the Spring 2007 data, possibly suggesting some annual asymmetry. Furthermore, the electron density to the south appears to be at a maximum around the end of spring and the minimum occurs around December/January. For northward Ne observations at Poker Flat, most occurred at an azimuth angle of 20.5 degrees. Here the minimum occurs again around the end of December or beginning of January, while the maximum appears to occur at the end of summer, around day 240.
Currently I have been working on plotting the median Ne for bin sizes of 5, 10, and 15 days over the normal data set. This makes the overall trend of the plots more legible, especially for the southward Poker Flat data where the spread is large. I will also learn to fit the data by multiple regression (using a least squares fit to solve for a number of coefficients).
Week of June 8 Much of my time this week was spent learning different ways to represent ionospheric data from the Poker Flat Incoherent Scatter Radar using MATLAB. Using the Madrigal database I downloaded the PFISR data in the altitude range of 300-325 km for all local times. For basic scatter plots of electron density over the year I wrote a script to calculate and plot the monthly median values. Included in the script and plot are error bars as well. These were calculated as the standard deviations in the daily data sets. I edited my script to include a very basic model that tried to describe the electron density as a function of the day number. There are five terms in the model, a constant and two trigonometric terms (representing Fourier components), a sine and cosine, for both 12-month and 6-month harmonics (or annual and semi-annual variations, respectively). The coefficients for these terms were calculated from the data set by performing a least squares fit through the multiple regression technique in MATLAB and the model values were then plotted over the scattered data. When comparing the line depicting the monthly median values with the model curve it was noted that the two were in relative agreement. One noted difference in the Poker Flat (northward) data from that taken at Millstone Hill is that there is a single minimum (in the winter) and a single maximum (in the summer), both of which are reasonably clear. Similar plots were created for electron temperature and ion temperature, and these graphs showed less regular fluctuations. These plots peaked in the spring and were at a minimum in the winter, at the same point where the electron density was minimized.
After completing this, I worked on creating another script that would produce a contour plot of the electron density for different hours of the day over the course of the year. The contours were created from the yearly model values described above, only a new yearly model was calculated for each hour of the day using a for loop. This plot indicated that the maximum electron density always occurs in the afternoon during the spring and summer, while it occurs closer to noon during the winter. Currently I am working on introducing the solar and magnetic activity indices, F10.7 and Ap respectively, into the model. The F10.7 and Ap indices are first normalized by subtracting the mean or median from the daily or 3-hourly values and then dividing this quantity by half of the range of the index values. These then become the terms whose coefficients are solved for in the least squares fit. Accounting for the solar and magnetic dependencies should allow the model to more accurately describe the data.
Week of June 15 I spent this past week at the 2008 CEDAR Conference in Midway, Utah. I found both the plenary sessions and the workshops to be informative and helpful. One interesting aspect of the conference for me was learning the ways in which the work of observers and modelers come together. The conference also proved to be helpful for me in describing thoroughly some of the mechanisms of the upper atmosphere. These included atmospheric gravity waves, traveling ionospheric disturbances, tidal interactions, Hall & Pedersen conductivities, and particle precipitation effects, among others. Some of these I had come across in my readings and the conference served to clarify my understanding of them, while others dealt with new subjects for me. Overall I felt the conference increased my understanding of the field. The list of workshops I attended is as follows: Space Science Instrumentation, Ground-Space Models for Studying Atmospheric Coupling, Space Weather Effects And Aeronomy Studies at the Plasmaspheric Boundary Layer, Research Opportunities at Millstone Hill Observatory, The Second AMISR Science Planning Meeting: PFISR I & II, and I-T Challenges Based on Continuous Observations through the IPY.
Week of June 22 This week I began working with the gridded Madrigal data files. The results of my project will eventually be drawn from this data. Using a Python script written by Bill Rideout (runProfileFit.py) I am downloading Madrigal data files from the Millstone Hill Radar (MHR) and the EISCAT Svalbard Radar (ESR). I will eventually download the IPY observations from the Poker Flat ISR (PFISR) as well. Bill's script downloads the data from the Madrigal database and then runs John Holt's Fortran program (profileFit.cmd) on the files. Currently I have downloaded the EISCAT data from March 2007 through December 2007, and the Millstone Hill data from March 2007 through February 2008. All of the data is separated into folders by radar and then by month. For example, the March 2007 EISCAT data can be found using the following file path: /home/jupiter/tdavid/Desktop/IPYwork/esr/mar07 Within each of these directories the data can be found in the output file titled 'bindat.out.'
After downloading the data I run John's program (profileFit_av.cmd) in the working directory. This program gives me several output files - which then allow me to use John's MATLAB codes on the data. My current issue is editing these MATLAB codes so they will perform the desired task. It is my impression that within several of the codes there are certain parameters and values that I may need to change with every data set. This matter should be cleared up when I learn the meaning of some of the variables. (For an explicit example, in the MATLAB program 'profileFit.m' I am unsure as to what the variable 'nalt' refers to and what value it should take.) Once the codes are working for the generic output files described above the project should come together smoothly.
Another issue I will eventually face is how to deal with data that runs the length of the entire IPY period. As of now I am working on collecting monthly files for each of the three radars being studied. I am unsure whether or not it is possible to string the monthly data together, or whether it is possible to append to an existing file the other months. Of course the simplest solution, though redundant, would be to download a new file using Bill's script for the entire IPY period for each radar. So the questions that remain for me are ones concerning the structure of the gridded data files and ones regarding the time periods that will be most important to study for this project.
This week I have also worked on generalizing my MATLAB codes for creating contour and scatter plots of the basic ISR parameters Ne, Te, and Ti. I hope to improve my script by allowing the user to control certain input values such as the parameter being plotted, the number of contours, axes scales, and the choice of turning on and off certain terms used in determining the fitting coefficients (i.e. F10.7, Ap, annual, semiannual, diurnal, semidiurnal, and terdiurnal variations). I've also modified Shun-Rong's scripts 'ipy4PF.m' and 'ionmodel.m' to account for diurnal, semidiurnal, and terdiurnal variations and will present the resulting plots to Shun-Rong for comparison to those from previous methods that included only annual and semiannual variations.
Week of June 29 This week I completed my MATLAB code that will generate a contour plot of basic ISR parameters (Ne, Te, Ti) with respect to day number and local time. It is simple, of course, to edit these axes so that I may plot yearly contours for altitude instead of local time. The script essentially removes the effects of magnetic and solar activity when plotting, making the contours smoother. However, the code may easily be edited to investigate annual variations for specific solar and geomagnetic conditions. Our empirical model is also more complex now that it takes into account cross terms of the basic variations (annual, semiannual, diurnal, semidiurnal). So in the updated code the least squares fitting is executed using the 13 terms listed below (or 18 including the two components to each of the fundamental temporal variations): -constant -normalized F10.7 index -normalized AP3 index And the sine and cosine components of each of the following temporal variations: -annual -semiannual -diurnal -semidiurnal -terdiurnal As well as the following cross terms: -diurnal*annual -semidiurnal*annual -diurnal*semiannual -semidiurnal*semiannual -annual+semiannual
Additionally, I was able to execute several of John's MATLAB codes on the output data from the profileFit_av.cmd. The codes I was able to execute were (plotFit2Data, plotContours, and plotDataAndFit) giving me Altitude-UT contours, Altitude-UT plots of each data point as well as the data minus the fit for each point, altitude profiles for various UTs, and UT profiles for various altitudes.
Finally, I began a script to calculate monthly averages of geophysical indices from the instruments at Millstone Hill (data downloaded from Madrigal) and then create a file of the following values for each parameter and month: [Month UT Altitude Parameter F10.7 Ap3] Thus there is a single monthly average for the daily F10.7 index and eight monthly averages for the 3-hourly Ap index.
Week of July 6
This week I worked on obtaining Madrigal data from the binary output of John's code (profileFit_mad.cmd) using the command isprint. To make these new monthly average data sets compatible with my existing contour and scatter plot codes I then reformatted the data and combined it with corresponding geophysical parameters, using my script (combine.m). This script calculates a monthly average of the F10.7 index and eight monthly averages for the 3-hourly Ap indices. I now have working MATLAB codes to produce UT-month contours and scatter plots from the monthly-average data sets. Additionally I have completed retrieving the pertinent IPY data for the following radars: Millstone Hill, PFISR, EISCAT Svalbard, and Sondrestrom Research Facility.
I am currently working on developing my scatter code for plotting each data point as well as the data minus fit value for each point. Comparing these scatter plots with my contours will help me determine which terms should be included in my least squares fitting by attempting to match the structure of the contour with that of the scatter plot. I will also check the average of the absolute value of the data-fit difference to minimize the deviance for different combinations of fitting terms. Additionally I am working on creating codes to produce month-height and month-UT contour plots by fitting yearly data repeatedly for a number of height or time bins. These plots are helpful when the relationship between a parameter and height or local time is unknown.
Week of July 13
This week I retrieved the monthly averages and Madrigal files for the remaining Sondrestrom data, as well as extended the range of my data for all radars through June 2008 wherever possible. It is noted, however, that the observing schedules for the four radars may have changed considerably after March 2008 so I will use caution when introducing these more current data into my codes. I also produced monthly averages and Madrigal files for PFISR twice, using different azimuth filters each time. For north-facing azimuth angles I looked between -30 and 30 degrees, and for south-facing azimuth angles I went from 150 to -150 degrees (measured clockwise). However, the contours I produced for each resulting data set were identical although the data sets themselves were different.
I also developed codes to produce Ne, Te, and Ti contours for the monthly averages with respect to altitude or UT. Both programs perform a multiple linear regression on the data within given UT or height bins (i.e. 24 separate year-long fits for one-hour bins of UT). It will be important to determine what bin size and technique (i.e. non-overlapping or overlapping) is best for representing the data as the contours vary greatly when changing these parameters. The selection of bins is more difficult for the height contours since the altitude nodes, while occurring at intervals that are always a factor of 4 km, have some gaps. The altitude nodes (for ESR) are as follows: 4 km intervals from 100-116 km, 8 km intervals from 116-204 km, 16 km intervals from 204-444 km, 32 km intervals from 444-508 km, and a 40 km interval from 508-548 km. The UT nodes occur from 0 to 23.5 hr at regular 0.25 hour intervals.
Additionally, I further developed my main contour plot code which now produces two scatter plots: one representing each data point and one representing the data - fit value for each point. The code also produces the median value of the absolute value of the (data - fit) values, giving another measure of the accuracy of the fit. Using this code I examined the accuracy of the fit for different conditions (i.e. the inclusion of different combinations of cross-terms while always keeping the constant term, magnetic and solar activity terms, and annual/semiannual and local time variation terms part of the regression). Simply looking at the median deviation described above I determined that for the EISCAT, Svalbard radar the optimal fit for electron density data is achieved when including only the cross-term produced by multiplying the semidiurnal and semiannual components. I also found that the cross-term produced by the sum of the annual and semiannual components affects the fit the least. After running John's MATLAB codes on each monthly average data set for ESR I noticed that November 2007 looked irregular. Looking at the MADRIGAL website it is clear that the radar was inoperative from October 25, 2007 through November 24, 2007 and only seven data sets from the end of November exist. I removed November, 2007 data in my code and found the electron density fit to improve significantly.
A few points that may require different treatment in analyzing the data is that there is incomplete UT coverage for Millstone Hill and Sondrestrom radars, and for PFISR for a few specific months. Another point to be noted is that while Sondrestrom seems to have thorough coverage through June 2008, the other radars have less consistent coverage after March, 2008. Finally, I have alerted John to a problem retrieving June, 2007 data for Sondrestrom, although data exists for this period.
Week of July 20
This week I completed writing my MATLAB script to plot altitude-month contours, depicting seasonal variations in height. I improved the script from previous editions by binning the data for each node above and below the given altitude. Additionally I edited the code to produce scatter plots of the actual data as arranged by height. For some radars such as EISCAT, Svalbard and the Poker Flat ISR I was able to produce altitude contours for the purpose of comparing daytime and nighttime behavior. The Millstone Hill Radar and Sondrestrom Radar had incomplete hourly coverage for most months and I was unable to look at nighttime data.
Additionally, I completed an hour-month contour which used half-hour data bins to perform a least squares fit to the data in a given UT range. This type of plot differs from those depicting model values as a least squares fit is performed for each UT bin as opposed to the entire data set at once. These contours are less stable than the model contours despite including time steps above and below each given UT. When comparing these plots to the model plots there are some apparent differences. Most notably in the EISCAT data, the maximum electron density occurs typically occurs at or after July for heights greater than 200 km, while the maximum density in the data itself as well as the UT contours is at June. It is noted that there appears to be an issue with the altitude scatter from the June, 2007 ESR data - with unusually high densities at high altitudes. When the June, 2007 data is filtered out for ESR, however, the peak density on the height contour is shifted from June to August.
In preparation for analysis this week I produced all of the important figures for each radar (that is scatter plots of the data arranged by time or altitude and contour plots of the empirical models and binned data arranged by local time and altitude). For all of the model contours I determined which terms should be included for the most accurate fit. All of the seasonal and local time components were included in the fitting. After testing a 4-monthly variation term in my script I deemed the term unnecessary or unsuitable for my final fits. Generally I worked with either all of the cross-terms or just the main one, annual*diurnal. It should be noted for EISCAT that the November, 2007 data set was removed as it was undersampled and consists of seven experiments at the end of the month. Looking at the data from this month it was apparent that the F peak was unusually low, below 250 km typically, and included an anomalous nighttime peak just below 450 km. The fit accuracy improved after removing this month. I will now produce the model contours for the typical altitudes of ~120 km, 200 km, and 300 km for height comparison at a given site.