Computer Science Senior Seminar

Project Abstract

Pick your religion:

Paper Outline

Pick your religion:

Project Overview

Pick your religion:

Paper (draft)

Pick your religion:

Project Todo

Get Perl script working for database extraction/formatting.
Determine which section of Cligen code I marked PARALLELIZE can actually be parallelized.

Project Journal

2005-09-12

I've been in contact with one of the maintainers of Cligen. He's cleared up some points for me. Cligen is actually a weather generator, not a prediction engine. That is, it doesn't what weather will occur tomorrow, but can give some models for what could happen. I don't think this will screw me up too badly, though.
The plan right now is to find the meat of Cligen and move it out into a C source file. I can then parallelize that with MPI easily, without needing to know much Fortran.

2005-09-19

I think I've determined the proper format for the Cligen station (.PAR and .GDS) files. The first line has the station ID, the country ID, the station name, the latitude and longitude, and elevation. The next lines cover every day available, with year/month/day, max. temp., min. temp., and precipitation. I guess I don't even need solar radiation for this.
I'm in the process of writing a Perl script that will extract the daily values from the database and plop them in a flat file ready to be added to a Cligen station file.

2005-09-20

The Perl script is going to be the critical thing to get done. Right now I'm trying to figure out an easy way to iterate by day in the Postgres timestamp. Ideally I'd do this completely within Postgres, but I need to figure out the right function or date output for that.

2005-09-28

Using the internal date_part() function in Postgres, I've gotten my Perl script to extract timestamps as seconds from the epoch. I'll be able to use this to get minima and maxima over one day (86400 sec).
I then use Postgres's to_char() to format properly each part of the date I need (YY, MM, DD), and also the data (XXXXX.X).
My next problem is going to be figuring out how often the rain meter is reset. If it's not regular, I'll have to be careful how I extract that data, as a reset in the middle of a day could potentially give erroneous data.
I will also need to calculate the minima and maxima temperature for each day, either with Postgres MIN() and MAX() functions, or with Perl.

2005-09-30

My biggest problem right now is finding sources for my paper. There's some books I have on ILL, but all the useful articles I can find on-line are in journals the college doesn't have access to. I think weather forecast/modeling is more esoteric than I thought.

2005-10-03

I've done a quick once-over on cligen.f looking for things to parallelize. I added PARALLELIZE comments at those points, and I'll concentrate on those for possibly moving them out into a separate C file for linking.

2005-10-05

Rather than get a Perl script format a file for Cligen, I'm going to get Cligen reading directly out of the database. I need to track down all points when Cligen reads a station file, and then rewrite those subroutines (in C?) to read from the database instead.
I'm waiting on Mary Tamm to write the code that will zero out the rain meter on a regular basis. I also have to write some code that will interprety the random times the rain meter got zeroed out before.

2005-10-07

I think I really need to get a book on FORTRAN. There's too many functions I don't understand.
Charlie recommends going with embedded SQL in C. Here's a link.

2005-10-10

I'm running into difficulties with embedded SQL. I think I need to understand the relationship between ecpg and the C preprocessor better.

2005-10-11

Found some good FORTRAN references:
http://www.ictp.trieste.it/~manuals/programming/sun/fortran/prog_guide/11_cfort.doc.html
I'm going to spend some serious time going over these. It looks like they have everything I need to learn FORTRAN.

2005-10-12

I'm going to use Cligen's sta_dat subroutine as my hook to SQL. It's where all the data is snarfed up from the station file.

2005-10-15

I think looking at lines 815 (sta_dat call), 2381 (sta_parms call), and 2704 (actually reading) will be useful for SQL integration.

2005-10-17

Two-directional C-FORTRAN communication ("Hello World"). Call C function from FORTRAN with varchar, then pass back to FORTRAN and print. Test char,int,array.
Copy and paste sample file to make it bigger and run longer.
Find FORTRAN book in Wildman.

2005-10-24

I've found many FORTRAN references in Wildman.
After tracing the problem to a possible interaction within C itself rather than at the C-FORTRAN interface, I wrote an equivalent C-C program. After some consulting with K and R, everything now works by passing pointers around within function parameter lists.
Next step is to rewrite FORTRAN to pass by reference within the function parameter list.

2005-10-28

I needed this.

2005-11-10

I'm working on concatenating an option-6 input file and rewriting the years to generate a big enough input for gprof to be useful. The problem is that around line 2316 in cligen.f the input file ends up with a leading newline. This needs debugging.
I'm looking at how I want to do the UI. I'm debating between the AJAX paradigm and just straight-up Perl CGI.

2005-11-14

I'm running into serious problems with FORTRAN character strings. Every string has a leading newline prepended to it, so none of the file I/O works properly. I'm trying to get a so-called TRIM function working that will strip the leading blanks and turn them into trailing blanks.

2005-11-15

Turns out that the problem is actually syntax errors within the input file. I'm working with a guy from Purdue in track down what's going on.

2005-11-15

Found a good article from Argonne, written in plain English about parallelizing weather algorithms: "Regional Weather Modelling on Parallel Computers". It's in CVSWeb in my BibTeX file.

2005-11-19

I'm in contact again with Charles Meyer from the Cligen project. He's giving me a hand with getting the input files to run properly.
I'm going to try to get the Forest Service input files working, because those apparently have a lot fewer errors.

2005-11-21

I finally have gprof data. Preliminarily, I'm looking parallelizing chi-squared, because that is used in a number of functions and uses upwards of 75% of the compute time.
In parallel with that, I'm going to expand my data set to refine how much time is spent in randomization/ chi-squared.

2005-11-23

I've got good gprof data from a bazaar node, from a run that was about a minute long. Still looks like ranset to me.
Chi-squared calcs are done with chicnt, which is a 3d array. I need to figure out how to pass multidimensional arrays between FORTRAN and C.

2005-11-28

I'm trying to figure out what is easier: declaring/allocating arrays in C or in FORTRAN.

2005-11-30

I think FORTRAN MPI is the solution.
Here's an example.

2005-12-07

I now think FORTRAN OpenMP with Sun's parallelizing compiler is the solution.
I have the database aspect mostly wired up. I've input an Option-6 file into a Postgres database, and then outputed it again. It all works, except for dates after 1900. I think there might be a leap year issue here, but I'm not sure.
Next step is to code up the CGI. It should allow outputing the input file, outputing Cligen's output, and changing the input database.

2005-12-08

Using a TEMP database that's a copy of the original for generating custom Option-6 input files was a mistake. Transactions+rollback is much easier.

You can follow my progress from the CVSWeb interface.

Many thanks to Charles Meyer of Purdue for his timely help in running Cligen.

Last updated: Thursday, 08-Dec-2005 05:04:16 EST