End of Semester Summary

•December 10, 2010 • Leave a Comment

This semester started off with Patrick introducing me to the OPeNDAP Project, which is a an open source server to distribute scientific data that is stored in various formats.  OPeNDAP is useful because of its ability to take data that is stored in one format and convert it to some other format that a researcher may want.  It also lets researchers look at some of the meta-data about the dataset.  From that meta-data the researcher may decide they only want to get a small subset of the data, and OPeNDAP is able to do that for them.

After Patrick explained to me the general structure and purpose of OPeNDAP, my first task was to get it installed on my machine.  This was a good experience for me as I had never installed a project from source code and I got to learn how to solve all the “fun” little problems that crop up, like resolving dependencies and making sure path names and environment variables are all correct.  All the trouble I had with it, however, was for the good as it allowed Patrick to find places in the OPeNDAP documentation that could use improvement and clarification.

After I had it installed, I began to familiarize myself with the code for it.  Shortly after I started looking through the code, there was an OPeNDAP development meeting at the TWC and I was able to sit in in-between classes and listen to what they had to say.  It was a very interesting to hear the thought process behind the design decisions outside of a classroom environment.  The thing I liked most about it was they got to choose how they wanted to accomplish the task, they weren’t constrained to using any specific technology, strategy, or methodology as most projects encountered in coursework are.

My task with OPeNDAP was to impliment a design change decided on at the development meeting.  I was to re-write the code so as to fundamentaly alter the structure of how it handeled requests.  As it was, OPeNDAP was structured so that a daemon process called the besdaemon would start up a process called the beslistener which would listen for incoming connections requesting data.  When the beslistener received a connection it would spawn a child process to handle the connection.  The purpose of the besdaemon was to continuously monitor the beslistener to make sure it didn’t go down, and if it did go down it would start it back up.  The modifications I was to make were to alter the code so as to make the listener process more robust. At a very high level, the changes I needed to make included creating a limit on the number of active connections allowed, devising a method for the main listener process (Abraham) to communicate with and keep track of its child processes (Issac) which were handling connections, as well as implementing clean and rude methods of terminating connections if the server was being shut down.

Before I could start making these changes, however, I was tasked with migrating the OPeNDAP content from the old wiki to the new drupal site.  This was a good learning experience in how drupal worked and was my first use of SPARQL queries.

Once that was done I was able to really dig in to the OPeNDAP code and start making modifications.  I really enjoy looking at new code and figuring out how it works.  The process of looking through and understanding what is going on and being able to look up functions I’ve never seen before and actually understand what the definition is talking about is very rewarding to me.  It really demonstrates to me how much I’ve learned in the past four years, and it’s a big confidence booster when I can figure it out on my own (but maybe with some occasional help from Phil and Dan 😉 ).

The biggest stumbling block I hit in my work was figuring out how Issac (the child listener which is spawned when a connection is received)  was going to get its PID to Abraham (the main listener) which needed to store its PID along with information about who Issac was connected to.  The problem was that Issac was spawned by forking twice, and Abraham only knew the PID of the intermediate child (which is the process that actually spawns Issac).  After thinking about all the possible ways I could get Issac’s PID to Abraham (where I got to enjoy the freedom of choosing how best to do it) I thought creating a pipe through which Issac and Abraham could communicate was the best solution.  Patrick agreed and I set to work.  After working on this for a while, I found that there was an issue with how I had set up the communications through the pipe.  Issac wouldn’t necessarily be ready for Abraham to start talking to him when necessary.  I brought this up with Patrick and he suggested adding threading.  I had never done threading before, so this was new to me, but after a little reading it turned out to be pretty easy to pick up.  With threading I could have a function in Issac that would always be listening for things on the pipe from Abraham and that solved my problem.

As the semester was drawing to a close, Patrick approached me and Linyun about working on incorporating OPeNDAP into the ESG project.  I was tasked with installing OPeNDAP on aquarius.tw.rpi.edu.  I thought this would be pretty straight forward, having already gone through the rigmarole getting it set up on my machine, how wrong I was.  Aquarius had a custom install of LDAP and it was wreaking havoc with the libdap library that OPeNDAP needed.  Eventually, with Patrick’s help, I found that all that needed to be done was add a few flags to the configuration and make a symbolic link that pointed the calls looking for the LDAP libraries to the new libraries, not the old ones.

Future Work:

As I will not be returning to TWC next semester due to the fact that I found a research position that is more in line with my interests, I will not be able to finish implementing all the necessary changes to the beslistener to get the desired functionality.  I Hope the comments I left in the code and on this blog are enough to help whoever follows in my footsteps finish what I started.  I was able to get the code I wrote uploaded to the OPeNDAP SVN mere hours before my computer experienced a catastrophic crash, so all my work is preserved.  As far as expanding OPeNDAP goes, I think it would be interesting to see how it might be useful in more than just a scientific data/research sense.  It would be cool if OPeNDAP could sit on top of a database of raw audio files (legal of cousrse 🙂 ) and could transcode them into the various formats a user might want (mp3, wav, FLAC, etc.) on demand.

The Undergrad Experience:

I was very happy to have the opportunity to work with the TWC this semester.  Patrick, John, and Joanne were very encouraging and made their best efforts to make sure we knew what we were doing and had what we needed to do it, though it would have been nice if the Xbox had gotten in sooner ;-).  I thought the links that John sent us were very interesting, especially the one about using the kinect to create 3D models.  There was also no shortage of opportunities to get involved with other projects going on at the TWC (I wouldn’t have minded an iPad to play with), but I simply lacked the time to get involved with more than what I was already doing.

As a final note, I would like to give a special thanks to all the Professors at the TWC, Professor Fox in particular for signing off all my paperwork :).  I would also like to thank John Erickson and Patrick West for all their help.  As is evident by this post, Patrick was a tremendous resource for me throughout this semester.  He is a great guy and really put 100% into everything he does.

Thanks for the Experience!


Jay Chamberlin

Continued installation on aquarius

•December 8, 2010 • Leave a Comment

I found the package I was missing to get the hdf4_module configured on aquarius.  The fix was
sudo apt-get install libhdf4g-dev
However, the ncml_module is still failing its tests. :-/ I’ll try a distclean and rebuild. Still broken. Apparently it needs the hdf5_handler too… needed to install libhdf5 headers.
sudo apt-get install libhdf5-serial-dev

HURRAY, ncml_module installed successfully!
here is the output of show version in the bescmdln:

It is important to note that in order to use the bes commands (besctl and bescmdln) the script /projects/esg/opendap/configure_environment.sh must be run so the proper paths are added to the environment.

Now that opendap is all taken care of, time to move on to Hyrax.  I got the servlet started no problem, but now I need to figure out how to configure it with opendap and thredds.  It’s supposed to pick up the opendap stuff automatically if besdaemon is running (which it is), but the default files aren’t there.  There are a few configuration files that I am trying to figure out, but there is an exam on campus waiting to be conquered, so I’ll pick this up later.

BESClient> show version;
<?xml version=”1.0″ encoding=”ISO-8859-1″?>
<response reqID=”some_unique_value” xmlns=”http://xml.opendap.org/ns/bes/1.0#”&gt;
<library name=”bes”>3.9.0</library>
<module name=”dap-server/ascii”>4.1.0</module>
<library name=”libdap”>3.11.0</library>
<serviceVersion name=”dap”>
<module name=”fileout_netcdf”>1.1.0</module>
<module name=”hdf4_handler”>3.9.1</module>
<module name=”hdf5_handler”>1.4.3</module>
<module name=”netcdf_handler”>3.9.2</module>
<module name=”ncml_module”>1.1.0</module>
<module name=”dap-server/usage”>4.1.0</module>
<module name=”dap-server/www”>4.1.0</module>

Libdap working on aquarius

•December 7, 2010 • Leave a Comment

So after much toil I have fixed libdap so that it will pass make check.  Before, it was complaining about libcurl not being able to find the version information for libldap and liblber.  All that was required was to fix the symbolic links for the libldap and liblber libraries in /usr/lib with the following commands

sudo ln -fs /usr/lib/liblber-2.4.so.2 /usr/local/lib/
sudo ln -fs /usr/lib/libldap_r-2.4.so.2 /usr/local/lib/

bes gave me some trouble for a while, but after I ran make distclean and then configured and remade it the rest of the installs went smoothly.  aquarius now has opendap installed with the netcdf_handler, dap-server, and fileout_netcdf modules.  The ncml_module is still giving me problems; all but one of the tests in make check are failing.  It appears to need to have the hdf4_handler installed.  I’ve been trying to figure out where I can find the header file mfhdf.h that the hdf4_handler is looking for, but it has been proving difficult.  I have installed all the hdf packages I can find and it’s still not working.  I will continue my investigation tomorrow.

Opendap and ESG moving forward

•November 30, 2010 • Leave a Comment

Opendap is now going to utilize threading to facilitate the communications between the main listener (Abraham) and its children (Issac).  This is should be an easy change and should simplify things greatly.  The only stumbling block is I’ve never done threading before :-/ so there will be some learning there.

Patrick was able to get libdap to compile today on aquarius, so installation of opendap can move forward there…. unfortunately libdap did not pass its tests, so there are actually still some issues there.


Pre Thanksgiving Installing

•November 23, 2010 • Leave a Comment

Attempting to install a few things on aquarius before thanksgiving.  While installing TDS I see that it is using some sort of OPeNDAP, which I was under the impression it didn’t have. http://aquarius.tw.rpi.edu:8080/thredds/

Patrick wanted the TDS to be installed in /projects/esg/tds but the installation only required that a .war file be placed in /var/lib/tomcat6/webapps and to restart tomcat.  Restarting tomcat was a little annoying.  I had to be root and set the environment variable CATALINA_BASE to /var/lib/tomcat before it would start back up.

I looked in to installing the LAS too, but it would be useful to have the catalogs we will be working with before hand so I’ll hold off on that until we get the catalogs.

Continued working on the beslistener and somehow broke the pipe which is used to communicate between the BESServerHandler and its child.  Not sure what I did, but hopefully copious debug statements will shed some light on whats happening.

Happy Thanksgiving.

Communicating children

•November 16, 2010 • Leave a Comment

I’ve been working on getting children listeners talking with the parent listener today.  I added a few new defines to the exit conditions header to facilitate their communication.  When a child exits,it now sends a message through the pipe to the parent.  Whenever the parent receives a new connection, before it checks if it can handle another connection, it looks to see if there is anything to read from the child.  If it reads an indication that a child finished, it removes that child’s pid file and decrements the _numConnected counter.  the parent will also ping the children it did not receive a termination indication from to make sure it is still alive.  I haven’t quite finished working out the details of the ping, mainly because I don’t know how long it might take for the child to respond and I am hesitant to have it block there while waiting for a ping response.  I might decide to have it send the ping out when it receives a connection and look for that ping the next time it receives a connection in the hopes that the child will have had a chance to respond to the ping by then.  I really just don’t know how long it might take a child to execute a command for the client, which is what dictates how long it would take for the child to respond to the ping.


•November 12, 2010 • Leave a Comment

Had a meeting with Patrick and Linyun Friday morning about integrating OPeNDAP with the Earth System Grid (ESG) project.  ESG currently uses THREDDS Data Server (TDS) to serve data, and they would like to replace it with OPeNDAP.

The question is whether or not Hyrax, the front end of OPeNDAP, can do enough of what THREDDS does for it to be a suitable candidate.  To determine this, it will be necessary to understand how ESG is currently set up.  This will involve looking up information on Live Access Server (LAS), Ferret TDS (FTDS), and NCML (the xml for netcdf).  To facilitate our testing of dropping OPeNDAP into ESG we will need to install THREDDS, and TDS on Aquarius and see if we can get some THREDDS catalogs from ESG to play around with.

Right now, the primary focus is just to get OPeNDAP into ESG.  In the long term we would like to integrate ferret, so we can graphically represent the data for users.