End of Semester Summary

This semester started off with Patrick introducing me to the OPeNDAP Project, which is a an open source server to distribute scientific data that is stored in various formats.  OPeNDAP is useful because of its ability to take data that is stored in one format and convert it to some other format that a researcher may want.  It also lets researchers look at some of the meta-data about the dataset.  From that meta-data the researcher may decide they only want to get a small subset of the data, and OPeNDAP is able to do that for them.

After Patrick explained to me the general structure and purpose of OPeNDAP, my first task was to get it installed on my machine.  This was a good experience for me as I had never installed a project from source code and I got to learn how to solve all the “fun” little problems that crop up, like resolving dependencies and making sure path names and environment variables are all correct.  All the trouble I had with it, however, was for the good as it allowed Patrick to find places in the OPeNDAP documentation that could use improvement and clarification.

After I had it installed, I began to familiarize myself with the code for it.  Shortly after I started looking through the code, there was an OPeNDAP development meeting at the TWC and I was able to sit in in-between classes and listen to what they had to say.  It was a very interesting to hear the thought process behind the design decisions outside of a classroom environment.  The thing I liked most about it was they got to choose how they wanted to accomplish the task, they weren’t constrained to using any specific technology, strategy, or methodology as most projects encountered in coursework are.

My task with OPeNDAP was to impliment a design change decided on at the development meeting.  I was to re-write the code so as to fundamentaly alter the structure of how it handeled requests.  As it was, OPeNDAP was structured so that a daemon process called the besdaemon would start up a process called the beslistener which would listen for incoming connections requesting data.  When the beslistener received a connection it would spawn a child process to handle the connection.  The purpose of the besdaemon was to continuously monitor the beslistener to make sure it didn’t go down, and if it did go down it would start it back up.  The modifications I was to make were to alter the code so as to make the listener process more robust. At a very high level, the changes I needed to make included creating a limit on the number of active connections allowed, devising a method for the main listener process (Abraham) to communicate with and keep track of its child processes (Issac) which were handling connections, as well as implementing clean and rude methods of terminating connections if the server was being shut down.

Before I could start making these changes, however, I was tasked with migrating the OPeNDAP content from the old wiki to the new drupal site.  This was a good learning experience in how drupal worked and was my first use of SPARQL queries.

Once that was done I was able to really dig in to the OPeNDAP code and start making modifications.  I really enjoy looking at new code and figuring out how it works.  The process of looking through and understanding what is going on and being able to look up functions I’ve never seen before and actually understand what the definition is talking about is very rewarding to me.  It really demonstrates to me how much I’ve learned in the past four years, and it’s a big confidence booster when I can figure it out on my own (but maybe with some occasional help from Phil and Dan 😉 ).

The biggest stumbling block I hit in my work was figuring out how Issac (the child listener which is spawned when a connection is received)  was going to get its PID to Abraham (the main listener) which needed to store its PID along with information about who Issac was connected to.  The problem was that Issac was spawned by forking twice, and Abraham only knew the PID of the intermediate child (which is the process that actually spawns Issac).  After thinking about all the possible ways I could get Issac’s PID to Abraham (where I got to enjoy the freedom of choosing how best to do it) I thought creating a pipe through which Issac and Abraham could communicate was the best solution.  Patrick agreed and I set to work.  After working on this for a while, I found that there was an issue with how I had set up the communications through the pipe.  Issac wouldn’t necessarily be ready for Abraham to start talking to him when necessary.  I brought this up with Patrick and he suggested adding threading.  I had never done threading before, so this was new to me, but after a little reading it turned out to be pretty easy to pick up.  With threading I could have a function in Issac that would always be listening for things on the pipe from Abraham and that solved my problem.

As the semester was drawing to a close, Patrick approached me and Linyun about working on incorporating OPeNDAP into the ESG project.  I was tasked with installing OPeNDAP on aquarius.tw.rpi.edu.  I thought this would be pretty straight forward, having already gone through the rigmarole getting it set up on my machine, how wrong I was.  Aquarius had a custom install of LDAP and it was wreaking havoc with the libdap library that OPeNDAP needed.  Eventually, with Patrick’s help, I found that all that needed to be done was add a few flags to the configuration and make a symbolic link that pointed the calls looking for the LDAP libraries to the new libraries, not the old ones.

Future Work:

As I will not be returning to TWC next semester due to the fact that I found a research position that is more in line with my interests, I will not be able to finish implementing all the necessary changes to the beslistener to get the desired functionality.  I Hope the comments I left in the code and on this blog are enough to help whoever follows in my footsteps finish what I started.  I was able to get the code I wrote uploaded to the OPeNDAP SVN mere hours before my computer experienced a catastrophic crash, so all my work is preserved.  As far as expanding OPeNDAP goes, I think it would be interesting to see how it might be useful in more than just a scientific data/research sense.  It would be cool if OPeNDAP could sit on top of a database of raw audio files (legal of cousrse 🙂 ) and could transcode them into the various formats a user might want (mp3, wav, FLAC, etc.) on demand.

The Undergrad Experience:

I was very happy to have the opportunity to work with the TWC this semester.  Patrick, John, and Joanne were very encouraging and made their best efforts to make sure we knew what we were doing and had what we needed to do it, though it would have been nice if the Xbox had gotten in sooner ;-).  I thought the links that John sent us were very interesting, especially the one about using the kinect to create 3D models.  There was also no shortage of opportunities to get involved with other projects going on at the TWC (I wouldn’t have minded an iPad to play with), but I simply lacked the time to get involved with more than what I was already doing.

As a final note, I would like to give a special thanks to all the Professors at the TWC, Professor Fox in particular for signing off all my paperwork :).  I would also like to thank John Erickson and Patrick West for all their help.  As is evident by this post, Patrick was a tremendous resource for me throughout this semester.  He is a great guy and really put 100% into everything he does.

Thanks for the Experience!


Jay Chamberlin


~ by chambj2 on December 10, 2010.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: