Thanks to Jorda from the lab, I was able to create a cusom mapReduce/Hadoop datatype. It's called wholeFileWritable. It allows me to read an entire file as a record. This is useful in the case of working with web pages. You dont want each map class to process one line of the HTML page, you want to parse the entire page in order to analyze the content and properly extract the necessary information.

I will post the class soon so anyone else that is working on Hadoop can make use of it! =) 

Also, funny thing, I noticed just today that college campuses look the same no matter where in the world you are. Check out these pictures, this could be anywhere in the northeast or the carolinas! Well, except for that last one since its got the castle in the background.

A  new issue came up which is very interesting. Java doesn't always work the way it's supposed to in Spain. The American Airlines website, the CNN.com website, and many other enterprise level sites do not work properly. Here, video will not load, applets will be missing, buttons will not function, etc.

I have been studying the issue and can find no reason as of yet for this occuring. I remote login to computers in the UK, Germany, and France and they all work just fine. I have tried about 10 different locations here in spain, well barcelona to be exact, and the same thing happens!

Please do share if you have had a similar experience. 

Today I was invited to observe some of the youth training camps for RCD Espanyol, one of the teams here in Barcelona. No it wasn't the Barcelona team you hear about on ESPN. This was the NY Mets of Barcelona soccer. You have 2 big teams. Barcelona(Yankees/Lakers) and Espanyol(Mets/Clippers).

Still it was tons of fun! As some of you may know I dislocated and big time messed up a few of my digits so I was unable to partake of any field interaction but I still had fun observing. Here are some pictures. 

I asked myself this question because we are working on building a MapReduce app that will run on about 40-60 nodes so we need to work on a dataset that is about 60GB  in size. The answer was right in front of me, whats the biggest dataset on the web...THE WEB!

I'm going to try and go about crawling the web and downloading 60GB of it. Well that or I'm going to try and find some spatial data such as high res images. I figure multimedia might work as well, though thats much more difficult to index so I am not sure how to work with it. 

The people here are great, we will definitley find some interesting results I am certain. Back to the field I go to try and gather up 60GB!

Ahhh, USASpending.gov launched a new "IT Dashboard" to provide statistical analysis over the spending data. Basically what I was trying to help IBM do. I am not sure how accurate it is but, it definitely is pretty to look at.

The biggest concern I have is that they are now adopting a completely new hierarchy which is the hierarchy for the CFO Act agencies. The CFO act agencies do not have a one to one relationship with the agencies in NIST SP 800-87 or FIPS 95-2!

I think I'm going to contact NIST, FIPS, FPDS, or anyone I can get a hold of and voice my concerns. We have a great research opportunity here and I'm going to do my best to exploit it

I realized there is no way to post-date these blogs. So, if you're reading this just keep in mind that the dates are not accurate. Barcelona is great so far settled into the apartment and found the lab. No one was there because it was late afternoon but, I will go back tomorrow morning and get all settled in.

I'm looking forward to working on a different domain other than government. It was fun but, I would definitely like to have a go at sports statistics and other historical data that we can use to predict the future! Hopefully we can create a crystal ball of sorts...or at least a magic 8 ball.

I've been trying to decipher the US government agencies hierarchy as contained in NIST SP 800-87 as well as its predecessor FIPS 95-2. Its become a mountain of a task due to all of the formatting inconsistencies and possible typos in the documents. The official government stance is that their hierarchy is contained in the PDF versions of the documents I mentioned and nowhere else. So we are forced to use those as our sources. systemT is doing a great job of helping us extract the information but, we are getting unwanted side affects due to the aforementioned formatting and typo issues.

I spoke with someone at NIST and they said that if we get this done we will be the first ever! = ) Lets keep working and see what happens.

Today I discovered something called systemT by IBM. Using regular expressions in a language they call AQL one can extract information in the form of tuples from any text document. This has made my life easier in so many ways!

Be certain to check it out, it's in a paper called "systemT". I can't post details here on the other things I'm doing due to the confidentiality agreement. 

I have been working furiously in the lab to assist with the extraction of the US government's agency hierarchy in order to facilitate the tracking of spending. It's required 12-15 hour days and many sleepless nights. Now, it was time to take on some first hand in the field research!

60 miles from any cell phone reception and even further from the nearest Walmart or McDonalds I stumbled onto this.

Outside of the IBM research labs I found a vast expanse of amazing scenery...and on one sunny day I also found cows, turkeys, wild boar, dear, and I could have sworn a montain lion. Well, it was either a mountain lion or a dry bush, either way, pretty scary!