Activities

• Continuing the porting-process of the SWRAD modules
• Running benchmarks (of entire WRF simulations) on different resources
• Working on final presentation

   

Problems

• To get some results on different systems, Gabriel was able to run the simulations on his home computer, which has an nVidia 250 gt, and an Intel Core2Duo CPU. This configuration actually got worse performance for both the CPU and GPU versions of the test. This is a very bizarre outcome, considering that his computer is more powerful in every observable regard. The operation system distribution being used was the same as well. Furthermore, the GPU results were around 15% slower than the CPU results. He tried running a gprof analysis, but the execution failed.

   

• Another difficulty that I've found about working with cuda is that, even though blinking on the screen gives an indication that there is a problem with the code that is causing a runtime error, but the location of the runtime error is very vague. For exaple, while using our testing technique in which we only run certain sections of the code, commenting out certain portions causes this ``blink.'' However, this has shown some of the shortcomings of this porting approach that perhaps porting first to C would have helped with. For example, if I have a partial swrad kernel that simply sets one of the output variables to a static variable, the code runs fine. However, if I start modifying other variables, in a subroutine, even if I don't make any changes to the same output variable, the ``blinkage'' occurs and the variable's values are unset. If I set the variable's values to some function of the other variables, the code works fine. This makes the porting process difficult, since we need to modify small sections of code at a time to compare our results to the Fortran results.

   

Plans

• Continue with the porting
• Run tests on Teragrid

   

Some Final Pictures

Picture during my final presentation

Picture of Gabriel and I working at Esteban's Media Lab

Activities

• Continuing with port of WRF to CUDA
• Experimenting with the alternative solution of porting to regular C first.
• Experimenting with output verification script (for testing)

   

Accomplishments

• A rudimentary, partial port has been created, but we have not been able to test it.

Problems

• The GPU kernel templates at the NOAA site come with some testing scripts that compare the outputs of the variables on the the GPU and on the CPU. However, we have not gotten this to work. It seems to work fine and give good results. However, the problem is that it always gives good results. To test it, I used a stripped-down version of the WSM-5 module and compared the output to the full, CPU output, and the result from this program is the same. This leads me to believe there is a problem with the testing script.

Plans

• Figure out the problem with the testing script
• On Friday we converted the CUDA kernel to regular C. This required some effort due to the lack of automatically-generated variables that we had by virtue of using Michaelakes' scripts. However, it did reveal some bugs that had not revealed themselves in the CUDA implementation. Theoretically Michaelakes' script should make testing easier, but so far we have not been able to use it successfully. As a result, we are not sure which direction to go (i.e. whether to use regular C first or go straight to CUDA).

   

Last weekend, Gabriel invited me for lunch at his house on Saturday. There I met his maid and his menagerie of cats and dogs. After that, I went back to Rio for some more exploration and picture-taking. On Sunday, Esteban took me, Gabriel, and another temporary student who was about to leave, to the National Park of Tijuca Corcovado, where the statue of Christ the Redeemer, which I had been seeing from my balcony since day one, is located. The trip to the top is long and fun, as it involves a pretty long drive up the mountain, with several twists in between. The size and construction of the statue is very impressive. The view of Rio from up there is spectacular. After that we went to Praia do Pepe in Rio. This is another very nice beach with plenty of activities going on. There we went to a place with good açai. When we got back to Niteroi, we stopped by the Parque da Cidade in Niteroi, which also gives a very nice view of the city (this time, from the other side of the ocean). To close the afternoon in style, we went to the famous serbet place in Sao Francisco, which has very good sherbet.

Ipanema at dusk

Me in front of the Christ statue

Gabriel and I at the top of Tijuca Corcovado 

Praia do Pepe

View from the Parque da Cidade

This past Saturday, I went to Piratininga beach. My original intention was to go to Camboinhas, but there is no direct bus stop there and I did not want to venture. As it turns out, Piratininga is very nice. There are mountains and hills to walk up (and to dive down to the Beach, for those who dare). Apart from that, this was more of a relaxation beach. There weren't as many people playing sports as there are in Copacabana and Icarai.

Unfortunately, my cultural experience on Sunday involved going to a clinic. Apparently I lost my second battle with Brazilian cuisine, and got a bad virus. Gabriel was extremely helpful in this event, advising me on what to do and with translating for me at the clinic.

Activities

• Fixing CUDA installation problems
• Creating Skeleton of the SW-RAD physics module
• Working on documentation
• Understanding code

Accomplishments

• Working CUDA skeleton has been developed. All (40+) parameters that are passed into the Fortran version of the SWRAD module are being passed to the GPU. Some tests were performed to ensure that the values being passed in and sent back get modified.

   

• We have a better understanding now of the WSM-5 code, including the code that they wrote for helping automate the translation from Fortran to CUDA, as described in the paper.

   

Problems

• We still faced several system-related issues this week. We realized that CUDA does not always produce the same results, is sensitive to programming errors (but does not fail gracefully). For example, we spent an entire day installing different versions of Ubuntu with different versions of the CUDA driver, after realizing that different results were being generated for the same code. In the end, it turned out to be a programming error that did not always reveal itself. We believe there are bugs with certain versions, although we haven't been able to narrow any specific issue wee have had to a version issue.

   

• One of the issues that we recently found out about is that there is a limit of 256 megabytes of shared memory for arguments. On 32-bit architectures, the skeleton works. However, on 64-bit, an error is reported at compile-time, similar to the one in [1]. This is a problem so many arguments have to be passed to the kernel. We may be able to mitigate this by eliminiating unneeded parameters. Reference: [1] http://forums.nvidia.com/index.php?showtopic103440

   

Plans

• I have been looking into the I/O and the test program provided by Michaelakes. All of the I/O is written after the entire WSM5 calculations are done. Therefore, to perform an incremental port, I think the best solution is to break the existing fortran code into sections, and only do one section at a time. When optimizing, we will need to rearrange things, but that can be handled later.

   

• This week we will start the actual porting. We have not thought much about optimization, but I have been trying to keep in mind my goal of modeling the performance of GPU-enabled programs while working. I think one of the biggest challenges will be accounting for the placement of certain variables in memory. In Michaelakes' paper, he explains how he put certain variables in registers since they were used often. The transfer of memory from CPU to GPU in general will be an interesting problem.

   

Paper Review

John Michalakes, Manish Vachharajani: GPU acceleration of numerical weather prediction. IPDPS 2008: 1-7

This paper discusses the GPU implementation of the WRF WSM-5 module. The paper includes motivation for porting to CUDA as well as for the module that was ported. For the majority of the paper, the authors discuss the procedure taken to port the module from Fortran to C, some basic optimizations, and some results comparing execution time on a single CPU and on GPU. A good overview of how they did the port is provided. They describe the challenges faced when porting to C. For example, globally addressed arrays had to be addressed locally in C. They also show how they use special directives to provide some transparency in porting from C to CUDA. These directives allow them to specify which variables should go in registers, and they do automatic indexing of n-dimensional arrays going to and from the GPU. The GPU implementation resulted in a 17x speedup, and the results obtained were verified to be indistinguishable from the Fortran implementation.

Pictures

Wednesday we had our first weekly meeting. Gabriel and I gave an informal overview of what we have been working on, and shared some of the problems that we are facing. Before us, Marcelo (far left) gave a presentation on CUDA in relation to a problem with Petrobras. The gentleman to his left, who works is involved in the Petrobras-UFF alliance, was kind enough to attend both of our presentations and share his insight.

Gabriel and I working at the lab

Activities

• Setting up the test systems
• Reading WSM-5 code and working on documentation.
• Developing skeleton for the SWRAD module, based on WSM-5 code. (The WSM-5 module is the one that has already been ported to CUDA)

   

Accomplishments

• Got CUDA working on my system and WRF working on Gabriel's system
• Got a running SWRAD skeleton

   

Problems

• There is not a lot of documentation in the code itself, so a lot of reading and trial and error has been involved. For example, there is a Perl module, which is discussed briefly in the IPDPS paper, that automatically maps variables' location in (video) memory, using some directives that are placed int he code. However, the location and syntax of these directives is not documented nor obvious.

   

Plans

• Start porting portions of the code to CUDA
• Document findings into a technical report, to simplify writing of the paper later on.

During the weekend, I had to pleasure of visiting the ``marvelous city'' of Rio. Esteban asked Gabriel and I to join him for a boat ride that takes you through Rio. This tour provides a great view of the city. It also gave a good view of the city I am staying at, Niteroi, since it is literally right across the water.

Esteban and I

Me on the boat, with Rio behind me

On Sunday I went on my own adventure to visit the famous beaches in Rio, Ipanema and Copacabana. The scenery and atmosphere there was awesome. The beach is full of people playing sports, sunbathing, swimming, and walking. (Copacabana alone stretches around 5km). Ipanema also has a "mini-mountain" for climbing.  Definitely some of the nicestbeaches I have visited.

Tuesday morning I had to get my first batch of laundry done. Fortunately, the guys that came before me did the
grunt work of figuring out how this works and explaining our needs to the laundry department here at the apartment.
The lady knew what to do so there was not too much confusion involved.

Tuesday night was an interesting experience. I went to join the closest gym, which is about 20 minutes away. The
membership included a physical evaluation and personal training session (I guess Braziians are all about being at-your-service).
I got to work on my Portuguese in a practical setting - strength training. Somehow I managed to get through this event using several hand gestures, interpolation of words, and use of my ``portunhol.'' Fortunately, Brazilians are typically very friendly and patient, so it was a fun experience. I also learned some exercise-related words and phrases along the way.

Picture of Copacabana

Local creativity and display of affection for American pop culture

This is my first blog related to my stay at Brazil. It is a summary of the past  week and a half. I arrived on Wednesday around noon and was taken to my apartment by my host advisor, Esteban Clua's, student Marcelo Zamit. There I met with one of the guys that was already staying at the apartment and later with my main team-member at UFF, Gabriel Gazolla. Our plans for the brief first week were simply for me to get oriented with the place and prepare the systems to start working on the implementation of the modules. This included setting up WRF on Gabriel's system and setting up CUDA on my system. We were able to do this, albeit with some minor setbacks.
 

View from the apartment

up-close view of the "Christ the Redeemer" Photo, taken from the appartment

Compiling WRF on Gabriel's machine was not too hard. It is an Ubuntu system, which I had never compiled WRF on, but there were no major setbacks. Installing CUDA on my system was more troublesome since I have a Sony VAIO Z with the ``hybrid graphics'' system. (Basically, there are two graphics cards in the system). Linux needs some work in order to use the nVidia card, which is the one that is supported by CUDA. After some web searching and some system-administration-related troubleshooting, we eventually figured it out and were able to compile and run some CUDA examples.

Monday, while we were ready to get started with the CUDAing, we ran into a problem. I was getting "no device
found" errors. After two or so hours troubleshooting by reinstalling different versions of CUDA, it turned out to be the fact that I had desktop effects enabled on my system.
Tuesday, we accomplished another milestone by finally compiling the WSM-5 kernel and successfully running it.
Apparently, one of the compiled binaries is left behind by the makefile and helper scripts. So I was running an
old version of one of the scripts, compare_snaps, for a couple of hours and wondering what the error might be.
As with many errors, we solved it in a weird way. I copied the files from my computer to Gabriel's. Since
they are different architectures, all the programs had to be recompiled (or else the system complained). This
is where we found the problem.
 
The rest of the week was spent looking at code and doing some basic code-porting tasks. We finished the week off by running the SW-RAD module that we are attempting to port, using the skeleton provided by John Michaelakes, and a dummy kernel.

 

Perhaps the view will provide inspiration while working

How I miss Brasil, this was probably the most incredible experience of my life.  I have never had so much fun working on projects, learning, and laughing so much in my entire life.  The times I shared with my fellow FIU PIRE students and the CS people at UFF will always be in my heart. (wow that is sappy) now for some great pictures.

Oh I forgot to mention my first love in Brasil.  THE FOOD!

Seriously the food.

Also the drinks.

....After our lunch, Zaidi, Seijiro and I went to the "Maracana" one of Rio de Janeiro's famous landmarks. I have been dying to go to this place for years, and to finally be given the opportunity to experience a soccer game at "Maracana" was amazing. We arrived earlier than what we should have, one of the PIRE students thought the game started at 4:30pm which in reality started at 6:30, so we had to wait for almost 3 hours for the game to start. In the mean time we took some pictures out the stadium

At 4:30, they started letting people inside the stadium, we still had 2 hours left for the game to start, so we walked around and started taking pictures of inside the stadium 

Now the game finally started and let me tell you, it was one hell of an experience. I've been to many soccer games in Miami, and nothing an compare to the atmosphere of the Maracana. We decided to sit on the Flamengo side, the fans their were rowdy, jumping up and down, banging the drums and singing through out the entire game. The game was a nail bitter, the final score of the game was Flamengo 2, Botafogo 2, it ended in a draw.