A quiet day and some statistical fun

I recently enjoyed the slides from Peter Aldhous on "Data-Driven Journalism", he is the San Francisco bureau chief for the New Scientist magazine and knows a thing or three about this topic. It follows similar comments from Tim Berners-Lee on the topic. It is becoming increasingly more popular to use tools like R to explore these datasets, take this Mexican mortality example for instance.

On the Irish side, we are slowly getting places. It is more thanks to efforts by individuals such as Gavin Sheridan of TheStory.IE that Freedom of Information (FOI) requests are being made to explore the dark areas of the public sector that could do with a little more clarity given the times we are now in.

This spurred me to put together a short little R example (Source code on GitHub) exploring just a fraction of this data for the Mid-West region.



On the flip side of this post, I'll raise my hand and say that my new job has been keeping me very busy indeed. However, if this kind of thing is something you would like to see or hear more about for a commercial opportunity then my team and I at the Irish Center for High End Computing can help. Take a look at the kinds of data analytic services we offer and you can see I've spent even more time in R in the last year!

R for Global Wind Wave / Wave Swell Forecasting

As a follow up to using freely available existing climate datasets, here's one that might be of interest to those surfers and sailors out there. It is similar approach to that used for the temperature forecasting so go read that post first, it will help.

The free information from the U.S. NOAA/National Weather Service's National Centers for Environmental Prediction, in particular its Environmental Modeling Center and its WaveWatch dataset. It is stored as a GRIB2 files and this needs to be pre-processed using wgrib2 (available on MacPorts) to the netcdf format. This stage is similar to the previous post.

There were a few little glitches with this dataset that are immediately visible and the fact there is a big white patch in the Atlantic is due to the maximum height being stored as 9.999e+20. This normally won't be a problem but unfortunately the data also stores dry land with this value. A workaround would be to know or compute longitudes and latitude such that we could tell if a given position was over land or over sea prior to processing it. Unfortunately as this is a bit of a quick and dirty hack, I didn't have the time so here is the basic approach warts and all.

# See http://www.nco.ncep.noaa.gov/pmb/products/wave/nww3.t00z.grib.grib2.shtml
# Data was from ftp ftpprd.ncep.noaa.gov dir: /pub/data/nccf/com/wave/prod/wave.20091227
# wgrib2 was installed using macports
# /opt/local/bin/wgrib2 -s nww3.t00z.grib.grib2 | grep "HTSGW:surface:an" | /opt/local/bin/wgrib2 -i nww3.t00z.grib.grib2 -netcdf WAVE.nc

library(ncdf)
waveFrac <- open.ncdf("WAVE.nc")
wave <- get.var.ncdf(waveFrac, "HTSGW_surface")
# Dirty hack to fix input model, look for another better solution
wave[wave>9.99999]<- -1
x <- get.var.ncdf(waveFrac, "longitude")
y <- get.var.ncdf(waveFrac, "latitude")
library(fields)
rgb.palette <- colorRampPalette(c("snow1","snow2","snow3","seagreen","orange","firebrick"), space = "rgb")
image.plot(x,y,wave,col=rgb.palette(255),axes=F,main=as.expression(paste("Significant Height of Combined Wind Waves and Swell in Meters 2009-12-27", sep="")),legend.lab="Meters")
# Add a rough outline for islands, countries, and continents
contour(x,y,wave,add=TRUE,lwd=0.25,levels=0.2,drawlabels=FALSE,col="grey30")
# Add the source of the file and ftp location
text(130,-75,"Data source ftpprd.ncep.noaa.gov pub/data/nccf/com/wave/prod/wave.20091227")

R for Global Climate forecasting

Joe has a great post on using R for exploiting existing climate datasets to create some really nice weather maps. Using the free information from the U.S. National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS), he shows how R can be used to interpret GRIB2 files after they are pre-processed using wgrib2 (available on MacPorts) to the netcdf format.

I've slightly modified the code from Joe but its pretty much as he wrote it.

# ftp.ncep.noaa.gov pub/data/nccf/com/gfs/prod/gfs.2009121700
# See http://www.nco.ncep.noaa.gov/pmb/products/gfs/gfs.t00z.sfluxgrbf00.grib2.shtml
# eoin-brazils-macbook-pro:~ eoinbrazil$ cp gfs.t00z.sfluxgrbf03.grib2 temp2.grb
# wgrib2 was installed using macports
# /opt/local/bin/wgrib2 -s temp2.grb | grep ":LAND:" | /opt/local/bin/wgrib2 -i temp2.grb -netcdf LAND2.nc
# /opt/local/bin/wgrib2 -s temp2.grb | grep ":TMP:2" | /opt/local/bin/wgrib2 -i temp2.grb -netcdf TEMP2.nc

library(ncdf)
landFrac2 <- open.ncdf("LAND2.nc")
land2 <- get.var.ncdf(landFrac2, "LAND_surface")
x <- get.var.ncdf(landFrac2, "longitude")
y <- get.var.ncdf(landFrac2, "latitude")
library(fields)
rgb.palette <- colorRampPalette(c("snow1","snow2","snow3","seagreen","orange","firebrick"), space = "rgb")
tempFrac2 <- open.ncdf("TEMP2.nc")
temp2 <- get.var.ncdf(tempFrac2, "TMP_2maboveground")
newtemp2 <- temp2-273.15 # Convert the kelvin records to celsius
image.plot(x,y,newtemp2,col=rgb.palette(200),axes=F,main=as.expression(paste("GFS 24hr Average 2M Temperature 2009-12-21 00 UTC",sep="")),legend.lab="o C")
contour(x,y,land2,add=TRUE,lwd=1,levels=0.99,drawlabels=FALSE,col="grey30") #add land outline
text(120,-85,"Data source ftp.ncep.noaa.gov pub/data/nccf/com/gfs/prod/gfs.2009121700")

The image continues to show cold conditions in Europe, and the continuing heatwave in Australia.

Following recent Infomaps with R trend - Visualizing Irish unemployment rates with a choropleth map

Following on the recent trend of InfoMaps from Flowing Data and a call to R users to attempt their own examples. I saw this lovely German visualization with R and decided it was time to do Ireland.



Looking at these images, I'm reminded of the lyrics of the song "What a Difference A Day Made", the one that goes like

What a difference a day made,
Twenty-four little hours,
Brought the sun and the flowers,
Where there used to be rain...

Playing with R to animate SBP / RedC polling data

Another quiet afternoon so I had a look at the excellent site, "From Data to Graphics" and came up with this little animation showing the last three years of Sunday Business Post polls results with the last general election results.

Visualising Lisbon 2009 and 2008 by constituency

The results are in. Lisbon 2 has been passed by Ireland. Voting results are available Irish Election, from RTE, and the IT. I've added a couple of visualisations using dot plots to help show where it all changed from last year.

The 2009 Lisbon Referendum Ireland voting results by region and constituency.


Here are the 2008 results for comparison.

Lisbon Red C Polling 2009 Dot Plots

I've been playing around with R and dot plots. Today's SBP poll from RedC was a prefect excuse to have some fun.

LisbonRedCPolling2009-IncludingDontKnows
Originally uploaded by bluepelican

LisbonRedCPolling2009-ExcludingDontKnows
Originally uploaded by bluepelican

Lisbon 2 Round up

It's almost time for Lisbon 2 so to I've brushed up my readings to take the new changes into account and decided to post a few of the more interesting pieces I've found online.

Elvis making his case against the treaty:


A very nice summary of the social side of the treaty, slightly pro but I guess it depends on how you view it.


Jim Corr and the New World Order.... oh and how it relates to the Lisbon treaty


Michael O'Leary - 5 Reasons to Vote Yes


The Forum for Europe has collected a number of debate transcripts that they held and printed them online.
Defense, Foreign Policy and Neutrality Yes: Pat Cox No: Joe Noonan
Ireland and Social Europe Yes: Ruari Quinn No: Joe Higgins
Institutional Arrangements Yes: David Byrne No: Susan George
Economic Implications Yes: Peter Sutherland No: Mary-Lou McDonald.

Neutral Sites with Information:
Citizen's Information Service - Guide to Lisbon Treaty
Referendum Commission

Guides to support the arguments
YES: The Improved Spoofers Guide to the Lisbon Treaty
NO: Europe of Freedom and Democracy (EFD) (including UKIP) Guide to the Lisbon Treaty

I've listened again to many false arguments and blatant lies so I'm voting YES. The times are different now and it is even more critical to be part of the wider European community for so many reasons. Whilst I'd love to give the Government a bloody nose by voting No, it is sheer stupidity to think that way as a referendum is not a general election and won't cause any change to the government. I do believe Ireland needs a change of direction and government, voting YES is the first step in that process.

Limerick City Election Results since 90 including 09 first count

Courtesy of John Browne

These figures show a scary trend for FF and for the now deceased PDs. These figures are based on the tallies for the 2009 LE results and are subject to revision but the trend itself is quite interesting for a major urban centre. The second point is the sheer personal vote of Willie O'Dea if these figures reflect a `party' vote.

** UPDATE 21:41 - New chart reflecting actual first count figures rather than tallies. **

Votewatch.eu or What did MEP X, Y, or Z do ?

It's that time again with the European election nominations just closed that you might actually be interested in what your MEP has been up to for the last term. Want to check how they voted or how often they were in Brussel ? Thanks to http://votewatch.eu you can.Read more »

Arduino Mega and Prototype Shield Assembly

I've had some Sunday afternoon fun and games with the assembly of a prototyping shield. It is the design from e-lioness and is for the new Arduino Mega board. The new Arduino Mega board is designed to wow, with 128 KB of Flash or program memory, 8KB of RAM, and 4 KB of EEPROM. The board has a massive 54 digital pins (14 of these offer PWM output), 16 analog io pins, 4 hardware serial ports, and I2C. It is compatible with most of the earlier shields designed for the standard Arduino. The firmware includes the now standard, auto-reset on upload, a pre-burned bootloader, and automatic power selection. It does need you to update your Arduino to at least Arduino version 0015 or later. Here is a little table comparing the latest hardware from the Arduino and a close mid capacity derivative from the RepRap project, the Sanguino.



The Arduino Mega is a very sweet option for installations and sensor heavy projects.

Here's the starting picture:


Here's how it ended up:


The process is visually documented in my Flickr images and doesn't really need any more documentation. I would possibly check around to see if other prototyping shields have a better track layout for the digital pins as this turned out to be a little messy.

An example of the fun you can have with the Arduino Mega is shown by Tom Igoe's post on ``8×8 LED matrix control on an Arduino Mega''. He has a great example using it to play Pong.

Sun, Sea, Sounds, and Arduino at SMC 2009

If you're interested in learning about the sounds of Porto and physical computing then consider participating in the upcoming SMC Summer School on Interactions with Environmental Sounds. It is going to be held in Casa da Música, Porto, Portugal from July 18-21. There is funding available from the COST Action ICO601 on Sonic Interaction Design, for third level students interested but the application deadline is approaching (24th of April). There is a whole range of interesting project proposals already online. Guest lecturers will be given by Marcelo Wanderley of McGill, Joel Chadabe of NYU (and EMF), and Xavier Serra from the Music Technology group in Barcelona. Tutors include Jean-Julien Aucouturier, Stephan Baumann, Bram de Jong, Luis Gustavo Martins, Rui Penha, Stefania Serafin, Federico Fontana and myself.

Check out the Summer School website for details and application forms.

CS4062 - Fourth Arduino Lecture

CS4062 Masters in Interactive Media - Fourth Arduino Lecture - March 18th 2009 - University of Limerick. This lecture presents a short review and introduction to programming concepts relevant to Arduino. This was aimed at a digital media / music technology masters student audience.

Arduino Lecture 4 - Interactive Media CS4062 Semester 2 2009

View more presentations from eoinbrazil.

CS4062 - First Arduino Tutorial - Talking to PureData and to Flash

CS4062 Masters in Interactive Media - Tutorial Example 1 Arduino and PureData - March 18th 2009 - This is a simple worksheet example for a class tutorial on getting PureData and an Arduino to communicate. This was aimed at a digital media / music technology masters student audience.

Tutorial Example 1 Arduino and PureData

Publish at Scribd or explore others: Notes Academic Work interactive media puredata

CS4062 Masters in Interactive Media - Tutorial Example 2 Arduino and Flash - March 18th 2009 - This is a simple worksheet example for a class tutorial on getting Flash and an Arduino to communicate. This was aimed at a digital media / music technology masters student audience.

Tutorial Example 2 Arduino and Flash

Publish at Scribd or explore others: Notes Academic Work interactive media digital media

CS4062 - Third Arduino Lecture

CS4062 Masters in Interactive Media - Third Arduino Lecture - March 11th 2009 - University of Limerick. This lecture presents an introduction to motors, LEDs and Arduino with examples. This was aimed at a digital media / music technology masters student audience.

Arduino Lecture 3 - Interactive Media CS4062 Semester 2 2009

View more presentations from eoinbrazil.

The slides with notes are available here.

CS4062 - Second Arduino Lecture

CS4062 Masters in Interactive Media - Second Arduino Lecture - March 6th 2009 - University of Limerick. This lecture presents an introduction to communications and Arduino with examples. This was aimed at a digital media / music technology masters student audience.

Arduino Lecture 2 - Interactive Media CS4062 Semester 2 2009

View more presentations from eoinbrazil. (tags: avr prototyping)

The slides with notes are available here.

CS4062 - First Arduino Lecture

CS4062 Masters in Interactive Media - First Arduino Lecture - March 4th 2009 - University of Limerick. This lecture presents an introduction to the Arduino with examples. This was aimed at a digital media / music technology masters student audience.

Arduino Lecture 1 - Introducing the Arduino

View more presentations from eoinbrazil. (tags: avr physical computing)

The slides with notes are available here.

Germans wise up to e-voting

Germany has gone back to the paper ballot today with a decision by the German Federal Constitutional Court to immediately cease the use of electronic voting machines, in particular the Nedap ESD1 and ESD2. The Court said its decision was based on the evidence that the current technology had defects and was difficult to control. The devices were felt to contradict the principle of public elections with vote transparency as their defects make it possible to lose secrecy and control over the counting of the votes. The case was brought based on the 2005 Bundestag election in Karlsruhe, however the court ruled on this aspect stating that they were no hints of errors and left the results of this election stand.

The case was brought forward by a father and son team. Joachim Wiesner, a retired policital scientist and his son, Ulrich Wiesner, a software developer. An expert witness against e-voting was Professor Wolfgang Lawer from Bonn. He spoke to the court and stated his concerns about a ``a vacuum of control after the act of voting'' and the misplaced ``blind faith'' that the electorate had for these machines. He felt this threatened free and open elections where counts could be seem by all and were less easily manipulated.

Germany first used electronic voting in 1999 with differing acceptance by states. The last election using this technology was held in a local election for Brandenburg in September 2008. There are currently 39 of 299 election districts using this type of technology, in particular the states of Brandenburg, Hesse, Nordrhine-Westphalia, Rhineland-Pfalz and Saxony-Anhalt.

Experts pointed out that whilst software modification could be discovered afterwards, hardware modifications to the actual device were much more difficult to trace. This type of hardware manipulation was demonstrated by the Chaos Computer Club who modified the innards of an Nedap machine in approximately one minute.



This ruling has not completely banned e-voting. Andreas Voaykule, vice president for the Federal Constitutional Court clarified the ruling stating that ``the tenor of the decision could lead people to the conclusion that the court was hostile towards the technology and had misgivings about the challenges and opportunities of the digital age''. The use of other voting machines is possible and he also pointed out that the court has not banned the possibilities of internet voting. The presiding judge, Justice Rudolf Mellinghoff, did ask about the option of a vote verified trail using a printed ballot. This resulted in an agreement between Nedap and the parliament (Bundestag), in principle to offer this option as one mechanism with a parallel computer and paper count for elections. This may be one possibility for the re-introduction of modified Nedap machines in the future. Professor Wolfgang Lower remarked on the paper option, ``We must ask where is the advantage of the computer as part of the election in this case?'' referred to the idea of having separate and parallel paper and computer counts. (Editing Note: My German is not bad so I think this is a fairly accurate translation, however I do stand open to corrections).

iPhone Developer University Program

iPhone Developer University Program
Originally uploaded by bluepelican

The Computer Science department, thanks to Mikael's efforts has officially joined the iPhone Developer University program. It is a free program designed for third level education institutions who are introducing development courses for iPhone or iPod touch applications. It will allow up to 200 students, provide group management, and allow publication of the applications to the AppStore. This has also been done in collaboration with the Technology Transfer Office in UL so thanks are also due to Paul D for his help.

As they say, keep watching - there is more in the pipeline but you'll have to wait and see.

Random Paris Pictures

Mixed weather here in Paris but here are a couple of photos I took earlier, more to follow.

Backup and Failure

JournalSpace closed down a couple of days ago over a serious oversight with regard to their backup policy (read more here). The story seems to resolve around their use of a RAID array as their only backup. It failed completely and overwrote six years of bloggers data. They did keep backups of the code for the system but bizarrely overlooked the database content which was the images, comments, and posts that made up user's blogs. It prompted me to made a backup of my blog here, I doubt Google are going away today or tomorrow but who knows. I've always tried to keep update with my backups using SuperDuper (with an external HDD) and JungleDisk (it uses remote disk space on Amazon's S3). I remember reading "Disaster Recovery Planning" a few years ago and really seeing how serious you need to be about these things when in business, if after that storm, disgruntled employee, or other unforeseen event you have got be able to start from scratch. It doesn't cost alot to keep backups, reading "Backup and Recovery" shows the numerous open source options out there. I pay a little extra for simplicity and ease of use but the lesson from JournalSpace is at the very least, keep regular backups on different servers or media, preferably off-site and simulate an disaster with actual recovery phase.

Looking around at the end of the year

Well its a funny old Christmas this year with ups and downs in the economy. For a laugh or cynical look at the current situation, you could listen to Joseph O'Connor and his radio diary from yesterday's Drivetime on RTE Radio One, it goes nicely with Frank Nally's column in the IT today.

On brighter news, 50% service has been returned to the Shannon-Heathrow route so maybe there are some minor Christmas presents. We'll ignore the loss of revenue to the region of the last number of months and the fact its only a partial return of the service and take it, however I'm sure the local Government politicans will do better in the New Year as its full service return or they're not fighting the case in my book.
Hat tip to John and Nev for the SOS photos.
Hat tip to Nev for the photo.

On a more worrying note it looks increasingly more likely that Dell will be phased down in a fashion to minimise the company's repayment of Government grants. This isn't particularly great news and it looks like this abandoned factory in Raheen may soon have more company.
Hat tip to slinky2000 for the photo.

It looks like the private sector will feel the pain in the New Year and one wonders with the ERSI report will that happen in the public sector, it should (as I've said before) but the lack of authority shown by the current Government doesn't inspire much hope in this area. All I can hope is that maybe some of the sagely advice given by more optimistic people such as Elaine will be taken and perhaps we'll have a brighter 2009.