I had a really amazing day at Rewired State yesterday. First of all, massive hat tip to James Darling, Richard Pope and Emma Mulqueeny for a genius idea brilliantly executed. Seeing the fantastic poster, with the phrase “Code a Better Country” emblazoned across a stream of binary, on my way out of The Guardian just made me grin. The quality of the work built yesterday there clearly shows that there is a huge pool of talent who could build some amazing things if everything opens up. 

Just imagine that, everything opens up. The Government builds APIs and changes tendering processes overnight. No more big systems integrators building crazy services by waterfall. The world of agile and the message driven loosely coupled world of the social and consumer web comes over and just builds great things. APIs are just a big box of possibilities where providing organisations can basically say, we don’t know what you can do with this - but it’s likely to be quite wonderful and unexpected. 

It happens all the time on the web and it’s where survival of the fittest comes in. Evolution is alive and well in the world of websites and moreover in the applications built on APIs and platforms. There are probably a wide variety of clients for Twitter which never see the light of day in terms of user numbers; why? because things like Twitterific, TweetDeck etc are great and people talk about their good experience and they rise to the top of the pile through social proof. I don’t think either of those apps really have a marketing budget, they probably never had a tendering process, they just started as a prototype and evolved. Something is bound to come along and be better too, and then people will use it, talk about it and it will grow and maybe if it’s good enough eclipse the current market leaders.

Here’s the cool thing though. Twitter doesn’t change, it’s still Twitter, it still has the same data, it’s just that survival of the fittest in software design builds better clients, and Twitter grows better because people do amazing things with it’s feeds which it may never have thought about doing.

It’s the same with the Facebook and MySpace platform, thousands of applications are built, but the number of them who have over a million users is very low. In all of these cases survival of the fittest and a selective pressure from users has created some amazing apps. 

Thanks to Ben Dodson for the pic

And so it could be with Government data. I’m sorry if I offended anyone from DirectGov yesterday in my quick 2 minute lightning presentation (*hate*that*format*). What I was trying to say was that if instead of building a website for the schools data they’d built an API then the community would have built amazing clients on that dataset. As far as I can tell the data is still either locked up in the spreadsheet or on their website and not available to all to build on. This is just a recipe for disaster and getting data out of step. 

Lots of the hacks yesterday involved screen scraping. This is such a scary last resort method in my opinion and just suggests that the data providers are being protectionist or more likely sadly unaware of possibilities (I actually think one problem might be the people who build the websites, fearing what would happen to their lucrative businesses if they built APIs and so they are not educating their clients). Tim O’Reilly as ever hit the nail on the head.

So this is why my hack was an API, built in just over 2 hours. Due to helping a fair bit of the beta test of The Guardian’s shiny new headquarters I didn’t have long to build something. I’d originally thought to build the API and a couple of clients as my project for Rewired State, then changed tack and wasn’t sure what to do, but a quick conversation over lunch with Robert Brook changed my mind back. 

The API uses AppEngine’s dataloading goodness to import the CSV data. At the moment only 10% of the data is there (I started building at 3pm, finished at 5pm and started loading data and wanted to be sure I could demo, so not all the data is there). However what is there is 3 endpoints.

An endpoint for individual items of data, in this case schools, but let’s abstract to items as the basic principle could be used for other APIs built on a single table of data.

An endpoint for search, this isn’t a geo-coded search, it’s just a full text search of the data. This however can do some great things out of the box using AppEngine’s searchable model, such as partial postcode searches.

There is also a browse endpoint for items. That query was for all the schools whose town was listed as Sheffield. This one is for all the schools whose Local Education Authority is listed as Sheffield, and this one, my favourite lists the schools whose headteacher’s title is Mr.

It needs performance tuning/caching for sure, and it needs a method for generating API keys for different levels of users (a band for people who can just GET, one for GET/POST/PUT) but those shouldn’t be too hard and you can’t expect miracles out of 2 hours. I had a great quick talk with Paul Clarke ad one of my reasons for writing up was so we can carry on the conversation with Paul and Robert about how we could use more of this sort of lightweight tech within government. 

Big question is though, what could you do with it… well again this is where the community comes in. You only have to see services like Tom Taylor’s “Snowmen near me”, UKSNOW and all the BNP map hacks which spring up within about 5 hours of something interesting happening to know that you’ll never know what people will build until they build it.. Here are just 4 ideas 

• Well the “is my school closed because of snow, earthquake, invasion of giant lizards” thing is clearly there
• Gavin Bell mentioned that it would be great to include catchment area data
• You could do a crowdsourcing experiment to see which schools/LEAs/towns had schools where the parents got their first choice school
• From the looks of many hacks yesterday I’m guessing someone might do some cool things with maps and heatmaps
• It could also power the Ofsted site and LEA sites where address details of schools are needed. If you gave a school a system where it could update its own details it could make the data always up to date.

Screen scrapes and spreadsheets aren’t data sources, even the best screen scraped source risks being broken and out of date and even if you put your spreadsheets under RSS then there’s a chance of the data on an API created on them being out of step. Also no one wants to reimport 22,000 lines from a spreadsheet and you’re just encouraging the wrong sort of keying strategy to come out of it as people would more likely than not use their database ids as internal keys rather than using things like the schools URN. Obviously people who love spreadsheets still can play nice with this as we know from the lovely demo from Rob McKinnon yesterday that spreadsheets play very nice with RESTful APIs. 

Hello State. Can we haz Rewire plz. KTHXBAI.