The data tell the story

A first-person account of Open Govt Data Barcamp and Hackfest

Who knew data was sexy enough to persuade 150-odd people out of bed and into a bunker on a sunny Saturday morning for six solid hours of talk and no pay?

Not just any data, but specifically the prospect of making non-personal government data more freely available and easier for Jo Public to get her hands on.

I guess you could blame Obama.

“There’s a trend going around the world for open data,” says Mark Harris, a consultant who previously managed government web guidelines at the State Services Commission and who helped organise the past weekend’s Open Govt Data Barcamp and Hackfest in Wellington.

The idea got a boost in January, he says, when US president Barack Obama issued a Presidential Memorandum on transparency and openness in government.

“Openness will strengthen our democracy and promote efficiency and effectiveness in Government,” the memorandum reads. It promises “to disclose information rapidly in forms that the public can readily find and use.”

Not just timely, but rapidly. Not just disclosed, but in a form that the public can readily find and use. That’s the guts of open government data right there.

It’s all well and good posting PDFs and text files and a few spreadsheets in the depths of multiple websites, which various government agencies do now, but you often have to be a detective to find them and masochistically fond of manual processes if you want to extract just those bits of data that you’re interested in using elsewhere.

As barcamper Brenda Wallace (@br3nda) said on Twitter last week: “PDFs are where information goes to die.”

The process of opening government data has begun. You can see available datasets on the Open Data Catalogue and get a feel for State Services Commission open government efforts on the In Development blog.

But there’s room for improvement.

“Watching the UK model with and Rob McKinnon’s is a classic example of taking government data and reusing it in a much more useful format than parliamentary services can provide, in my view,” says Harris.

Another is, a New Zealand version of which is being built by Jonathan Hunt and came in for a lot of attention at Saturday’s barcamp and Sunday’s follow-up ‘get-things-built-day’ hackfest.

Fixmystreet allows you to report a problem in your neighbourhood—a pothole, maybe—without having to faff about finding which authority/department/helpline to approach. The website figures out the appropriate local authority and sends the report for you.

It’s a good example of how efficient projects can be when the government opens data and allows interested outsiders to develop it, says Rumi Shivaz, a consultant who helps large public sector organisations innovate and who, like me, was one of 19 sponsors of the barcamp.

“That simple application [fixmystreet] was developed in like 80 hours. It is something that would have cost hundreds of thousands of dollars to build if you took the traditional approach.

“You’d have to talk to all the councils up and down the country, and now try and get them in a room, and that’ll cost $100,000 easily with travel costs and all that. And there’ll be huge argument about what should be in scope and what should not be in scope. And another few months go by and a few more hundred thousand dollars go.

“The traditional way of doing stuff now is quite ineffective. There are better ways of tapping into some creative energy and to solve large-scale problems. “

A barcamp is an unconference which works on the principle that the best conversations at conferences happen in the hallways so you may as well just set up in a hallway and dispense with the rest of it. It’s driven by the participants, says consultant Mike Riversdale, who was chief crowd wrangler on the day.

“The schedule is made up of things you actually want to talk about rather than what the organisers think you want to talk about—and also everyone is charged with being a participant, not just a sitter and listener.”

Participants included geeks, people who work or once worked in government, consultants of various stripes and a few stray media folk such as myself.

The day started with us writing ideas for discussion on post-it notes which we slapped onto time slots pencilled on the wall. A bit of rearrangement and light negotiation followed, then it was talk time.

The ideas ranged from discussing why data should be open, what’s required to make it open and how outsiders can support government in making data available, through to specific applications data might be used for.

There were around 20 sessions which collectively served to enthuse and enlighten but also to highlight sizeable barriers to usable data.

In a session about health, for example, I started to get an idea of how hard it is to get information input in a sufficiently consistent and structured way to allow for easy extraction later.

We heard, for example, that there are up to 16,000 non-governmental organisations involved in health in one form or another in NZ. They vary hugely in size (think of the Red Cross through to a local sports group offering healthy activities), and in how robust their information reporting processes are.

Imagine that some of the information they—or anyone in any other field for that matter—collect is written up by hand. A busy data entry operator struggling to decipher the scrawl later may not have much incentive to linger over a couple of extra data fields, especially when he or she may not be aware of how useful those extra details might prove further down the data chain.

The value of clean data was also touched on in a media session. Jason Ryan of the SSC talked about efforts to use microformats in government releases to make them more useful to news organisations. In turn, I suggested news organisations should more consistently use metadata to make their own content more useful, and that more free data may help new forms of independent journalism evolve.

Freelance journalist and blogger Keith Ng raised an interesting question about the impact of more readily available data. Given the propensity for media and others to misinterpret data sometimes, what might happen if we provide more data for even more people to potentially muddle and misuse? Is government ready to respond to any increase in misinterpreted data in the public sphere?

Sam Farrow, formerly of NZPA and now ensconced in the IRD, later sounded another cautious note, briefly raising the spectre of unintended consequences. He made the point that our inherited system of government has been stable for a very, very long time. It’s not inconceivable that its arcane, slow and deliberate processes, no matter how frustrating they may be, may actually contribute somehow to its very stability.

Could making government data available more ‘rapidly and readily’ have unintended consequences? Who can know?

There were a couple of moments when I asked myself if we barcampers weren’t more motivated by a personal fascination with data than any clear mandate for a greater public good.

But I was so impressed that people were spending their own time working together on projects they think are important, that it made me think of what communities looked like in my grandparents’ day.

Family lore says that when the good people of Wardville in the Waikato decided they needed their own school, my grandparents donated some land, everyone pitched in with materials and labour, and the thing was built.

There may have been some lobbying of government for funds, and I’m not sure how a teacher was procured or paid. But the school was built because the community perceived a need and acted on it.

Open data geeks, it seems to me, are a community who have perceived a need and are acting on it.

They see clear economic and social value in open government data.

Shivaz, pointing to the transport sector as an example, says open data is an ingredient for entrepreneurship. “Here is an opportunity, by opening up transport related data in New Zealand, for people to develop some stuff here that can be marketed or consulted on a worldwide basis.”

“A very simple example is when NZTA opened up traffic camera feeds [in October 2008]. One developer in Auckland did a really simple iPhone application and now he’s using the same code base and developing a similar type of application for much larger states in the US. He’s making as much money as he is in his day job from this small part-time venture. This is just scratching the surface.”

Harris also sees innovation arising once resources are made available, “For the last 15 years government has talked about innovation but hasn’t really resourced it. Okay, there’s a small cost to putting the data out but it’s a smaller cost than actually turning that data into information. So put the data out and let that cost of changing it into information be borne by the people who are willing to pay it. Plus philosophically it’s our data, we paid for it through out taxes so why can’t we have access to it?”

Work started on several barcamp ideas during the hackfest, including fixmystreet and Sahana, an open source disaster management system.

Another project looked at how to pull out election night data at polling station level to allow it to be overlaid on a map—a potentially attractive application for news publishers. Another aims to document successful case studies of open data so people in government agencies can see the benefits for themselves.

The OpenNZ wiki is probably a good place to watch in coming months as these projects start taking shape, and there’ll no doubt be plenty of blog action. There’s also an Open Government Ninjas group if you feel like jumping in.