Adventures in Open Data
Local news startup Civil Beat today posted a piece on the names and salaries of state House and Senate members and staff. As before, they posted a watermarked PDF on Slideshare with the download feature disabled, and invited people to become “full members” to use their searchable database.
But if you’d like to download and play with the data for free, it’s been up for over a week at the Hawaii Open Data Project. Specifically, you can access the following documents:
2009-10 Senator and Senate Staff Salary
2009-2010 Representatives Salary (Legislator Monthly Salary)
2010 Permanent-OM-Support Staff Payroll
As you can see, the Senate provided the information in a single document, while the House separated Representatives from staff. The House staff data was provided as an Excel file, which Google Docs easily converted.Â I also requested job descriptions and duties, and both offices provided or referenced administrative manuals and rules that contained them. You can find everything on the wiki page.
Earlier this month, I reported that my request for information was denied by the Department of Human Resources Development. (If you haven’t been following the comments on that post, I encourage you to go back and read them.) I was trying to get the same data that Civil Beat had acquired and locked down, in the hopes of making it freely available.
At the time, I was discouraged. It seemed that DHRD had closed the door that it claims Civil Beat had erroneously walked through. So I feared that the other agencies I contacted — the University of Hawaii and the state Legislature — would similarly deny my requests.
I was pleasantly surprised to encounter a great deal of cooperation and responsiveness instead.
On the DHRD front, I filed an appeal with the state Office of Information Practices, which promptly acknowledged my inquiry. I’m now awaiting DHRD’s written response to the request from the OIP (due later this week).
Meanwhile, the UH Office of Human Resources, the office of Sen. Colleen Hanabusa, and the office of Rep. Calvin Say, all fulfilled my first and only information request — on time, completely (and electronically), and with no resistance nor fanfare.
So as in the case of the Legislature data above, if you want to see or search through a list of all UH employees and their salaries, you could certainly sign up for Civil Beat (which also offers its thoughtful analysis of the data). Or you could just download it for free, and draw your own conclusions:
UH Employees and Salaries
Whither Civil Beat?
The last few weeks have been interesting and educational, to say the least. I fully support Civil Beat’s efforts to develop a sustainable business model, and to realize the dream of making money by practicing good journalism. But I happen to believe that a media outlet should be selling its reporting, analysis and commentary, andÂ not the public information that it’s based on.
I concede that it may take considerable time, effort, and money to get the information, but I feel that the information ultimately belongs to everyone, and should be made freely available.Â In fact, doing so helps keeps journalists credible, as anyone can analyze the data themselves to confirm or refute the journalists’ conclusions. I think this is a big part of why most media outlets provide source documents and data alongside their coverage.
But, yes, the media landscape is changing, and I may simply be out of date or out of touch in my ideals and expectations.
Ragnar Carlson, editor of the Honolulu Weekly, posted a comment on Friday that noted that his newspaper was the first to get a copy of the Environmental Impact Statement for the city’s multi-billion-dollar rail project.Â It was a government document, obtained via a public information request, but the Honolulu Weekly did not share it, even when asked by other media outlets. (The final rail EIS isÂ now available online, courtesy the city’s HonoluluTransit.org site.)
Did I feel great about it? I felt great that we got it first, and less great about holding on to our copy. But what are you going to do? Newspapers are going out of business every day. At some point, you have to be willing to say â€œthe facts in this document are public information, the presence of it in the community is our work, thatâ€™s our business and so we are holding on to it.â€ If journalists continue to give away their work product, newsgathering organizations will go away. And everyone will be worse off for it.
Like many bloggers, I have the luxury of reporting on any topic I like, without having to earn a living doing so. Practicing journalism is a lot harder and more complicated for those that do it professionally.Â I understand why the Honolulu Weekly and Civil Beat did what they did. I simply disagree with them.
So what is theÂ Hawaii Open Data Project? Despite the formal-sounding, un-creative name, it’s basically a wiki page I started to keep track of these public information requests. Of course, I’m a newcomer and an amateur to this sort of thing, so I also added links to public information sources (courtesyÂ Ian Lind), and to other public information quests (likeÂ Doug White’s). I’m sure Larry Geller over at Disappeared News has unearthed some gems as well.
Since it’s a wiki, I’m hoping other local open data advocates feel inspired toÂ add to the collection.
So far, I’ve largely been following and duplicating Civil Beat’s requests (and if you know of any outstanding or new requests,Â please let me know). Going forward, my “data liberation ninja” friend and I are brainstormingÂ other data sets to pursue and publish. Like @kerryrm, he’s cited the “8 Principles of Open Government Data,” which will probably serve as the basis for future work.
I am partial to more statistical data myself — public health, crime, education, traffic, tourism, etc. Stuff with numbers that can be mashed up well or used to debunk false perceptions of every state thinking they rank last in some report. My goal is to normalize the data, present it in a searchable form, do some light analysis and hopefully provide a simpler way to mash up the data. My presentation skills are horrible, but hopefully, I can build a framework for someone that can.
The dream would be to have something like the set of Databases, Tables & Calculators put online by the federal Bureau of Labor Statistics. Imagine a site at data.hawaii.gov with downloadable data, online tools, and a full API to allow anyone to access and analyze government information.
Until that great day, we’ll build what we can on our own. If you’ve got ideas, or if you’d like to contribute, please comment below… or just head over to the wiki and start something.
Thanks for posting Ryan. Wow, those are some low salaries for the State Senators and Reps. They must make up for it in other ways.
I don’t get it. I’ve noticed you aren’t following the Star Advertiser’s Foia requests…. it’s because they don’t make them.
Seriously, CB is the best thing to happen to Honolulu –news wise — in the last 13 years (since I’ve been here).
I’m glad that they find the stories that no one else seems to be going after and follow through on them.
I think that both CB and Honolulu Weekly deserve credit for bringing more eyeballs to these docs and don’t hold it against them one bit.
I really appreciate your body of work — blog, byte marks, lost, and the rest. Just because we disagree. on something, doesn’t mean I’m not still a fan.
I challenge readers to submit more docs to the initiative too.
Russell, I’m not sure I understand what you’re saying. What don’t you get? I was an early fan of Civil Beat, and I still support its overarching, stated goals. I definitely give them credit for going after stories previously left untouched, and did so in my very first post on this project.
And to clarify (again), I think it’s fine that they charge for access to use their “searchable database,” a useful transformation of the documents in question. I work for a company that does the same thing — sells access to public data, on the basis that our collection, organization and presentation of that data is a viable commercial option to what average people can do on their own. And yes, it’s inevitable that governments will eventually make everything available for free, through the great work of companies like yours. The journalist in me looks forward to that day, even though the employee in me might dread it.
I love that Civil Beat is trying to do original, subscriber-supported journalism.
I love that they’re finding ways to give paying members new ways to interact with data.
I just wish they’d let that reporting and programming stand on its own, and pass on the documents they’ve obtained so others can do their own conversions and research… and affirm the integrity of their reporting. Instead, they say, “We used the public’s right to information to get this document, but we’re only sharing it if you pay.”
But as I said above, I might just be out of touch with the new realities of newsgathering. Civil Beat is certainly trying to forge a new path. All I can do is make my case, though certainly many may disagree.
Great post. I like your arguments and will follow with great interest the progress of the Hawaii Open Data Project.
Ryan, you and the ninja have done good work and a great service to the citizens of Hawaii. I have no doubt that one day Data.Hawaii.gov will exist, but until then it’s nice to know there’s the Hawaii Open Data Project.
It is truly a pity that @civilbeat have yet to see the light. I thought for sure they would release the PDFs. At the very least we have seen their true colors and that makes this whole endeavor worthwhile. Disappointing, as it may be.
If you’re looking for inspiration for the types of datasets out there that might be worth persuing, there’s http://nationaldatacatalog.com as well as http://www.datasf.org/ and many others.
Keep me in the loop as I’d love to help out in any way I can.
Ryan, I considered posting my records to Document Cloud. Try check it out. It is similar to the wiki you have set up, but has sparkly widgets and a nationwide scope.
Aloha, Ryan, finally made it over after multiple Ian Lind prompts and am glad that I did.
I want to hope that policy makers or, perhaps more accurately, policy executors, will pick up on your data.gov suggestion and run with it, although that may sound counterintuitive. Or even naive.
First, a bit of history. Tho’ many don’t remember, or don’t wish to give credit even if they do, the UIPA was passed during John Waihee’s tenure as governor and, under his administration, OIP had a robust presence in the executive branch. Katy Callahan (sp?) and her then-deputy Hugh Jones (now deputy AG and HSBA president) started the thing from scratch, adopting records retention/disclosure rules, training agencies in same, issuing thoughtful opinions on what is and is not public, and generally helping pry open the cover on government and make it a little more accountable, or at least a little less mysterious.
Of course, the irony is that the UIPA was also used to obtain documents that made the Waihee administration, or parts of it, look bad, rightly or wrongly, and that lesson was not lost on his successors. First under Cayetano and then under Lingle, OIP’s resources and authority were slowly eroded until it became what it is today — a vestigial organ.
Your efforts and those of like-minded persons/entities will keep this issue in the public eye, I hope, and encourage legislators to fund OIP appropriately and the new governor to reinvigorate the office and its vision.
Hey, I can still dream, can’t I?
It’s amazing how many hurdles are in place for public information.
Ryan, I don’t know where you find the time, but you’re doing a great thing for the community.
I can rationalize Carlson’s goal, but not with the way he is going about it. As far as CB, that they were able to obtain documents because they are available to the public but then turn around and want to sell it to said public is unreasonable. Best of luck in your endeavors, Ryan.
All this public data and people looking for ways to monetize on it, such a shame. We appreciate what you do, Ryan, not from a for-profit motive.
Ryan, how do I get edit privileges at the wiki? I set up an account and got the confirmation email, but still no joy.
(I ask because I have added another post to my collection.)
Hmm! You should be able to edit once you click the link in the email it sends, to verify that you signed up with a “real” email address. Did you click the link and still not get ‘edit’ privileges when you click ‘edit’ on the page?