Civil Beat Puts Public Data Behind the Paywall
The headline proclaims, “Civil Beat Shares Hawaii State Employee Salaries“ — but there’s a catch: you have to pay to get full access to the information.
To be sure, there’s a reasonable and deep debate to be had over whether the names and salaries of government employees should be public in the first place. Â I was most intrigued, however, by how Civil Beat was looking to capitalize on the story.
They deserve a lot of credit for navigating the twisty halls of government to get at the information, but I’m annoyed at how hard they’re making it to benefit from their efforts.
Journalism is Hard Work
Civil Beat, the news site backed by renown Internet entrepreneur Pierre Omidyar, triumphantly announced their data acquisition this way:
“The biggest cost in state government is people. They’re what makes it tick. But how much we’re paying them isn’t so well known — even though the information is a matter of public record and should be available to any citizen who asks. At Civil Beat, we’re committed to truth and transparency.”
As a proud journalism grad who survived the public records class taught by the great Prof. Beverly Keever at UH, reading words like “public record” and “transparency” make my heart sing.
I also know how much work it takes to get information that we the people (not the press) should rightfully have access to. My college memories are peppered with musty basement archives, cranky clerks and half-broken photocopiers.
And for hopefully good reason, the state Department of Human Resources Development took a few months to provide the information to Civil Beat following its initial request in early May. A lot of the time was probably spent consulting lawyers and redacting entries, as the list of salaries doesn’t include several major branches of government, including the Department of Education and the University of Hawaii, nor any position protected by labor contracts (which are many).
Ultimately, though, the information was finally freed from the belly of our government and delivered to Civil Beat, apparently as a 300-plus page PDF sorted by department. The online news site was able to start a series of articles based on the data, including “Hawaii’s Lowest Paid State Workers” and “Hawaii’s Highest Paid State Employees.” Of course, you need to be a paying member to read the articles. But they also put the raw, original data online for people to review.
Well… kinda.
The Free Tease
If you wanted to see the PDF that the state compiled, Civil Beat posted it online via SlideShare. But while you could page through the document, Civil Beat explicitly set the document viewer to prevent people from downloading it.
Yesterday afternoon, the SlideShare document disappeared completely, but it was quickly replaced with a new version. You still couldn’t download it. The only difference was that Civil Beat moved its “Honolulu Civil Beat” watermark from the bottom of each page to the dead center of each page.
I asked Civil Beat about the change on Twitter, and they replied: “Moved watermark. Okay to promo CB after all time/effort it took to get the data, no?”
I certainly can’t begrudge them some “promo” for their work. But I also realized that the new watermark also stops people from trying to extract the data by printing out screenshots and running the pages through OCR.
Not that anyone would think to do that, of course.
Since a PDF is hardly an efficient way to work with tables of names and numbers, Civil Beat also put together a searchable database on their site that you can use to look up information by department, first name, last name, title, and salary range. Again, though, you have to be a paying member to get the full benefit of this undoubtedly solid coding. Visitors can only play with a “sample” database of only 50 out of the over 14,000 records provided by the state.
“Get a taste for free, pay for the full meal.” Certainly not uncommon in business, and definitely not unusual on the web. So why does Civil Beat’s approach bother me?
The Big Picture
In short, I don’t like that Civil Beat acquired the data under the auspices of “public data,” but is then turning around and selling access back to that same public.
Yes, I know that many businesses depend on this very model (including my employer). Like Civil Beat, it’s usually not so much the data that’s being sold, but rather its presentation and interpretation. And the response to criticism is obvious: you can always get the information yourself.
What makes Civil Beat different, in my view, is that it’s a journalistic enterprise. And not just any journalistic enterprise. They explicitly distinguish themselves from conventional media by focusing on civic affairs, aiming to create an informed public via an online “civic square.” As they said in announcing the salary information, it’s public information acquired in the name of transparency. Crippling access to the information seems to go against this vision.
Can’t a Startup Make a Buck?
Of course, Civil Beat is also a business, and needs to make money. But I think they should focus on doing so in the way they’ve been doing it from the beginning: selling their original, independent, investigative reporting.
Perhaps I’m deluding myself, but I think most news agencies would make any public data they receive freely available. Why? Because their value to the reader and consumer is by providing context and commentary. It’s great that you can get the Environmental Impact Statement for a new Mauna Kea telescope, or a list of bankruptcies for the last year, but what does it mean?
Indeed, Civil Beat says it takes “a holistic approach to news,” one that helps people “learn about and understand, debate and discover the important issues facing Hawaii.” If they do a good job with their coverage, people will pay for it.
Getting a list of 14,000-plus state employees and their salaries is only the beginning. It’s also information that belongs to everyone. I’d like Civil Beat to release the public data to the public, then kick ass and grow its membership with top-notch journalism.
Do It Yourself
Civil Beat did what any citizen could do: request information from the government. And while it’s not always a simple or quick process, I can tell you that it’s considerably easier than it was fifteen years ago.
Sometimes, the simplest plan works, so I called the Department of Human Resources Development and asked, “Could I get the same report you prepared for Civil Beat?”
“If you file the same request they did,” they replied.
Okay, not quite as simple, but they helpfully directed me to the Office of Information Practices, where I downloaded the form, “Request to Access a Government Record” [PDF]. It bears noting that you don’t have to explain who you are or why you want the information, just provide a way for them to contact you and deliver the information. I described the dataset Civil Beat received as best I could.
Yes, there are fees for processing record requests. The fees are explained as $2.50 for 15 minutes to search for a record, and $5.00 for 15 minutes to review and segregate a record. If Civil Beat had to pay $2.50 per record, they would’ve spent $35,000 for 14,000 records. Fortunately, you can request a waiver of fees in the public interest. The form explains how, and the criteria for a waiver:
- The requested record pertains to the operations or activities of an agency;
- The record is not readily available in the public domain; and
- The requester has the primary intention and the actual ability to widely disseminate information from the government record to the public at large.
Based on the above, I felt I qualified for a waiver, so I attached a request letter explaining as much. I noted that they had already compiled this information, so that it would hopefully not take much time or effort to fulfill my request. I also couldn’t help but note that unlike Civil Beat, I wholly intend to “widely disseminate information from the government record to the public at large,” rather than charging for it.
I should hear back from them in 10-20 days. And if I get the information, I’ll make it freely available.
Meanwhile, we can argue about whether names and salaries of state workers should be public information in the first place.
Hat tip to @kerryvm, who asked, “Are Capitalism and Transparency mutually exclusive?“
Excellent read Ryan and thanks for putting the time in to explain further the context of the tweets I was seeing yesterday. Looking forward to seeing an update to your request.
Great post and perspective. What isn’t clear is the data set provided to Civil Beat. If the team had to spend their own time and effort to clean up and format the data into a more friendly format then I don’t necessarily mind that it’s behind a paywall.
Civil Beat, by design, is catering only to a social elite that doesn’t mind paying $20 a month for online news that most sites offer free. I’ve told them that they would get more than four times more subscribers if they charged just $5 month and gave the first month free. Evidently, they’d rather not deal with the unwashed masses.
I look forward to you posting those records when (or if) you get them. I was frustrated when I couldn’t access them on Civil Beat, even though I’d bought a day pass for access.
Gee Why: Absolutely they should capitalize on their work to enhance the data. Their searchable database? Premium offering. But they say they got a PDF, and suggest that it’s the PDF they watermarked and posted to SlideShare. That’s the “public data” in this case. Why not let us have at it to do our own enhancements?
Greg: I think the pricing is extremely prohibitive and definitely creates a relatively small echo chamber, a pity given their otherwise laudable emphasis on conversation and feedback.
T. Rex Bean: Interesting. So even members couldn’t get the original PDF or get at the electronic version of the data?
Hmmm, Civil Beat would have to give the final say about the document they received but informally, I heard there was quite a bit of work done to get into a consumer-friendly format.
Great post, Ryan. Let’s hope Civil Beat does the right thing and simply releases their PDF sans watermark. They really have nothing to lose and everything to gain by showing respect for the community at large.
Thanks for the great post, Ryan. As I tweeted to you a moment ago, I don’t agree that the identities should be publically available, but I commend your work to challenge Civil Beat’s paywall on information in light of their stated purpose. You’re showing your chops as a true journalist.
I apologize in advance for derailing this discussion, but I have a comment on Civil Beat’s pricing. I am a full member of Civil Beat. At the second beatup, there was a pretty lengthy discussion about Civil Beat’s business model. Unfortunately there isn’t a stream available to watch, so I can only comment from memory, but, basically, I came away more convinced that the $20 per month subscription was reasonable and worth the hit on my monthly budget. For a startup that needs to be viable quickly without chasing after funding, takes no advertising, produces iterative, in-depth, high-quality, investigative reporting, and provides both an online and offline point of civic exchange, I think the $20 isn’t bad at all. Starting at a lower point increased the likelihood that they could not grow or even survive without raising rates–something they wanted to avoid. I doubt highly that they are pricing themselves to exclude “the unwashed masses” from the service. And I’m certainly not of the social elite. Twenty dollars here really represents a choice to curb $20 elsewhere. Granted, I can make that choice, but so can a lot of us. It wasn’t that long ago that we were paying nearly as much to get print delivered to our doors. Were we all part of the social elite back then? Watching Civil Beat emerge has been really instructive in helping me realize the value of really getting what you really pay for, especially with regard to news media. So much of the news business is imploding around us in large part because everyone expects all of it to be free or nearly free (ie., $5/month).
Thanks again for the work, Ryan. Can’t wait to see what happens.
Yikes, seems some folks ( http://twitter.com/jayrosen_nyu/status/21613016240 ) might be a bit confused. Nobody is arguing that Civil Beat doesn’t have the right to put the database behind their paywall. All we want is for @civilbeat to release the actual source PDFs they received via their FOIA request from the State of Hawaii. I don’t think that is too much to ask.
@chrisdaida: Thanks for your extensive comment. What Civil Beat is worth is certainly a relative thing. I’m among those who’ve always supported the concept, but don’t see the value at the current price point.
@kerryrm: I love that Jay Rosen tweeted the link, but yes, I have no problem with paying for the value Civil Beat adds with its reporting and analysis. Of course replies to Jay say, “Yes.” It’s charging for the original document that the state provided (and the public data within) that’s at issue.
I’d like to offer an example of how a daily newspaper did a similar list of government employees.
http://www.guampdn.com/guampublishing/special-sections/guamwatchdog/GovGuamStaff.html
The Pacific Daily News on Guam, a Gannett-owned publication and sister daily to the former Honolulu Advertiser, runs an annual list of the names, salaries and positions of every local government employee. Disclaimer: I have been a reporter and later assistant news editor of the Pacific Daily News.
The entire list is made available as part of the printed product, which costs 75 cents daily.
The list is also available in an online searchable database. You can ring up different departments, or ask for specific names of employees.
Their names are a matter of public record, and so are their salaries. The highest paid employee has historically been the island’s medical examiner, who rakes in an annual $247,944.48 including benefits. I distinctly remember him asking for a raise before lawmakers, joking with his dark humor that “as you can see senators, my patients are dying to see me.”
Please do not interpret this as a criticism from me regarding Civil Beat’s decision. I certainly respect their policies, and their business model is different than that of a tiny corporation-owned daily newspaper on a small island, with a government employee base that is considerably smaller than that of this state’s.
I am only offering up an example of a similar list being made available on names and salaries of government employees, as CB reporter Noelle Chun did regarding the Sacramento Bee’s list of public employees.
As usual, a very good, well thought out explanation of the situation. I appreciate the logic behind your reasoning. The idea of transparency is too often batted around and it is interesting to see to what degree news organizations like Civil Beat will embrace transparency. Public information gotten through FOIA should be made available but is a watermarked, non-downloadable version sufficient? Maybe, maybe not.
I suppose you could brut force the existing Slideshare version and extract the information into a spreadsheet. After much labor you would have a file useful for analysis. Perhaps that is exactly what Civil Beat wants us to go through. After enduring as much difficulty as they did to get to that point, we would appreciate the work they put into obtaining the information and associate value to it. If not, we are given the watermarked screenshots to indulge in.
I wonder if there are many civil beat articles that generate this much intelligent discussion
Ben: Just an fyi: Anyone is free to read the discussions, so you can evaluate the quality of the interaction yourself. I’m actually very pleased with the kind of exchange that goes on there. It’s one of the big reasons I bought in. For the most part, members really do their part to live out the charge to “be civil,” even with vigorous, heated disagreement. Compared to many of the news site discussion boards where people are for the most part anonymous, and the “d**kwad theory” is in full effect, the discussions on Civil Beat actually give me the sense that worthwhile, potentially fruitful interactions are taking place. Further, many of the commenters are knowledgeable experts of the matters on which they speak and take the conversations to the next level.
I hope you get a chance to check it out. Let everyone know what you find, especially if it’s other than what I’ve described here.
No one has commented on the most important aspect: Is this news? Or is it just voyeurism?
Why do we get a cheap thrill about prying into someone else’s private information?
What does it really tell us about the cost of state government? It doesn’t even include DOE, UH, or those protected by unions. We could get a better idea of actual government costs by reading and analyzing state budget documents that are readily available.
Why is it necessary to embarrass individual state workers by publishing something that most people prefer to keep private?
So much for this so-called new standard of investigative journalism. With this, Civil Beat is just using tabloid sensationalism to titillate the elite (and yes, until the pricing is affordable to a broader base, CB is only appealing to an elite).
Ryan, I hope you succeed with OIP. But Once you get the personal data you seek, I hope you rethink the part about sharing it with everyone.
Greg: I’m with you. Releasing identity is an unnecessary encroachment of privacy–even if the information is available via FOIA. I can see the value in releasing the positions and salary, but identity?
I’m not sure I’d call it tabloid sensationalism as CB is merely following many states where this sort of practice is just accepted as normal, but it is unnecessary. I dunno, maybe I’m just lacking that journalistic zeal for public records.
Ryan, pardon my asking you a pointed question: Can you tell us why YOU would see fit to identity?
Greg, the reason no one has commented on that aspect is because I noted up front with this post that I was focusing on the public data aspect, not the privacy aspect. But yes, “there’s a reasonable and deep debate to be had over whether the names and salaries of government employees should be public in the first place.”
I feel like we need a whole new space for that debate, and I also don’t feel entirely qualified to weigh in. Nonetheless, to answer Chris’ question, my position boils down to this:
Hawaii law doesn’t just ALLOW for all this information to be available, but REQUIRES it be available. In other words, the right of the public to know who is being paid, and how much, was explicitly carved out. Yes, this may be uncomfortable or troubling to public employees, and they probably weren’t fully informed about this provision when they accepted their government positions, but they are simply held to a higher standard of transparency and accountability than everyone else.
A local Big Island blogger of ill repute once bragged online about wasting time on the web at his state job, and you bet your sweet bippy that people took his boasts much more seriously than a similar statement by a private sector worker. He was paid by our tax money, and we don’t want our tax money wasted, no matter how insignificant this one person’s position was.
It’s reasonable to say that we should be informed as to how salaries are allocated within our government. Civil Beat did a good job adding the important context I talked about when they explained why the Department of Health has so many of the top salaries. With this information public, we have a chance to evaluate state employee pay, and try to effect change if we want to.
Do government employees get the short end of the stick because this information can be exposed? I suppose. But those jobs come with a lot of other benefits and protections that aren’t available in the private sector. It’s a value judgement, and nobody is forcing any state employee to stay a state employee if their values rail against disclosure of their name and salary.
Finally, as for why the names are important? I’m sorry to say that the names can often be the most important piece of information. Sure, we can say, “Janitor III” shouldn’t be earning six figures (just an example, that’s not happening here!), and a lot of useful information comes from only titles, departments, and pay ranges. But the names add the final, but most vital, piece of the transparency puzzle.
We could look at a $100,000 salary and title and say, “That’s high, but reasonable.” But that judgement would be very different if the name of the person with that title is the governor’s niece, or the wife of a prominent labor attorney. I shouldn’t go as far to say that this kind of abuse of government resources is more common in Hawaii than in other states. But some might.
Ryan, like the other commenters, I appreciate your efforts, insight, and perspective on this matter.
Regarding your reason for publishing government workers’ salaries: Shouldn’t the information also then include respective job descriptions?
Good question, Traci. Maybe the job descriptions aren’t among the things that Civil Beat requested, or maybe they’re not among the things classified as public (though I’d doubt it).
I’d say this data set is the first step. If you see a questionable job title or job holder, you should easily be able to research further and find the job description.
It is my opinion that if public money is being spent, the whats, wheres, whys, and whos must be public.
Ryan, thanks once again. I really appreciate your thorough, thoughtful engagement on this matter.
I’m aware of the legal requirement. Maybe I’m just naively wanting to find a middle ground between legally requiring that information to be available and rendering the vast majority of employees the same kind of dignity that anyone in the private sector would appreciate and expect. I certainly see the value of exposing the hypothetical example of the “governor’s niece,” but surely such a thing would be the exception. What disturbs me is essentially a presumption of guilt before innocence over the entire employee base.
I’m open to revising my position in this debate. It just seems that the argument for disclosure is being framed in extremes that are only helpful in, well, the extremes. Yet in every case, it has subtle but dehumanizing effects on your reasonably employed state worker. It’s to these that I think our law and the journalistic community need to adapt.
David Briscoe has a comment similar to that point on the Civil Beat discussion forum.
http://www.civilbeat.com/comments/3671/
Surely we as the public can and ought to find a more measured and delicate tool than the sledgehammer that is our current approach.
Well, that’s all I’ll say on the matter. Thank you for being patient with me “hogging the mic” on your blog.
aloha ryan, great article; good work.
Good luck, Ryan.
As you might know, I recently waited a year for a UIPA request to be filled. As for the fee waiver, the rules I was referred to only allowed the fee waiver for the first $60 in fees, so you could be in for quite a big tab. Did CB say how much they paid?
Oh, and since the State created a PDF for CB, I would argue that, for the purposes of calculating fees, you are essentially requesting one record, not thousands of records… heh.
By the way, I totally agree that the Slideshare/no-download tactic is lame. In contrast, I am (incrementally) scanning and putting all of the records I received online for anyone to download and I am also making freely available my “value added” spreadsheet cataloguing the records. Furthermore, if all goes well, I hope that the Capitol website will mirror the records, too.
So, Ryan, coming up on ten days, i.e. the amount of time the UIPA says they should take to respond to your request.
Please, give us an update if/when you’ve heard anything (or obtained the data).
I’ll absolutely post an update. I sent a request the same day to the state House and Senate (which reportedly delivered its data to Civil Beat on Aug. 25). Here’s hoping everything comes through.
Spoke at length with Marie Laderta, director of the state Human Resources Development Department, but the summary is that they’re denying my request because of onerous compilation requirements. Lots of other interesting statements as well. I’ll post an update soon.
Ryan,
Great job finding out how to do all of this and actually doing it! I submitted a request for information recently and I got blocked by county lawyers. I am seeking legal help to follow through. Check out my summary at https://docs.google.com/document/d/1FFhVHxO_jVVrBE9Q3J0OCGkgv15aoY5xL9mNWPSQHUc/edit?hl=en_US