Data Request Denied, Now Plan B
I had a nice conversation this afternoon with Marie Laderta, director of the state Department of Human Resources Development, in response to my request to secure the same data her office prepared for Civil Beat (and which Civil Beat hid behind its paywall). She was professional, articulate, and demonstrated that she understood the background and intent of my request, and even the basis of my objections to Civil Beat’s actions.
But ultimately, she said she was denying my request for the information.
She said a number of interesting things, and I disagree with a good many of them. Unfortunately, I’m much more comfortable in the realm of geekery than I am in the realm of public policy. While I’m not giving up on my request, I’m happy to report that the data in question has been freed, anyway.
What’s All This, Then?
If you’re just tuning in, I recently cheered the work Civil Beat had done to get a list of almost all state employees, their job titles, and their salaries (or salary ranges). At the same time, however, I jeered the counter-intuitive work Civil Beat did to limit access to it: watermarking and adding DRM to the document, and stashing the extracted data behind its paywall. This is a “commitment to transparency“?
When Civil Beat made it clear, in part through chipper responses via Twitter, that they weren’t going to release an unencumbered copy of the digital file they got from the state — a document that, I believe, is itself public information — I decided to request the data myself.
I filed a “Request to Access a Government Record” request, along with a waiver of fees in the public interest, with the department. (You can find the forms and some guidance at the state Office of Information Practices website.) I crafted the request to mirror the data requested by Civil Beat.
But I also acknowledged that my request could require time and effort to fulfill. So I said that I would be satisified with a copy of the output of a materially similar request recently fulfilled by their office. In other words, I wanted the same data the OHRD gave to Civil Beat, so I’ll settle for the same document they compiled for the news site.
I noted, parenthetically, that I also wouldn’t charge anyone to access it.
Then, I waited. The department, like all agencies, had ten days to respond. Sure enough, at exactly 5 p.m. yesterday, Laderta called me. Unfortunately, I was on the air at Hawaii Public Radio, so she could only leave a voicemail. I called back this morning, leaving a message for her, and we finally connected late this afternoon.
Not In So Many Words
We spoke for about 10 minutes. She provided thoughtful explanations, and answered several questions, including questions I asked more than once to make sure I understood the substance of her response to my information request. Insofar as Civil Beat was concerned, she said:
- The Office of Human Resources Development did not retain a copy of the file provided to Civil Beat. This is common practice. I asked if she could check the “Sent” folder of her email. She said that if they had it, they would have provided it.
- The Civil Beat request took several weeks to fulfill, and involved discussions to refine its scope to exclude information that OHRD could not provide. She noted that this process consumed significant resources, therefore costing taxpayer money. She said she knows they could have assessed a fee, but chose not to do so.
- She told me that I could go to the Civil Beat website to “get the information for free.” I explained that the display of the PDF document was restricted such that the public could only “look at it,” and not download it, and that you need to pay to manipulate the data in any useful way. She suggested that the Civil Beat setup should still be sufficient, but she also conceded that it was not what she expected.
Since she could not produce the document provided to Civil Beat, and since I did not feel I had sufficient access to that information anyway, I said that the crux of my request remained: to have the information compiled and provided to me directly. Laderta’s response?
- My request for the information is being denied because of the onerous “compilation” component. Basically, the amount of work it would take to put together all the data was excessive. Though she did not cite chapter and verse, she said that OHRD had been advised that “compilation” was one of the grounds on which requests could be denied. She again noted the time and effort consumed by the Civil Beat request, and mentioned constraints including RIFs, furloughs, manpower and budget.
- Most interestingly, she also said that the Civil Beat request should not have been fulfilled in the first place, and that if Civil Beat made the same request today, it would be denied on the same basis used to deny my request.
I told her that I found many of the points she raised to be troubling, but thanked her for her time. She was similarly gracious as we ended our call. I did tell her that she’d probably hear from me again.
Next Steps and Sidesteps
At the very least, I’m going to ask Laderta for a written response to my request, hopefully addressing the above points. From there, I’ll likely ask the Office of Information Practices to determine whether OHRD can indeed say that public information is only public if it’s not too much work to make it public.
But I know both OHRD and OIP are understaffed, overworked, and have better things to do that deal with a whiny blogger. Doug White’s tale of an information request that dragged out over a year is not encouraging.
Fortunately, as I had hoped when I first raised the issue of putting public data behind a paywall, there was a technical solution to the ideological problem.
For the past few days, I’ve been able to collaborate with a fellow open data advocate and data liberation ninja. The product of his considerable talents? A public, freely downloadable, usable spreadsheet of the data Civil Beat tried to lock down.
Instead of posting the PDF document provided by the state (or the watermarked, ostensibly copyrighted PDF that Civil Beat created), we’ve acquired what mattered most: the information contained within. Information provided by the state, and information that belongs to all of us.
Well… most of the information. Here it is:
That’s a Google Docs spreadsheet. You can browse or search it online, or download the data to your computer and play with it. Get it as text, as a comma-separated-values (CSV) file, as a Microsoft Excel file… you can even get it as a largely useless PDF, albeit without any pesky watermarks.
But you’ll probably notice that there are no names included. This is because the aforementioned data liberation ninja is also a privacy advocate. As I’ve engaged in debates all week over whether the names are essential data or merely needless provocation, I think it’s a fair compromise. I’m hoping we might be able to provide the entire dataset in the near future.
In the mean time, as a bonus, here’s an additional batch of information you might find useful:
As you may recall, the original dataset from the state did not include information from the University of Hawaii. Civil Beat separately requested salary information from UH, and got it, posting more stories (and doing the same thing with the data).
Fortunately, UH is required by law to present several annual reports to the Legislature, and this includes an “Annual Report on Salaries” (HRS 304A-1004).
It’s posted online as a PDF, but PDFs are hardly user friendly, and UH inexplicably locks its reports down with DRM. So again, the Google Docs spreadsheet linked above is wide open so that journalists, bloggers, or curious citizens can do whatever they want with the data.
The Legislative salary report also does not include names, while Civil Beat’s report does. Civil Beat’s information is also more current. But this is a good start.
I still have salary information requests pending with the Legislature (state House and Senate) and the University of Hawaii, and again, whatever I obtain, I will make freely available. After today’s rejection from the OHRD, however, I’m doubtful that I’ll get much.
In the mean time, Civil Beat continues to acquire massive volumes of public data — efforts that have apparently prompted at least one state office to turn down subsequent requests — but continues to push people into paying for access to that data. I’m losing faith that the news site is the principled, idealistic journalistic enterprise it claimed to be.
Though I guess the paywall should’ve been my first clue.
Ryan, you’ve done a great public service by obtaining this information. Keep up the good work.
Ryan, impressive write up!
Ryan, you’ve uncovered him and that poor excuse of a website for what they really are: scum. May his soul rot in hell.
“The Office of Human Resources Development did not retain a copy of the file provided to Civil Beat.”
Wait, what?
“The Office of Human Resources Development did not retain a copy of the file provided to Civil Beat. ”
Wait, huuuh?
The Office of Human Resources Development did not retain a copy of the file provided to Civil Beat.
Wait. So this document, which took several weeks to compile, at a substantial cost to the state, was not retained in any way, shape, or form by the State.
So, if internet access had suddenly gone down at the moment the document was sent to Civil Beat, and the document was lost, DHRD would have had to compile it all again … from scratch?
And if they had turned it over the CB, and Cb then argued that it was incomplete, and sued them, DHRD would not retain a copy to prove to a court they had fully complied with the law?
Ridiculous.
Laderta’s compilation “exemption” sounds completely suspect, if it is not outright bogus. I would definitely appeal that denial. (If for no other reason, then appeal it because if it were allowed to stand agencies could use it as a precedent for future denials.)
If it takes time and is/was “too onerous,” well, tough. That is why the UIPA authorizes holders of the records to charge requestors fees for their work to search for, segregate, and reproduce the records.
By the way, when I parenthetically mentioned setting a precedent above, the precedent-setting concept works both ways. DHRD has already filled the same request once. That response set a precedent. DHRD may not pick and choose who is allowed the records. Equal protection, and all that Constitutional Law stuff…
Good work, so far! I hope you have the endurance to press on.
Ryan,
Fascinating read and great reporting once again. Good context, detailed, and thoughtful…why I love reading your blog. I too find it suspect that your request was denied. Love your follow up as to whether it might in in the “Sent Mail” box. I bet it is.
Besides that, the whole process seems like a problem as Doug points out in his comment. And LavaSusan makes a good point too.
I’m not going to act like I know the laws of public information, etc…I don’t…but what I’m reading between the lines is there are problems with public policy oversight and efficiency. No surprise. But if we are indeed living in a new normal of needing to do more with less, I sure hope mid and top level management are helping their staff realize what this means. Even if it’s just saving a document so you don’t have to recompile it again.
Mahalo again Ryan. Keep up the great work.
Nathan
Ryan, amazing work! It’s a shame @civilbeat didn’t step up and do the right thing, but I guess that was to be expected. They are first and foremost a business. The public interest clearly comes a distant second.
I think the “excuse” you were given is rather bunk. I agree that you should appeal. Something is definitely not kosher at OHRD. Also, I’m going to pass your post around to some data journalists to get their feedback/advice.
On a somewhat related note, I like the sound of OpenData808.org. Someone should start an open data catalog for Hawaii data.
Hi Ryan,
This is interesting stuff. Thank you for sharing your experiences. As a contributing reporter for Civil Beat and HPR, I’m particularly interested in what some of your commenters are saying.
For me, the question is: Is the value in the kind of work Civil Beat does (multiple attempts to secure documents, weeks of narrowing the scope of requests, sorting data, making a database searchable, and putting it into context for readers) valuable enough that the public is willing to pay for that effort? There are people who are working full-time on obtaining and distilling this kind of information. Does the public believe they ought to make a living to do so? Public Radio, for example, is supported by the listeners who answer “yes.”
I love the idea of free journalism. It’s sad to see the era of free news ushered in by the Internet waning. But I also love being a reporter, and I believe passionately in the importance of the hard work journalists do, and their right to be paid for it.
Curious to read more thoughts from others…
Thanks for your comments!
@kerryrm: Data liberation ninja and I definitely envision a public data repository of some kind, at least until the state puts up ‘data.hawaii.gov’ with APIs. We can dream!
Doug: Your own experience with these requests was both instructive and somewhat daunting. I hope to persevere, but I doubt I have the dedication you did.
Adrienne: Thanks for your comment. As I articulated in my original post, I’m all for paying for good journalism. It’s paying for the source data, public data, that I object to. If Civil Beat wants to capitalize on the results of its work to transform the PDF data into a searchable database, fine. But “branding” the PDF itself and locking it down goes against the higher principles Civil Beat espouses.
I contend that all legitimate news agencies would freely disseminate documents obtained from the government through a public information request, no matter how much work it was to get them. Their kuleana is reporting, analysis, and commentary based on those documents, but the source data belongs to all of us.
Hi Ryan,
Thanks for replying so quickly! To clarify, the Civil Beat data obtained via numerous requests is available freely to the public.
You don’t need a membership to see the raw source data: http://www.slideshare.net/civilbeat/hawaii-state-employee-salaries-as-of-july-2-2010.
I can understand how the Civil Beat stamp might bug people who want to re-post the information without attribution, but it doesn’t obscure any of the information itself.
Adrienne, I’m fully aware I can “see” the document. As my previous post outlined in detail, and as I commented above, that’s hardly making the file available. Civil Beat says it got a PDF from the state. Why not post that PDF publicly and unencumbered? Why slap a watermark on it and make it view-only? In fact, if the document was produced by the state, what attribution beyond OHRD is it reasonable for Civil Beat to expect?
Why do you need more then just “view only”?
I absolutely LOVE how civilbeaters and others claim an “embedded, watermarked flash-based slideshow” of the raw data is available. If OHRD had provided them the information they requested in THAT format, I’m pretty certain @civilbeat would have thrown a fit. And yet, they think it is perfectly acceptable to call what they’ve posted as “public” and “accessible.” Ugh. They clearly don’t get it over there and it is becoming painfully obvious. Such a pity. I had such high expectations for their model as the future of journalism. Instead, it appears that they are just a repackaging of the same practices and mores of a dying industry.
Fair question. Why? Because access to the file would allow the public to extract, manipulate, and derive useful information from it — basically, to do what Civil Beat did. Again, the original file is a government document, not CB’s. It was fortunately possible to get at the data anyway, but no thanks to CB.
p.s. Here are the 8 principles of Open (gov) Data. These are specifically for GOV data, but there’s no reason Journalists and others that deal in data can’t also embrace them.
http://www.opengovdata.org/home/8principles
So you want to manipulate a Government file from your standpoint and opinions?
I think you may be misinterpreting what I mean by “manipulate.” I don’t mean “distort,” but rather, “do work with.” Sort it, make calculations, plug it into formulas to derive trends and outliers, and so on. See the excellent “Open Data Principles” link posted by @kerryrm above, and the “Machine Processable” item.
Just to be clear: I am a Civil Beat contributor, but I had no role in the reporting or presentation of this particular story. Only the editors can speak to the publishing decisions. I thought the concern was with regard to access to the information itself. That’s what I see as most important. The data is out there, freely available.
As for my comment about attribution, it’s the industry standard to give credit to news organizations that exclusively obtain data — even if that data is mandated to be public. That’s not just a courtesy but a way to let the reader/listener/viewer know where the info originated, and who may have had a hand in its compilation/presentation among the way. If another organization gets the data from the state themselves, then publishes it, of course that’s a different story.
As for frustrations with media, keep the comments coming! If it’s the duty of the press to hold the government accountable, the people must hold the press accountable. It makes me happy to see so many people getting fired up about local journalism. Thanks again for facilitating the conversation.
P.S. I am writing from a first generation iPhone (I know!), so please excuse any typos.
“DRM” in a pdf file is false security anyway. There are several ways an average person can gain access to the data. UH only thinks they are protecting their data.
I haven’t looked at the Civil Beat salary data, but watermarks are also sometimes easily removable. The removal could be problematic if the material were indeed copyrightable, which I think one might dispute. In any case, for personal use, if a watermark bugs you, it can often be removed.
When the “DC Madam” released her redacted list of client phone numbers, I and others found that the redactions were also easily removed.
This is perhaps a side issue, but since you’re a technical kind of guy, I mention this.
@Ryan Open source is good, it’s even better when one doesn’t have any biased opinions or intention on the matter, like some media and people with malicious intent.
Excellent work, Mr. Ozawa, though I agree with you that your friend should not have redacted the names from a public document. I hope you can soon correct that.
As for compensating reporters for work they’ve done — well, yes, they should get paid, but they should get paid for adding value to pubic information, not limiting access to public documents. And, by adding a watermark, keeping the file in pdf format and other onereous additions, that’s exactly what Civil Beat is doing here. Sadly, public information is often so jealously guarded by the state and city that obtaining it can be regarded as something of a coup.
Finally, the state’s rejection of Mr. Ozawa’s request for THE SAME INFORMATION is outrageous and the reasons it gives for doing so are laughably unbelievable.
Adrienne: I do appreciate your comments and participation, and recognize that you do not speak for Civil Beat as a whole. It doesn’t appear Civil Beat’s engagement reaches much further beyond its own system, Facebook, and Twitter, so any voice is better than no voice at all. As far as your summation, mine would be that the statement that “the data is out there, freely available” is false.
Larry: Yes, PDF DRM is easily defeated, and was how our data liberation ninja got at the data. But you raise a great point in whether it’s even appropriate for Civil Beat to slap a watermark and effectively claim credit for and ownership of a state-generated document.
@Adrienne “The data is out there, freely available”? The first thing I understood from Ryan’s post is that this is exactly NOT the case.
Hey Ryan, another Civil Beater here. Like Adrienne, I did not work on the project in question, and I am not able to speak for our organization. My comments here are entirely my own.
First, I’d like to congratulate you on your efforts to gather government documents in the public interest. If more citizens did what you’re doing, I think we’d all be in a better situation. I hope you’ll continue to press forward and highlight the sometimes bizarre workings of our government (no retained copy??). I can tell you, as a veteran of a few of these open records fights, that perseverance is required. I hope that, like Civil Beat, you won’t keep the government employee names private, because I think those are critical. I’m rooting for you.
That said, there are two separate issues that are being discussed here regarding Civil Beat’s work. And I’m sorry to say that I think you’ve unfairly conflated them.
In your opening graf, you say Civil Beat “hid (the public data) behind its paywall.” In your closing, you write that by leaving on a watermark and preventing downloading, Civil Beat “continues to push people into paying for access to that data.” Both statements are factually incorrect.
The important distinction that needs to be made, that you have repeatedly failed to make, is that the PDF file is not behind a paywall. It’s available for all to see. There is no public data behind the paywall. Everything that’s in the searchable database (available to members) can be found in the PDF (available to all). To imply that paying members can download the PDF or view a watermark-free version while non-members cannot is false.
What Civil Beat does provide to its members are valuable tools to easily manipulate (search, sort, analyze, not distort) the public data that everyone has access to and can already manipulate manually for themselves. Is that business model different from publications that charge readers to view reporting on and analysis of publicly available data or announcements? Civil Beat is providing a service — organizing data in a format that’s easy to understand.
In the comments section, you characterize the situation as Civil Beat “capitaliz(ing) on the results of its work to transform the PDF data into a searchable database.” This is far more accurate than your earlier statements — though I think the term “capitalize” carries a negative connotation here. And you say you’re “fine” with that approach. Glad to hear it.
The last thing I’d like to leave you with is the idea that this data — the raw public data — would not currently be available in any format unless my Civil Beat colleagues had pushed forward, refined their request, and put considerable time and effort into merely obtaining it. I think you and your readers are starting to see how arduous that trek really is.
Civil Beat, now four months old, has made that raw, public data available to the public (for free), something neither the government nor any Hawaii publication — print or Internet — has ever done. If that doesn’t speak to a commitment to being a “principled, idealistic journalistic enterprise,” I’m not sure what would.
Looking forward to hearing more thoughts from you and readers.
In the interest of fairness, I have to respond: The data — that is, the names, positions and salaries that make up the document — is absolutely freely available. It’s the format in which it’s provided that’s at issue, right?
The really great news, I think, is that there are a lot more freedom of information requests already in the works at Civil Beat, and many more opportunities for these kinds of discussions as some really fascinating information emerges.
Adrienne and Michael, I appreciate the extensive comments, but as I think I’ve noted, we simply differ on what defines “freely available.” A PDF posted to Slideshare with the download feature explicitly disabled does not qualify. In my opinion. Michael even refers to the restricted display of that PDF as equivalent to having access to the “raw public data,” which it is not. In my opinion.
The PDF document the state provided — a digital file that can be infinitely copied and distributed without causing any inconvenience to Civil Beat — is the “raw public data.”
Civil Beat stopped just short of doing the right thing. And in that gap, we find the added watermark and the disabled downloads, deliberate acts that erode claims of being “principled” or “idealistic.”
Meanwhile, Civil Beat says, “Good on you for learning how much work this is!” Yes, I’m replicating requests Civil Beat has made because Civil Beat won’t release the file the state invested considerable time and effort into providing. A simple act that would take nothing more than a click of a button.
The net result? Government agencies duplicating work, spending more of the tax dollars that Civil Beat supposedly wants to help Hawaii spend more wisely.
Wow. Great work, Ryan. Persistent but got what you wanted (sort of). That public information is only made public if it’s not too much trouble to compile is in it of itself troubling. Please post updates as your petition to the Office of Information Practices.
Thanks Ryan for approving my post and for your quick response. While I respect your passion and appreciate your goals, I still think you’re incorrectly conflating terms.
The “data” is comprised of the names, positions and salaries of 14,000+ state workers. That raw public data is available to the public for no charge. I welcome you to point me and your readers to any single “datum” (i.e. Laura Thielen’s annual salary) that is behind the paywall. In absence of that — in other words, if each and every datum Civil Beat received from the government is available to the public — then the only factually accurate conclusion is that all of the raw public “data” (plural of datum) is available to the public.
What is not made available to the public is the “document,” the “file,” the “format,” the “compilation,” that said “data” can be found in. A “document” is itself not “data,” and I think those tech-savvy readers would agree with me on that point. You may see this as splitting hairs, but I see this as accurately defining my terms so as not to create any confusion or mislead anybody. Getting it right is important.
The “document” Civil Beat received from the government is not currently available for download — to paying members or non-members. It’s not as if Civil Beat is charging for access to it, and to imply that would be, again, incorrect. Civil Beat members are provided valuable tools to manipulate the exact same raw public data that you and everybody else can already manipulate themselves. For those readers here who remain confused, just go to http://www.civilbeat.com and verify my claims for yourself.
I understand your frustration with the government for failing to retain a copy of already-compiled records. I’ve been where you are. And I’m rooting for you to succeed — honest.
But lashing out at Civil Beat for declining to provide you a similar document obtained from the government previously pursuant to its own work and own request — and blaming Civil Beat for government duplication and wasted taxpayer dollars — is pretty irrational.
It is splitting hairs, and fortunately, I’m splitting those same hairs. I appreciate the effort you’ve taken to point out the distinction.
Ultimately, yes, it is the “document,” the “file,” the “format,” the “compilation” that I am seeking. And I am frustrated that Civil Beat is not making that document available so that the public can work with it the same way Civil Beat did.
It is my contention, and I state this as belief and not fact, that the “document” is more the property of the state and therefore the public, and not of Civil Beat. And even if it was, it wouldn’t harm Civil Beat to release it, since what we need from Civil Beat is the analysis and commentary based on it.
I’m sorry you feel I am lashing out. I was hoping I was making a pretty rational case, albeit one with which anyone can reasonably disagree.
I’ve been interested in, and supportive of the declared mission of, Civil Beat for longer than it’s been around. Since Peer News was first setting up shop in Kaimuki. I’m a huge fan of many of the people involved.
But yes, the decision to withhold the “document,” the “file,” the “format,” the “compilation” is what I find frustrating. And because of that decision, I am lamenting that I am causing grief to Ms. Laderta in order to secure my own.
Relax, Ryan, I have read somewhere that Ms. Laderta gets $103,512/yr to compensate her for any “grief” you may inflict. :)
…and rest assured it won’t be Laderta herself grieving about doing the work if your request is ever filled. Heh.
If everyone at civilbeat really believes that a watermarked, embedded, flash-based version of what was once a PDF (itself a proprietary format) constitutes “raw,” “open,” and “available” data they are embarrassingly out of touch. (perhaps Ms. Laderta will embed the results of the next civilbeat FOIA request at slideshare for their viewing pleasure.)
Michael (or anyone at CivilBeat, for that matter), can you PLEASE explain to us why it is that you refuse to make the PDF available for download?
What exactly is the value of the PDF to civil beat at this point? Clearly you are making a distinction between the original “source” PDF and the publicly available version of the data by 1.) preventing download 2.) watermarking the document 3.) embedding it on slideshare as a flash file.
The information has already been extracted from the PDF and placed into a database by @civilbeat. What value does the source document hold now? Would it not be the right thing to do to release it and obviate this back and forth, not to mention the wasted time and effort on all sides?
As far as I’m concerned, @civilbeat is in the wrong on this issue and will remain on the dark side of transparency until a “non-watermarked & downloadable” version of the source document is made available to the public whose tax dollars paid for its creation.
Wow, pretty passionate debate. Ironically, the questioning of CB’s integrity is being conducted in a very civil manner – something I think the CB team would be proud of.
Ryan’s concern, whether popular or not, is commendable and has brought to light issues that would otherwise be overlooked. Just the fact that people are conversing over values, responsibility of organizations/media/public servants/bloggers, rights to electronic ownership and work ethics shows me that a collective consciousness is alive and kicking. Regardless of personal opinion, I think everyone would agree that -having’ this discussion is valuable in itself.
Nah, the discussion is meaningless unless/until CB gives it up. Heh.
Ryan,
Thanks for challenging both CB and the State on this. In the past, i spent months trying to get access to government documents which, IMO, should have been released in a timely fashion, but were withheld until they lost much of their time-sensitive value to me.
As I understand intellectual property law, CB has no legal claim of ownership of the document. even if they posed the original query, and endured the frustrations of dealing with the state agencies to get the info, none of that gives them ownership of what is, under the very law they used to request it, a “public document.”
They may add value for those who pay to use their interactive search features, but that is a separate point. You are not demanding to use those search features. You just want access to the original document.
CB, having endured some expense and staff frustration to get the document and seeing some economic advantage in having exclusive control of it, tell you, in essence, “Go through the hassle we did and get your own!”
You think that is not “idealistic” enough. Well, I never thought CB was particularly idealistic. I think they are searching for a new model for economically sustainable journalism, since the old one is obviously collapsing all around us.
So at best what we can hope for is “Idealism minus pragmatic self-interest.” Which strikes me as reasonable. If they start “getting rich” with the new model, well that might suggest the balance of idealism and self-interest is outa balance. But let’s have that discussion IF we approach that situation.
I have a hard time believing the OHRD does not have a copy of the document they provided to CB. But maybe it is true, because it is basically an admission they are incredibly stupid and who wants to admit that. I can imagine (if I strain) a policy for not retaining emails past a certain time period. Or mistakenly believing the information they provide to the requestor is (perversely enough) privileged. That is dumb, but state agencies seem to have settled upon a lot of DUMB standard operating policies. Hasn’t it been the source of frustration when you have to deal with a bureaucracy filled with seemingly intelligent people adhering to absurd policies which might make sense at each small logical step, but lead to absurd behavior. And like all good bureaucrats, these intelligent people are too meek and cowed to just do the commonsense thing to solve the problem?
But IF they truly did not retain a copy, i can understand their refusal to generate a new one. From their point of view, they have very limited staff with competing demands on their time. Yes, the info they provided to CB is not COMPLETELY available to the public, but the difference between the ability of the public to manipulate the data as you (quite reasonably expect) and what the public can access for free through CB’s website probably does not justify the expenditure of time and effort, especially if you are claiming the right to get the info for free. Heck, if I were running the necessary “triage” calculations, I would also deny your request.
But you and your ninja buddy hit upon the right solution. CB has absolutely no proprietary claim on the original document or data. If you folks can use technical means to liberate the data, CB has no moral or legal right to object. And I respect ninja’s ethical qualms about releasing the names. I am uncertain if they should be released or not. But I am glad he felt it important to make his own determination.
So thanks for conducting what I think was a noble AND interesting experiment. And thanks for releasing the information to the public. Hopefully, watchdog groups (and eccentric gadflies) will reformat the data and find useful patterns, all for the public good.
As it happens, I learned (painfully) the State email retention policy during my UIPA saga. Here are the relevant memoranda and rules:
The general schedule for record retention
30 day retention schedule
first amendment to 30 day schedule
second amendment to 30 day schedule, extending it to 60 days
Hi again everybody. The conversation here is sufficiently interesting that I felt compelled to respond, just before heading out to a holiday weekend BBQ.
I think @Kolea pretty much hits the nail on the head, particularly the part about “Idealism minus pragmatic self-interest,†but I wanted to add a little more context and acknowledge @kerryrm’s questions. So let’s take one big step back.
A new media company comes along with the stated goal of sustainable public-interest civic journalism. Within just a few months, the new media company releases to the public the largest collection of previously undisclosed government salary data in the history of Hawaii. It does so for free despite the costs associated with obtaining the data and despite the fact that it is under no obligation to do so.
But that data, though released in its entirety to the general public for no charge, was not formatted in the manner that some citizens would prefer. So the reaction from those frustrated few, some of whom decline to use their real names, is to question the media company’s principles and say it’s on the “dark side” of transparency. That’s the same media company that just disclosed a mountain of public data never before available to the public, just to be explicit.
Another anonymous commenter calls the people who work at the new media company “scum” and says, “May his soul rot in hell.” Who is the “he” in question? It goes unsaid, but everyone here can take a guess. So while the discussion here has generally been a constructive one, no, @Eathan, I’m not terribly impressed by the civility of everyone involved.
This should go without saying, but I guess it can’t. If there is a “dark side” — if there is an us-versus-them dynamic in the fight for access to public documents — the journalists at Civil Beat are clearly, unquestionably, unequivocally on the side of sunshine. Go ask some state employees about Civil Beat if you’re not sure.
I understand the frustration with the paywall, and I understand the disappointment with Civil Beat in the particular instance under discussion here. But nitpicking some of Civil Beat’s business decisions — which I think is a perfectly healthy endeavor — shouldn’t lead to wild hyperbole. Please realize who your friends are, and don’t vilify them over a small-kine disagreement like the preferred formatting of already available public data.
As for @kerryrm’s questions: Why won’t Civil Beat make the document available to download? What value does it hold now? Those are fair questions, and I’m sorry to say that they are above my proverbial pay grade.
As I said in my first comment, I can’t speak for Civil Beat. I joined the conversation here to correct some of the factual inaccuracies in Ryan’s post that I feared were leading to confusion among his readers about what Civil Beat is up to.
So unfortunately, I can’t give you what you’re asking for. All I can do is tell you that your concerns have been heard and will be considered.
Happy holiday!
One last note @Ryan — the “chapter and verse” you’re looking for is HRS 92F-11(c)
http://capitol.hawaii.gov/hrscurrent/Vol02_Ch0046-0115/HRS0092F/HRS_0092F-0011.htm
“Unless the information is readily retrievable by the agency in the form in which it is requested, an agency shall not be required to prepare a compilation or summary of its records.”
Good luck!
Michael, if others at Civil Beat are not following this thread, after your holidays, why not suggest that they mosey on over here, where civil conversation is and has been available for the longest time, and join in themselves.
Having you and Adrienne participate is great, but it’s something very much like revealing the data under discussion partly but not fully. You say you can’t speak for CB. So where is CB? Is it against the business model to converse on another website? That’s not the Hawaii way, anyway. Speaking to CB through you both like this is something like wearing rubber gloves as I type.
I have no personal interest in the data. There’s a thing in Hawaii which it could feed about tearing down anyone who makes even a fair shadow of what might be a decent wage when compared to the Mainland. Still, it is public information and could be valuable in many ways. Were it available as data, that is.
While you laud Civil Beat’s investment in obtaining public data, that would be valuable to us if CB were indeed a journalistic effort. Just having an editor at the helm of a business doesn’t define it as journalism. Other factors come into play. Perhaps I can’t name them all, but I suggest that being read by the public is a big part of it. There are other experiments around the country that succeed while being available to their community.
The protection and limitation of access to the government data does not turn CB into a Wikileaks. What is often muddied in discussions of CB is the fact that it is a closed group. It’s somewhat like one of those investment newsletters where for $1000 a year you get reports and the ability to question the authors. I used to subscribe to one of those in a previous (pre-Internet) life. The way this matter is being handled is very closed-group, including CB’s absence from the conversation except through surrogates.
Ryan proposes to release the same data to everyone. Now, that’s journalism, even if he doesn’t make a living from it. Many Hawaii bloggers habitually provide links to source data. Even the newspaper sites began to do that, to a limited extent.
CB has choices. I don’t know the relative importance of this set of data in the scheme of things. but CB could simply release the source file, as others would do. Perhaps scholars and mainstream journalists on our TV stations or newspapers, not to mention the blogs, would thank them for their public service and be able to sit down with the data or have their news analysts review it. It would cost CB nothing, and instead of the criticism CB is getting among potential customers, it might have generated some goodwill and even new business.
CB makes its own choices and I don’t object to a business conducting itself as it must in order to make a profit.
CB could similarly be in the spotlight if it does the same with other data sets in the future, especially if something really valuable is tantalizingly cloaked as this is.
I just went over there to check on the UH salary data. Like those mailers that tell me I’ll read about the “10 most dangerous over-the-counter drugs” if I subscribe to a health mag, the article ended, “Here’s the list: Member content.” and then the blue box, “Get All-Day Access to All Posts $1.49.”
As to the Slideshare salary display, to say CB is making that available to the public is a joke. There are 334 pages to scroll through, hard to read even full-screen and partly obscured by the watermark. CB is releasing very little by posting that image. It probably pisses off more people than find it valuable, not to mention the employees whose salaries are now revealed so that CB can make money off of the data. Below the image is a clear demand for money in exchange for access.
For a mere buck and a half a state employee can now find out what the boss and co-workers make.
To have research value, the list, with or without the names, should be made available as Ryan and ninja are doing.
Hint.
@Michael Levine. I appreciate your employer’s position but I think characterizing the dispute as “not formatted in the manner that some citizens would prefer” is disingenuous. Naturally, some people will not be happy that your employer elected to make the raw data as unavailable as possible without just simply restricting access pay to play. And your employer may have good reasons as a “new media company… with the stated goal of sustainable public-interest civic journalism”. However, recycling the dismissive argument (used by secretive governments the world over) that the raw data was “made available” only makes people, who accept your employer’s business model, question the integrity of your employer and whether the stated goal is an aspiration or a marketing tool.
Contrary to your first comment, the .pdf is not available for free. The document is not available in any format other than a screenshot image. In analyzing access, you don’t look to how your employer uploaded it, but the characteristics associated with access. In this case, it is a qualitative slight improvement to having someone take a high resolution picture of someone holding it in their hand for the camera. The raw data was not made available but a screen shot of the raw data. (And there is a difference between those things.) It is your conflating the two that I think offends the sensibilities of even those that might otherwise support your employer’s business model.
A lot of assumptions being floated about journalism in this thread.
When we talk about a free press, weʻre not talking about money. Iʻm not quite sure how the two concepts became so conflated, but it strikes me as quite dangerous that they have.
The internet does not mean the end of the economy. Work still requires resources to produce. Including journalism.
The argument about public data behind a paywall is compelling enough, as long as one assumes that news produces itself for free and has always been free to consumers. But it doesnʻt, and more importantly, it hasnʻt.
If the Advertiser had acquired these documents in 1990, we would not be having this conversation. It would have cost you $.50 to read a copy of the paper that contained the information. That is to say, behind a paywall. And no, you would not have been able to export that information with the click of a mouse to use for your own purposes. You would have had to write it down, or pay a few cents to photocopy it. A small price to pay.
News became available for free on the web for a while, just like music before it. That does not mean that news became free to produce. It still costs money to have a people in our community whose full-time jobs are to perform the watchdog role that our Constitutions envisions for the press.
There is a lot of resentment out there toward reporters, including a lot of it here in this thread. I find that sad and disturbing.
With all due respect to Ryan, heʻs been around a lot longer than Civil Beat. He could have submitted these FOIA requests at anytime through the years. But he didnʻt. He, like the overwhelming majority of bloggers, is following the lead of the professionals. Thereʻs a reason for that. Professionals do a better job most of the time at most tasks in most fields.
Someone always pays. If there is something wrong with that, then thereʻs something wrong with the basic economic principle of work. More likely, there is something wrong with the way people view the economics of quality journalism.
Journalists are beginning to talk back on this point, and itʻs long overdue.
I’ve been lurking, following this thread from the beginning and am perplexed by one nagging question: what is your intention/goal in having/knowing/distributing the salaries of all these civic employees? Is it to antagonize someone you think makes too much money? To carry it around and whip it out whenever you have a bad day with a gov’t employee and say “look how much they make!”? I rarely see stuff like this floated around without malicious banter about how government emmployees make too much money attached – is that the goal?
I have worked for myself in my own business, private industry for others, local government, and academia. These salaries are not high, and probably for Hawaii, pretty low.
So my question again is: what is the big deal knowing them?
Robert Jameson has touched on the point that I had wanted to make from the beginning . What IS the big deal knowing them? Its not as if state workers are CEO’s taking million dollar bonuses at the expense of the public. In fact one can see from the provided data that if anything they are grossly underpaid relative to the cost of living in Hawaii.
Also did you even stop to think about how revealing these salaries may cause huge conflicts between state employees? “Oh my co-worker is making this much and I do more work than him!”
Have you even found anything by parsing through the data that looks suspect?
Lastly, if you are so adamant about getting a file instead of a watermarked slideshow, do the work and plug the data into Excel yourself. Or maybe you could hire a poor state data-entry worker who makes $30,000 a year to supplement their income.
Heck take screenshots of the slideshow, pdf it, OCR it, copy and paste into Excel.
Brian and Robert, you make a valid point as to the journalistic value of the information in question. I believe it’s valuable, as does Civil Beat, but that can certainly be debated.
My focus has been, as I hoped I made clear in my overlong blog posts, the acquisition of this public data by a ostensibly journalistic enterprise (that is also a business), but subsequently restricting its availability.
As to your suggestion to just extract the data from what was posted, this has been done and was announced above.
Though I’ve yet to post an update, I’m happy to say that the other public information requests I’ve made have been fulfilled without any hassles. So my cynicism that I would meet the same resistance I got from DHRD was unfounded.
Ryan, I simply find your characterization of this baffling. Civil Beat is an “ostensibly journalistic enterprise?” Ostensibly? What else would it be? And what is meant by your inclusion of the phrase “That is also a business”?
You are clearly suggesting that somehow being a business mitigates or at the very least casts a shadow on Civil Beat’s claim to be a journalistic organization.
I hope you will explain your thinking. I must tell you that in a lifetime in and around journalism, I have never once heard it suggested that journalism cannot be practiced as a business.
Ragnar, I think I’ve articulated my opinion reasonably well, and your prior comment did an even better job of explaining the opposing, easily prevailing view.
It is my view that an agency that was “journalism first, business second” would have had no problem releasing a copy of the PDF that the state prepared, rather than restricting it. So I’m noting that an agency with the inverse prioritization might not.
Ultimately, I agree that Civil Beat is a different animal and a new hybrid and doesn’t have to fit into any specific box. I’m simply disappointed in the animal that Civil Beat is revealing itself to be. And I say this as someone that was probably the most excited of anyone I know by the first hints of its arrival.
Ragnar is taking a very narrow view of document requests. This isn’t 10 or 20 years ago. Making record requests, getting those records and then playing hide and seek with them is not the custom that has developed with the completely different information-sharing environment of now (not 10 years ago). In today’s information age, reporters share raw data so that readers and critics can use their own analysis to test the conclusions that occur in all reporting. Part of the reason the advertiser didn’t do that 10 years ago was because it was technologically impractical. However, since the widespread availability of broadband, .pdf format, etc., sharing raw data is quite customary.
Real journalists are known for their analysis and mental library of relevant historical information to put the raw data in context not for playing hide and seek with raw data.
I had a friend who used to tell me that music should be free. He could never explain why that was true, and he would get angry when I would ask whether he meant that musicians should volunteer their labor for his benefit. But he was sure it was true. Music should be free. That was as far as it went.
What he meant, of course, was that music was free, at that moment in time. And that he liked it that way, because it meant he could have music for free. The “should” was simply an expression of his desire that that situation might continue.
What all of this boils down to: over the past 20 years, people have been spoiled into believing that they have a right to free reporting. There is no intellectual or moral basis for it…it’s just what folks are used to, and they don’t want to pay for something they are used to getting for free.
The notion that journalists should volunteer their labor is ridiculous on its face.
@Ragnar: You aren’t really listening to what I’m saying and by mischaracterizing what I wrote, it’s hard to have a discussion. I didn’t say anything about free or “right to free reporting.” I specifically stated where I think value comes from in reporting. (My main point which has been left wholly unaddressed.) Expecting the sharing of raw government data is not like pirating music because the requester of the data did not materially participate in the creation of the records. The metaphor doesn’t work exactly (maybe if we were talking about sharing reporter’s notes or something but we’re not). The Honolulu Weekly, for example, has historically been a place where the value of the reporting was on precisely what I previously stated to be the value of real journalism. Readers participate in a complex system whereby revenue is generated without them directly having to pay the paper. Keeping government records behind a paywall is what all of those realtor info services do and they make good money from doing that. But that isn’t journalism.
I’m not exactly sure where the passion that animates the polemical/moral tone of your arguments come from, but I think it too narrowly frames the issue (in addition to mischaracterizing my arguments and Ryan’s as well). If you look at the law that enables journalists (and members of the public) to obtain government records, “Opening up the government processes to public scrutiny and participation is the only viable and reasonable method of protecting the public’s interest.” HRS 92F-2.Law adopted to allow access to government records for public scrutiny and participation derive from great interest by society. It has been in this spirit that journalists have shared and made records they obtain publicly available. You see, it is a policy supported by the people (which made it law) that has provided a rational mechanism for the public to obtain government records. It is not the same as other methods of obtaining information such as interviews, contacts, etc.,.
I also think something is lost when reporters sees themselves as simply collectors and not reporters because the primary value of journalism in an information age is analysis and bringing in of illuminating context. Real journalism is not about using laws intended to open up government to public scrutiny by limiting public scrutiny to those that can pay. The pay part should be for the analysis not for the publicly available government records and this is even more true when the originating agency will not produce the records for anyone else now.
Well, I certainly apologize for being glib. I was responding to your use of phrases like “custom” and “customary” to refer to what are essentially brand-new developments in media: the newsgathering organization does the heavy lifting, and then everyone else profits from that work free of charge. That and words like “ostensible” and “real journalists.”
I disagree with a number of your premises, in particular that newsgathering has limited value in the information age and that it’s analysis of the information that counts. I would argue that the reverse is true. Technological changes have made analysis/opinion the province of everyone, but the facts themselves still require gathering. Indeed, as news has gone “free” and newsgathering organizations have suffered, facts are in increasingly short supply. You mention Honolulu Weekly. I am editor of that paper, and in the two years I have been there, my aim has been to move away from the jeremiads that once characterized weekly papers and toward more straightforward reporting, and for this very reason.
There is a crisis facing democratic government right now, and it is the crisis of the collapse of the business model for newsgathering. Despite a lot of mumbling about “dinosaurs,” a lot of very smart and committed people have been trying very hard to figure out how to keep professional journalism alive in an environment in which most citizens are not interested in paying for it.
Civil Beat is one attempt at a solution. I personally favor the non-profit approach, and it’s one that is showing signs of life elsewhere. But I have no ethical problem with what they are doing: they are trying to build a market for quality journalism that does not currently exist. The Wall Street Journal has a similar model. The Weekly (also a for-profit business) has another one, and I can tell you firsthand that it is not succeeding right now: the Weekly can afford, barely, two full-time journalists. That is not enough, and we are not able to do the things we’d like to do. One of those things, incidentally, was exactly the report that eventually appeared in Civil Beat…I got the idea from the Texas Tribune in January and simply have not had the resources to pursue it. I am glad they did.
I’m getting a little far afield here. The larger point I am expressing is astonishment at the use of words like “ostensible” to describe Civil Beat’s journalism, or your use of the phrase “real journalists” to exclude what they are doing. And at the resentment toward them for being a business. The primary newsgathering organizations have for the entire modern era been businesses, and they have always “hidden” their reporting behind a paywall. What needs to be adjusted, in my view, is the attitude of the public toward journalism.
The Weekly was the first organization to acquire a copy of the Final EIS on rail. Civil Beat, and others, asked us for a copy of that document. We didn’t provide it, and we didn’t post the whole thing on our web site, and we didn’t hear a word about it. We simply printed excerpts and analysis in the paper. Did I feel great about it? I felt great that we got it first, and less great about holding on to our copy. But what are you going to do? Newspapers are going out of business every day. At some point, you have to be willing to say “the facts in this document are public information, the presence of it in the community is our work, that’s our business and so we are holding on to it.” If journalists continue to give away their work product, newsgathering organizations will go away. And everyone will be worse off for it.
As a final note, I would say that the music metaphor holds perfectly, and the request did in fact create something that had not previously existed in this case: a comprehensive listing of the salaries of government employees. That is the work product of the reporting that was done.
Even the state charges money to see this kind of information. The notion that a reporter is somehow compelled to give it away for free, after the considerable expense in time and money that was required to produce it, is very hard for me to understand.