Cruelty to Analog

The protection of peer review and peer reviewers

At the ARDG meeting on Wednesday, EFF presented a position on the protection of peer review and peer reviewers. We proposed that one sentence be added to the ARDG's proposal solicitation document, but the ARDG refused to add it. Our position and proposal are included below.

"Submitters acknowledge that responses to this RFI may be subject to technical review and subjected to testing by independent third parties for the purpose of evaluating submitter claims about such technologies, and that the detailed results of such third party evaluations will be made public."

Independent, third-party evaluation and testing -- that is, peer review -- is essential to evaluating the accuracy of claims made by vendors about technologies. In response to this RFI, many technical claims will be made. Some of those claims will be factually incorrect, either because vendors desire to misrepresent the properties of technologies for commercial advantage or because they lack sufficient information to recognize why their claims are wrong. As our colleague Scott Craver mentioned in a presentation to the ARDG, peer review is the only way we know to arrive at the truth about empirical claims made for technologies.

Unfortunately, some researchers, such as Mr. Craver, have in the past been threatened with legal liability when they sought to research or to disclose the results of their research about the properties of technologies such as those sought by this RFI. This is unacceptible both from the point of view of free expression and scientific inquiry and from the point of view of determining scientifically, and in a public and transparent way, whether or how well technologies actually work for particular purposes.

Researchers need assurances that they will not be punished or harmed for carrying out their research. Without such assurances, a substantial amount of important independent research simply will not occur, to the detriment of public evaluation of vendor claims.

EFF would like the ARDG to consider how the RFI might be amended to provide such security for independent researchers.

ARDG Final Product, Part 2

At the July 9 ARDG meeting, the ARDG co-chairs let slip a couple more veils, teasing us with a bit more about the shape of the final ARDG report. There’s also a timetable.

The vast majority of the meeting was spent dickering over the precise working of various questions in the “analysis matrix.” (For more about what this “matrix” is all about, read the entry prior to this one.) The ARDG co-chairs do not want the matrix published until it is finalized. Imagine roughly 30 or 40 questions like this one:

9.2 Are the contractual requirements affecting the technical aspects of the proposal, if any, completely described a priori in non-confidential documents?

The ARDG co-chairs hope to complete the matrix and invite responses from vendors by early August. Vendors will be invited to respond in writing by September 15 and invited to attend and answer questions at the October 1 ARDG meeting in Washington DC.

So, what will the Final Report look like? Well, it hasn’t been finally decided yet (we’ll discuss it at the next meeting, set for July 30 in DC). It will, at minimum, include the self-reported responses of various technology vendors in response to the questions posed by the analysis matrix. Of course, that will amount to little more than a stapled collection of spec sheets and marketing literature.

Michael Epstein (Philips) suggested that there would be an opportunity to ask vendors to clarify any incomplete or ambiguous responses at the October meeting. He quickly stressed, however, that the ARDG is not empowered to “grade” the technologies against each other. So there will not be any probing evaluation and testing of the technologies. But there may be something a bit beyond the self-reported information, if only “comments and clarifications.”

I’m still not at all sure what this final report will be used for. Perhaps it will be submitted to policy-makers, so that various partisans can advocate for technology mandates premised on technologies described in the report? (“Mr. Commissioner, we strongly believe that you should issue regulations requiring everyone in the marketplace to implement technologies 12, 32, or 46 in the ARDG Report.”) I just don’t know. Stay tuned.