[tor-reports] PETS 2016 Reportback

[Aaron Johnson]

>> Is it true that PoPETS reviewers aren’t supposed to deanonymize
>> themselves? If so, that is not a consistent policy across all peer
>> publication venues. For example, I have seen intentionally-signed
>> reviews at ACM CCS. My understanding is that the main reason
>> reviewers are anonymous is to allow them to provide frank
>> assessments. If a reviewer isn't worried about that, then that
>> concern wouldn’t seem to apply.
> 
> There seems to be some confusion here. My understanding of standard
> expected behavior, at least for the corner of "academia" that I am
> familiar with, is that one should not identify or discuss papers that
> one has reviewed outside of the context of reviewing that paper.  In
> that sense, Isis would be correct that she should not reveal on such a
> list as this (regardless of access restriction) which papers she
> reviewed.

I took "I'm not supposed to say which ones I reviewed” to mean “I’m not supposed to say which published PETS papers I reviewed”. Unpublished submissions should of course remain confidential. I don’t see any problem with a reviewer identifying themself once a paper has been published.

> But Aaron is also correct IMO that reviewers are not precluded from
> identifying themselves to authors (once the review and selection
> process has been completed).

I am not certain that reviewers are expected to wait to identify themselves until after reviewing and selection has been completed. Signing a review sent to the authors would often violate that, given the rebuttal phase many venues have. My opinion is that they need not wait.

> But even if they said that it is completely OK to
> discuss publicly papers that one has reviewed, I would not do so.  I
> feel sure this would violate the standard community privacy
> expectations of those that submitted.

I assume you are only talking about papers that have not been published. Expecting a reviewer to hide the fact that they reviewed a published paper is not a norm I am aware of or one that I would support. Some situations in which you might justifiably want to identify yourself as a reviewer are:
  1. You want to publish your review, which you put non-trivial time and effort into and which you think other might benefit from reading
  2. You want the authors to quickly understand the context (e.g. your review) in which you are (later) discussing their work
  3. You want to explain to somebody why you supported the paper for acceptance and thus in part why it was accepted
  4. You want to qualify your understanding of a given paper by explaining that you reviewed an earlier version and thus are unaware of changes that were made after submission

If there are good reasons to prohibit identifying yourself as a reviewer of a paper after its publication, I would like to hear them (I can imagine some weak ones: (i) you want to contribute to the anonymity of the other reviewers, and (ii) you don’t want to enable a quid pro quo between the authors and reviewer for accepting a paper).

> Once they have made their work public, I of course feel
> free to discuss the paper, although this does not thereby automatically
> license unrestricted revealing of reviews and discussions
> that took place during evaluation.

Any restriction on a reviewer publishing their own review after a paper’s acceptance is a fairly extreme limitation on freedom of expression. I assume you must be talking about restricting reviewers from publishing *others’* reviews, which I would agree is a norm, although I think it’s a harmful one. It seems to me that everybody would benefit if all reviews for published papers were themselves published (possibly with reviewer anonymity) by the venue itself alongside the paper. That’s a different debate, though ;-)

Best,
Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-reports/attachments/20160805/f53b52ce/attachment-0001.html>