[tor-reports] PETS 2016 Reportback

[Paul Syverson]

[To tor-reports moderator: I think we've moved way off tor-reports,
but I feel funny responding but cutting off the thread. Feel free
to do so if you think it appropriate however.] 

On Fri, Aug 05, 2016 at 11:20:51AM -0400, Aaron Johnson wrote:
> >> Is it true that PoPETS reviewers aren’t supposed to deanonymize
> >> themselves? If so, that is not a consistent policy across all peer
> >> publication venues. For example, I have seen intentionally-signed
> >> reviews at ACM CCS. My understanding is that the main reason
> >> reviewers are anonymous is to allow them to provide frank
> >> assessments. If a reviewer isn't worried about that, then that
> >> concern wouldn’t seem to apply.
> > 
> > There seems to be some confusion here. My understanding of standard
> > expected behavior, at least for the corner of "academia" that I am
> > familiar with, is that one should not identify or discuss papers that
> > one has reviewed outside of the context of reviewing that paper.  In
> > that sense, Isis would be correct that she should not reveal on such a
> > list as this (regardless of access restriction) which papers she
> > reviewed.
> 
> I took "I'm not supposed to say which ones I reviewed” to mean “I’m
> not supposed to say which published PETS papers I
> reviewed”. Unpublished submissions should of course remain
> confidential. I don’t see any problem with a reviewer identifying
> themself once a paper has been published.

Personally, I agree if all you mean is saying, "I reviewed this paper."
If you make your review available, however, you may be revealing things
about a version of the paper prior to the published version, a version
that the authors may not have agreed to reveal beyond the trusted reviewers.
> 
> > But Aaron is also correct IMO that reviewers are not precluded from
> > identifying themselves to authors (once the review and selection
> > process has been completed).
> 
> I am not certain that reviewers are expected to wait to identify
> themselves until after reviewing and selection has been
> completed. Signing a review sent to the authors would often violate
> that, given the rebuttal phase many venues have. My opinion is that
> they need not wait.


I don't understand "signing" here. I did not say or intend that
reviewers could not ever reveal themselves to authors at any other
points. I did intend that when the evaluation is all done, it is
OK. Other points in the review process are more subtle, probably
should involve consultation with editors, etc.  I was already
overly long, so I was limiting myself to where I thought there was
not such ambiguity.

> 
> > But even if they said that it is completely OK to
> > discuss publicly papers that one has reviewed, I would not do so.  I
> > feel sure this would violate the standard community privacy
> > expectations of those that submitted.
> 
> I assume you are only talking about papers that have not been
> published. Expecting a reviewer to hide the fact that they reviewed
> a published paper is not a norm I am aware of or one that I would
> support. Some situations in which you might justifiably want to
> identify yourself as a reviewer are:
>   1. You want to publish your review, which you put non-trivial time
>   and effort into and which you think other might benefit from
>   reading

I addressed this above. If the paper is published with no changes
after submission, sure. If your comments apply only to the published
version, sure. That's tantamount to commenting on a published paper.
Beyond that it gets subtle. If they significantly rewrote the paper
after reviews, then your review could basically be describing a
preliminary version that they did not intend to be public.  I don't
think it is up to you about whether or not that version should be
publicly revealed or discussed. (There is also a potential concern
about what you may reveal about other reviewers, but I'd prefer to
drop that as a point than get into it as well.)

>   2. You want the authors to quickly understand the context
>   (e.g. your review) in which you are (later) discussing their work

I don't see why you can't just do this directly with the authors rather
than in some public way. As noted, I do think there are times it makes
sense to de-anonymize reviewers even before evalation is complete. An
obvious example is during shepherding. 

>   3. You want to explain to somebody why you supported the paper for
>   acceptance and thus in part why it was accepted

No disagreement, subject to the caveats concerning 1 obove.

>   4. You want to qualify your understanding of a given paper by
>   explaining that you reviewed an earlier version and thus are
>   unaware of changes that were made after submission
> 

OK, provided you are not significantly revealing things that the
author did not agree to be public, e.g., the paper as I reviewed it in
June 2016 made forward secrecy claims that showed they were completely
unfamiliar with the concept, or failed to distinguish in ways that
mattered timed mixes and threshold mixes, etc.


> If there are good reasons to prohibit identifying yourself as a
> reviewer of a paper after its publication, I would like to hear them
> (I can imagine some weak ones: (i) you want to contribute to the
> anonymity of the other reviewers, and (ii) you don’t want to enable
> a quid pro quo between the authors and reviewer for accepting a
> paper).

Right. I agree those could matter but are much weaker than the things
I raised above.

> 
> > Once they have made their work public, I of course feel
> > free to discuss the paper, although this does not thereby automatically
> > license unrestricted revealing of reviews and discussions
> > that took place during evaluation.
> 
> Any restriction on a reviewer publishing their own review after a
> paper’s acceptance is a fairly extreme limitation on freedom of
> expression.

I completely disagree. You have been shown something in confidence.
That what you were shown in confidence has a causal and other
connections to something that was ultimately published (about which
you should be free to comment) does not entitle you to reveal things
about what you were shown in confidence. 

> I assume you must be talking about restricting reviewers
> from publishing *others’* reviews, which I would agree is a norm,
> although I think it’s a harmful one. It seems to me that everybody
> would benefit if all reviews for published papers were themselves
> published (possibly with reviewer anonymity) by the venue itself
> alongside the paper. That’s a different debate, though ;-)


Apropos to my last comment, if the existing reviewing paradigm were
changed so that authors (variant: and other reviewers) understood that
if a paper is accepted then they agree to public revelation of a
submitted draft, then this would make more sense. But that is not I
think at all the understanding authors have now.  We could also get
into whole other publication paradigms with iterative versions,
comments, etc. being part of the "publication". I was trying to avoid
stepping on that slippery slope here, which I think your above
statement does. (Glad to discuss it elsewhere.)

aloha,
Paul