[tor-reports] PETS 2016 Reportback

[Paul Syverson]

On Fri, Aug 05, 2016 at 08:46:54AM -0400, Aaron Johnson wrote:
> 
> > Prior to the event, I reviewed many of the PoPETS (the quarterly
> > academic journal run by PETS) paper submissions.  Several of these
> > were pertinent to my interests.  I'm not supposed to say which
> > ones I reviewed, and I find this requirement to be in harsh
> > conflict with cross-community open discussion of ideas.  This may
> > be the only time I'll ever be opposed to anonymity, but academia's
> > manditorily-"anonymous" submission/review system should be
> > destroyed.
> 
> Is it true that PoPETS reviewers aren’t supposed to deanonymize
> themselves? If so, that is not a consistent policy across all peer
> publication venues. For example, I have seen intentionally-signed
> reviews at ACM CCS. My understanding is that the main reason
> reviewers are anonymous is to allow them to provide frank
> assessments. If a reviewer isn't worried about that, then that
> concern wouldn’t seem to apply.

There seems to be some confusion here. My understanding of standard
expected behavior, at least for the corner of "academia" that I am
familiar with, is that one should not identify or discuss papers that
one has reviewed outside of the context of reviewing that paper.  In
that sense, Isis would be correct that she should not reveal on such a
list as this (regardless of access restriction) which papers she
reviewed. This has many purposes. One is to protect the authors from
having their work in progress judged prior to them feeling it ready
for scrutiny outside of contexts they choose to trust. I would
consider this along the same lines as not being forced to discuss ones
half-baked ideas outside of a chosen trust context.

But Aaron is also correct IMO that reviewers are not precluded from
identifying themselves to authors (once the review and selection
process has been completed).

The above reflect I think longstanding standard practice for blind
reviewing of academic work. There is lots of ongoing debate about
appropriate ways to make public, evaluate, select, discuss, publish,
etc. academic work. But it looks like here there is a misunderstanding
of standard practice in play here.

> 
> I would encourage you to bring this concern to the PoPETS editors
> (Claudia Diaz, Rachel Greenstadt, and Damon McCoy
> <pets17-chairs at petsymposium.org
> <mailto:pets17-chairs at petsymposium.org>>). If this is the PoPETS
> policy, then I oppose it as well.

I would also encourage checking with the current editors for their
expectations. But even if they said that it is completely OK to
discuss publicly papers that one has reviewed, I would not do so.  I
feel sure this would violate the standard community privacy
expectations of those that submitted. Even if there is some statement
in the submission instructions that noted this, I would want some
really loud explicit, are-you-sure-you-understand notifications in the
submission process. Otherwise it would have all the moral validity of
way-outside-norm-use-and-expectation clauses clickthrough agreements.

I don't know of any venue that explicitly spells out that it is
acceptable for reviewers to e.g. identify themselves to authors after
the fact. I would hate if there were elaborate detailed description of
all this, varying subtly for each venue or worse, some "official"
standard.  Likewise, there is simply an implicit non-disclosure honor
trust rather than something one actually signs. That could turn into a
nightmare, making reviewing an even more onerous task to volunteer for
than it now is. I remain a fan of institutional/cultural memory in
this context. (Take this message as the extension of such.)

I have personally contacted authors of papers in the single-blind
review context after the evaluation for the venue is completed.  And
in the rejected double-blind submission case I have told the chair(s)
that I would like to discuss things further with the authors after
decisions are made and asked them to contact the authors to see if
they are willing. I have also asked authors if I can discuss their
work further in public or in specific contexts (e.g. share with NRL
colleagues) with an agreement about whether sharing further requires
new permissions. Once they have made their work public, I of course feel
free to discuss the paper, although this does not thereby automatically
license unrestricted revealing of reviews and discussions
that took place during evaluation.

HTH,
Paul