Thinking about Open Peer Reviews

Over the holidays, I saw an interesting article on F1000Research

Soergel DAW. Rampant software errors undermine scientific results [v1; ref status: approved with reservations 2, http://f1000r.es/4w2] F1000Research2014, 3:303 (doi: 10.12688/f1000research.5930.1)

and felt inspired to comment on it.  The editors asked if I would use my comment as the basis for a review, which I agree to, leading to my first open peer review:

Katz D. Referee Report For: Rampant software errors undermine scientific results [v1; ref status: approved with reservations 2, http://f1000r.es/4w2] F1000Research 2014, 3:303 (doi: 10.5256/f1000research.6338.r7096)

This made me think about the peer review process, and what specifically was different about open peer reviews (where the reviewer is known) vs traditional closed peer reviews (where the review is anonymous.)

I mentioned this in a tweet, and someone pointed me to

Aleksic J, Alexa A, Attwood TK et al. The Open Science Peer Review Oath [v1; ref status: indexed, http://f1000r.es/4ou] F1000Research 2014, 3:271 (doi: 10.12688/f1000research.5686.1)

which led me think about this even more, leading the following, which is also posted as a comment on the Open Science Peer Review Oath paper.  Comments are welcome.

Reader Comment 22 Dec 2014
Daniel S. KatzUniversity of Chicago, USA
​The suggested oath, broken down by principles and items, is quite interesting, with some of the principles having clear value and little overhead, while others that have potential value also add large amounts of possible overhead.  I suggest that these be broken down into two types of open reviews.  Specifically:

Passive review:
Follows all of Principles 1 through 3, and elements x – xii of Principle 4.  Here, the idea of an open review basically is a review that is signed to make sure the process has appropriate integrity.

Active review:
Includes all elements of Passive review, as well as items xiii and xiv of Principle 4:

  • xiii) I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible
  • xiv) I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, so that your experiments can be repeated independently

The idea of an open review here seems to be that, in addition to ensuring integrity in the review process, there is also extra work being done beyond a standard review, and the person who does such work should be credited for doing so.

I am uncertain about where elements xv and xvi, as written:

  • xv) I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use;
  • xvi) I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability

would fit.  These are neither active nor passive, and as written, they don’t match the review function, but are even more active, more a collaboration than a review.  I suggest that they be rephrased as:

  • xv) I will check that the data that underpin the published concept are made available in a manner that provides long-term unrestricted access, towards transparency and re-use;
  • xvi) I will check that any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access, both for reproducibility of results and software availability

so that they could be part of an Active review.

Of course, this specific remedy is just a suggestion, but I the overall point I want to make is that the added work to be done by the reviewer beyond what is now standard needs to be explicitly considered in both the oath itself as well as the description of the oath.

Competing Interests: none

Advertisements

Published by:

danielskatz

Assistant Director for Scientific Software and Applications at NCSA, Research Associate Professor in CS, ECE, and the iSchool at the University of Illinois Urbana-Champaign; works on systems and tools (aka cyberinfrastructure) and policy related to computational and data-enabled research, primarily in science and engineering

Tags , 1 Comment

One thought on “Thinking about Open Peer Reviews”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s