Report from the Peer Review Workgroup

Download

Abstract

The OSI2016 Peer Review workgroup focused on peer review in the context of open scholar­ship. The group agreed that greater openness and transparency would improve accounta­bility, minimize bias, and encourage collaboration, but did not underestimate the challenges of openness, nor the variation in readiness across disciplines and publishing mod­els. The group recommended facilitation of peer review outside the traditional publication process—for example, in the context of preprint servers and after publication—with incen­tives for broad participation. These incentives need to include a cultural shift in recognition of peer review as a valid activity contributing to career progression.

OSI2016 Workgroup Question

Managing the peer review process is one of the major attractions and benefits of the current publisher-driven publishing environment. Would it be possible to maintain peer review in different system — perhaps one where peer review happens at the institutional level, or in an online-review environment? How? What is really needed from peer review, what are the reform options (and what do we already know about the options that have been tried)?

Peer review is the worst form of  evaluation except all those other forms that have been tried from time to time. - with apologies to Winston Churchill

Introduction

Peer review is a much-discussed topic. The peer review workgroup at OSI2016 was well aware that there is already consider­able material available on peer review: a little formal research; a great deal of thoughtful and insightful commentary; organizations such as PEERE,[1] which aims to improve the transparency, effi­ciency and accountability of peer review; and a range of experiments and innova­tions already taking place. Much of the debate is about the different flavors of traditional peer review in the context of the editorial assessment that leads to publi­cation in a scholarly journal or monograph—double-blind or single-blind versus open—and which flavor is best suited to minimize the acknowledged limita­tions of traditional peer review.

Conscious of the sheer quantity of think­ing and innovation already taking place in peer review, the workgroup agreed that within our timeframe of two days we would be unlikely to reinvent peer review, and that it would not be a good use of time to go over ground that had been well covered already. We therefore agreed to focus on peer review in the context of open scholarship and to identify immedi­ate priorities for further work.

Overview of recommendations

After an evaluation of the entire scholarly process from idea to post-publication commenting, and after conducting an analysis of the strengths, weaknesses, op­portunities, and threats of traditional peer review (see Appendix: SWOT), our workgroup identified goals that could be implemented to move peer review toward a more transparent and inclusive process. Our recommendations are as follows:

• Move toward greater transparency to improve accountability and minimize bias.
• Move toward greater inclusiveness by encouraging wider participation in for­mal and informal peer review at different stages in the development of research outputs.
• Explore separating the evaluation of rigor and soundness before publica­tion (e.g., through technical checks) from the evaluation of interest, im­pact, and originality after publication in order to speed time to publication.
• Identify new approaches that lessen rather than increase the burden of re­viewing and decrease the waste of reviewer’s time.
• Conduct more evidence-based anal­yses of different forms of peer review.
• Address incentives and motivations to participate in peer review, not only in the context of rewards or credits for in­dividuals but also in terms of the importance of peer review for promo­tion and tenure.

We recognize the following challenges:

• There are differences among disci­plines, publishing models, generations, platforms, and so on that affect the practicalities of a move to com­plete transparency.
• Incentivizing a cultural shift toward openness is required not only for “tradi­tional” peer review but also (and even more so) for some of the innova­tions that we discussed.

Underpinning all of our discussions was the question of motivation and incentives. The value of peer review needs to be acknowledged, whether through formal credit systems or in simple rewards for taking part, but also this work needs to be valued by the employers and funders of those doing peer review and to count mean­ingfully towards career progression.

What is peer review?

Peer review was defined for the purpose of this discussion as follows:

• Peer review is the critical evaluation of scholarship by relevant, non-biased ex­perts who give constructive feedback.
• Peer review provides validation of scholarship with the goal of its im­provement.
• Peer review may include (and tradition­ally does include) judgment of impact on and interest to a field.

According to this definition, peer review serves more than a gatekeeping or filtering function in the scholarly dissemination process: it can and should contribute to and enhance the literature.

For the sake of convenience, this report uses “publication” in order to refer to the traditional, formal publication of results in a journal or monograph. We acknowledge, however, that publication can be formal or informal, can include making research outputs publicly available throughout their development, and that these outputs can potentially benefit from some form of peer review at any stage.

The categories of peer review we defined and discussed were:

• “Pre-publication” peer review refers to any form of review by one’s peers prior to formal publication (i.e., as a journal article or monograph).
• “Traditional” peer review refers to the formal review process typically under­taken as a contingent of formal publication.
• “Post-publication” peer review refers to any form of evaluation following for­mal publication.

Recommendations for pre-publi­cation peer review

Our group recommends the following actions with regard to pre-publication peer review:

• We encourage the use of preprint servers so that work in draft is openly available, which enables wider feed­back and commentary, leading to greater readiness for formal review in the context of a publication. Peer re­view aside, we believe that preprint servers in themselves are beneficial, es­tablishing priority and increasing the speed at which information is dissemi­nated, while also encouraging collaboration through feedback and commentary.
• In this context, we also encourage the facilitation of a flexible, nonlinear pro­cess of peer review outside of and supplementing journal-based peer re­view that facilitates many kinds of scholarly engagement and collabora­tion (e.g., through new tools and broader cultural acceptance). The type of review conducted will depend on the nature of the output, the stage in de­velopment of the output, and the timing of the output.

Questions:

1. Is there a role for preprint servers in the publication of scholarly mono­graphs and could they benefit from a form of peer review in this context?
2. How could participation by a wide spectrum of commenters be encour­aged, so that all research benefits?
3. Is the option of anonymous review necessary to encourage participation, but with credits only available for signed reviews?
4. Should reviewers be acknowledged in the formal publication?
5. To encourage participation in review at this early stage, should there be a closed (facilitated) route—for exam­ple, overseen by learned societies?

Recommendations for traditional peer review

Our group recommends the following actions with regard to traditional peer review:

• We recommend that all disciplines work toward a culture of openness in peer review.
• We encourage the exploration of and deliberation on the problems, real and perceived, regarding transparency in peer review.

We acknowledge that we need to hear from a broad spectrum of authors as im­portant stakeholders, in order to better understand the difference in acceptance and enthusiasm for more open peer re­view in different disciplines. Evidencing these differences:

• A number of journals already operate fully open peer review, without a blind review process, with signed reviews that are published alongside the fi­nal article (e.g., many BioMed Central journals, BMJ Open).
• Frontiers journals publish the name of the reviewers on the articles.
• eLife publishes the editor’s decision letter, including a summary of the re­viewers’ reports as well as the authors’ response. The reviewers and editors col­laborate on the key points that need to be addressed by the authors.
• The Journal of Design and Science from the MIT Media Lab will offer open re­view on a collaborative publication platform, encouraging community col­laboration rather than formal peer review.
• The EMBO Journal publishes the un­signed reviewer reports, editor decision and author responses with the published article.
• PeerJ reviewers may choose to sign their reports.

The benefits of fully open peer review are generally accepted to include the follow­ing:

• less likelihood of bias, or transpar­ency if a reviewer has a potential competing interest
• accountability of reviewers
• reviews may be more constructive[2]
• if published, the review becomes part of the published record and the re­viewer can be credited with the work.

The workgroup recognized that not all disciplines are ready to make the leap to a fully open and transparent process of peer-review, and also that there are areas where full openness might not be appropri­ate—for example, for ethical or security reasons. Furthermore, the group was aware that some journals choose to use double-blind peer review in the belief that it tackles some of the issues that open peer review also addresses, or for broadly cultural reasons (e.g., fear of retribution, etc.). More evidence-based research on the different forms of peer review would be useful.

Meanwhile, we encourage preliminary steps that encourage the cultural shift needed on the road toward fully open peer review. Such steps could include decoupling the elements of fully open peer review by

• making the signing of reviews possible but optional, encouraging reviewers to identify themselves to the authors if they so wish;
• allowing reviewers to choose to have their names published along with the pa­per; and
• publishing unsigned reviews with the paper.

Third-party tools may be used to facilitate the disclosure and publication of reviews. A few options include:

• Peer Review Evaluation (PRE)[3] pro­vides third-party validation of the review conducted on a journal’s pub­lished papers, through a widget showing the level at which the paper was reviewed. This widget also gives publishers the ability to include the re­views alongside published papers if they lack the technology to implement this themselves.
• Publons[4] allows individuals to create verified reviewer profiles for tracking their review activity. In addition, pub­lishers can partner with Publons to encourage reviewers to create profiles and publish reviews on the Publons plat­form.

Recommendations for post-publi­cation peer review

Our group recommends the following actions with regard to post-publication peer review:

• We recommend the facilitation of post-publication review of tradition­ally reviewed publications.
• We believe this is more likely to work if it is formalized and facilitated.
• We recommend experiments with crowd systems that incentivize broad, representative participation—for exam­ple, with a currency, rating, or credit system.
• As with review at other stages of the development of the research output, any credits or ratings should be acknowl­edged by employers or funders of those doing the reviews as valid metrics in career progression.

We acknowledge that the benefits to re­search of facilitated post-publication peer review (detection of errors and miscon­duct, continuous improvement of the literature) must outweigh the complexities that this might introduce into scholarly communication, such as the sheer amount of information that could become availa­ble; the technology required to underpin such a system; and the processes required to facilitate useful discussion, review, and response.

Questions:

1. Should comments on and discussions about articles be appended as part of the published record, or should the arti­cles themselves be updated on the basis of post-publication review, just as they are with pre-publication and during-publication review?
2. What would be the implications of a multiplication of versions for indexing and citation, and for the concept of the version of record?

A number of forms of post-publication commentary are already available, both on publication platforms and on third-party platforms. The comments made available on some of these platforms are anony­mous and include:

• informal unsolicited discussion or commentary after full peer review (e.g., in the comments section of the publication platform itself) and third-party options such as PubPeer,[5] PubMed Commons[6] and ScienceOpen[7]
• publication after a technical check followed by invited full peer review (e.g., with F1000 Research, the article appears in PubMed only after two peer reviews have been received[8]).

Potential issues to consider

Our workgroup included members who have published research, but no members whose primary role was conducting (or reviewing) research, nor employing or funding researchers. We cannot design new models of peer review without the input of these stakeholders.[9]

Further, the recommendations above led to several questions that should be as­sessed further:

1. Would encouraging the extensive use of preprint servers for all scholarship and encouraging the formal publica­tion of reviewer reports contribute to information overload?
2. Are there standards for validating reviews?
3. Are we technologically ready for pre­print servers?
4. What impact will peer-review outside formal publication have on journals? What will the role of the journals be? So­cieties vs. publishers? Are journals the “filters”?
5. What is the sweet spot in the timeline for the open review? Is it field-depend­ent?
6. Would costs be shifted rather than reduced? Publication of multiple ver­sions of scholarly outputs may incur additional costs.
7. Can we kill the concept of the “ver­sion of record,” allowing updates and re-review?
8. How can fears—real or perceived—regarding possible retribution in an open review culture be addressed? How will potential cases of retribution be addressed?

Conclusion

On our first day we produced a SWOT analysis of traditional peer review (see Appendix: SWOT). The weaknesses can be grouped under these broad headings:

• possibility of bias
• not designed to identify misconduct
• delays publication
• may conceal conflicts of interest
• lack of incentive to participate (e.g., credit).

We agreed that more open peer review should, by its very openness, address is­sues of bias and conflicts of interest. Extending peer review outside the formal publication process and opening it up to a wider cohort of reviewers might increase chances of identifying misconduct. Pre-publication review in the context of pre­print servers should mean that papers are submitted when ready for publication, reducing rounds of re-review by the jour­nal. That said, by formalizing peer review during multiple steps in a publication pro­cess, attention must be paid to the burden on reviewers. Expecting a more formal review process at multiple stages of scholar­ship could exacerbate reviewer burnout. Providing credit may incentivize reviewers but it not yet known whether credit would in fact encourage more people to participate as reviewers.

The workgroup concluded that open re­view at all stages of the research process would include the benefits of more democ­racy, more people involved, more likely to spur collaborations, and more sharing of knowledge at an earlier stage (prior to formal publication).

The following actions are recommended. Over the short term, we should:

• collate a library of literature on peer review in the context of open scholar­ship, including published research, good commentaries, and blogs;
• encourage peer review for grant fund­ing and for outputs other than research paper (e.g., data sets and soft­ware);
• develop a better understanding of how peer review is conducted in the hu­manities and social sciences, especially for scholarly monographs, in order to assess how it could be more open; and
• collate an overview of current, innova­tive peer review initiatives, again with a focus on open scholar­ship.

Over the longer-term, we should:

• carry out pilot studies or focus groups in order to gain a greater understand­ing of different perspectives of researchers with respect to open peer re­view in different disciplines (e.g., genomics, engineering, psychology, his­tory); and
• standardize the use of DOI for peer reviews and for the identification of dif­ferent versions (i.e. pre-publication, post-publication), working with Crossref, NISO, and others.

OSI2016 Peer Review Workgroup

• Bev Acreman, Commercial Director, F1000
• Peter Berkery, Executive Director, American Association of University Presses (AAUP)
• Caroline Black, Editorial Director, BioMed Central (SpringerNature)
• Chris Bourg, Director, MIT Libraries
• Becky Brasington Clark, Director of Publishing, Library of Congress
• Angela Cochran, Director of Journals, American Society of Civil Engineers (ASCE)
• Kevin Davies, Vice President for Business Development, American Chemical Society, and Publisher, C&EN
• Rachel Dresbeck, President, National Organization of Research Development Profession­als (NORDP) and Director of Research Development and Communications, Oregon Health and Science University
• Catriona MacCallum, Advocacy Director, PLOS
• Paul Peters, CEO, Hindawi Publishing
• Bobby Schnabel, CEO, Association of Computing Machinery
• Francisco Valdés Ugalde, Director General, Latin American Faculty of Social Sciences (FLACSO) in Mexico

 

Appendix: SWOT of traditional peer review

Strengths

Weaknesses

Relies on trust/goodwill· Trusted by researchers Peer review adds value Provides some level of validation by experts Mostly trusted by most research communities Imperfect, but the best system we have to date It does work within its limits Voluntary/free Encourages care and rigor Filters for a target audience Expert scrutiny Often leads to improvements or discovers flaws Adds credibility to published works Can sometimes spot flaws Improves papers when it works properly Improves science and stimulates thinking Sets criteria for acceptance, thereby motivating authors to improve quality[10] Favors discussion and feedback Tried and tested Careful reading is a benefit

Lack of openness hides bias· Biased with regard to gender, affiliation, country, discipline, which interfere with objectivity and em­power certain views and/or paradigms Not transparent – biases go uncovered Susceptible to conflicts of interest (amongst review­ers, editors) Single-blind peer review allows reviewers to veil criticism behind anonymity[11] Not ‘blind’ enough Unintentionally promotes conservatism (especially grants, but that’s a different conference perhaps…) Doesn’t promote innovation Negative/inconclusive papers not published Dependent on trust and goodwill, which is eroding Perceived credibility No credit for reviewing Not designed to identify (and doesn’t protect from) fraud and misconduct Data in supplementary material often overlooked Complex methods in multidisciplinary papers Review of only one research object (article) at one time period Little training for peer reviewers Increasingly difficult to find reviewers; open access journals may not attract quality reviewers Reviewers review for journals and editors, not for their peers Element of chance — only 2 or 3 reviewers out of many potential opinions No independent scrutiny and analysis Too few eyes The longest part of the publication process — can be time-consuming, slow — which delays publica­tion.[12] This might mean that important data is withheld from public/researchers. Reviewers at some journals delay publication by imposing burden­some/non-critical demands on authors Scooping Unwieldy system for managing is cost- and re­source-intensive Peer review stops on publication Doesn’t add value

Opportunities

Threats

Pre- and/or post-publication review could be a new model· Fully transparent post-publication review for journals Fully transparent pre-publication review for books Becoming more public Open, post-publication peer review Credit/recognition for reviewers an essential part of scholarly ecosystem Cascade review can reduce inefficiency[13] Automation/de-skilling of some elements–leave it to people to judge results Quality/science/impact Better tools for matching qualified reviewers to content In an online environment it is possible to make peer review more of an ongoing pro­cess Open review promotes transparency[14] Portable peer review Remove shackles of print/mail and develop existing system for digital world

Peer review is an attention portal that adds value, so changing it could be threatening· It is unclear whether researchers will continue devot­ing time to peer review if they are not incentivized to do so If not done by the journal where does that leave the journal? Does it matter? “Managing peer review” becomes commercial prod­uct People thinking it’s fixed Novel ideas and emerging subjects disadvantaged Throw everything online and hope for the best leads to lots of shoddy information Flawed research still gets published (e.g., STAP, Benven­iste, etc.) Closing the scientific mind Gaming/fraud/cheating Bias Corruption Time (waste of extensive amount of time finding reviewers)

 

Notes

[1] PEERE (New Frontiers of Peer Review), as of June 7, 2016: http://www.peere.org

[2] Kowalczuk et al, “Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models.” BMJ Open 2015;5:e008707, doi:10.1136/bmjopen-2015-008707.

[3] Peer Review Evaluation (PRE), as of June 7, 2016: http://www.pre-val.org

[4] Publons, as of June 7, 2016: https://publons.com/

[5] Pubpeer, as of June 7, 2016: https://pubpeer.com/

[6] PubMed Commons, as of June 7, 2016: http://www.ncbi.nlm.nih.gov/pubmedcommons/

[7] Science Open, as of June 7, 2016: http://www.scienceopen.com

[8] “Peer Review Status and Indexing of Articles,” in About F1000 Research, as of June 7, 2016: http://f1000research.com/about

[9] Different stakeholder groups in peer review and peer review processes include authors, reviewers, and editors; readers, public, patients, and practitioners; institutions, promotion & tenure committees; funders; policy makers; industry, investors, and journalists; publishers and publication aggregators.

[10] Ware, Mark, “Peer Review: Recent Experience and Future Directions,” 2011, New Review of Information Networking, vol. 16, p. 29, http://dx.doi.org/ 10.1080/13614576.2011.566812; or see also Ware, preprint, p.5, as of June 7, 2016: http://www.markwareconsulting.com/wordpress/wp-content/uploads/2011/01/peer-review-in-2010-2011-01-02-preprint.pdf

[11] Ibid, p. 32 (p.6)

[12] Ibid, p. 32 (p.6)

[13] Ibid, p. 35 (p. 9)

[14] Ibid, p. 37-42 (p. 10-13)