Skip to content
Publicly Available Published by De Gruyter June 17, 2015

What is Peer Review, and Does it Have a Future?

  • Steven W. Sowards EMAIL logo

Abstract

Peer review underlies the present system of scholarly publication. The present essay examines what authors, reviewers and editors are saying about peer review and its future. Factors discussed include peer reviewer motivation, the process and potential alternatives, validity and reliability, recognition and rewards for peer reviewers, and the impact of technology and Open Access on scholarly communication. There is widespread interest in preserving and improving peer review.

Zusammenfassung

Peer Review ist das Fundament des geltenden Systems wissenschaftlichen Publizierens. Der vorliegende Beitrag prüft, was Autoren, Reviewer und Editoren über den Peer-Review-Prozess sagen und wie sie dessen Zukunft einschätzen. Behandelt werden die Themen der Reviewer-Motivation, des Review-Prozesses an sich und seine möglichen Alternativen, die Validität und Reliabilität des Prozesses, Anerkennung und Belohnung für Reviewer und die Auswirkung von Technologie und Open Access auf die wissenschaftliche Kommunikation. Zusammenfassend zeigt sich bei allen Gruppen ein großes Interesse, das System Peer Review zu erhalten und zu verbessern.

1 Introduction

Peer review underlies the present system of scholarly publication. Peer review itself – the process of scrutinizing and validating research findings prior to dissemination – is itself the topic of numerous research studies, as well as opinion pieces and commentaries by researchers, journal editors, and peer reviewers (themselves almost always researchers in their own right).

The habit of science leads scholars to observe, critique and offer ideas when they think about peer review. The impact of the digital revolution on information-sharing – including publishing and scholarly communication – also promotes reflection and investigation of potential improvements.

There is an extensive literature on peer review. Representative of past summaries of the literature are a two-part article by Campanario in 1998[1], books by Weller in 2001[2] and Shatz in 2004[3], and a white paper by Ware in 2008[4] (updated by a review article in 2011)[5]. Interest is extensive in the health sciences, with conferences such as the International Congress on Peer Review and Biomedical Publication (Rennie et al. 2012)[6]. Scholars in the social sciences and humanities also are exploring the future of peer review. Kathleen Fitzpatrick has shared monographs for pre- and post-publication review through the Media Commons web site (Fitzpatrick 2014)[7]. An equally interesting part of the literature examines peer review of grant applications and data but falls outside the scope of this essay.

Today’s writing about peer review looks at the motivations of reviewers, the mechanics of the process, the outcomes of review, problems in the system, and solutions that range from minor enhancements to revolutionary changes. The present essay examines what authors, reviewers and editors are saying about peer review and its future.

2 Why take part in peer review?

Peer reviewers devote substantial time to their work, without payment and – at least under the traditional model – without recognition. A variety of formal surveys and informal inquiries have elicited consistent explanations about motivation. Sataloff reports that “By far, ‘playing your part as a member of the academic community’ was the most common reason [...]” (Sataloff 2011, 47)[8]. Ghazoul (2011)[9] and Francis (2013)[10] cite the same motives. Simons-Morton also points to the prestige and improved writing skills assumed to accrue to reviewers (Simons-Morton et al. 2012[11]).

This positive attitude is rooted in appreciation of the journal-based scholarly communications system. As researchers and scholars who submit their own manuscripts for examination, peer reviewers understand the need for such a system. A recent survey found that “the vast majority of reviewers believe that reviewing is an important part of being a researcher” (Mulligan, Hall and Raphael 2013, 149)[12].

These comments indicate a shared perception that peer review is worthwhile, and is a mark of membership in the scholarly community. The social component often is cited: one study quotes a researcher who says that peer review “‘should be looked at as more of a process, a social activity’” (Couzin-Frankel 2013, 1331)[13]. Other analysts point to “the reciprocity motives that typically drive human behavior in many social interactions” (Squazzoni, Bravo and Tákacs 2013, 293)[14].

3 The mechanics of peer review

While the motives for taking part are generally agreed upon, “peer review” can mean very different things in terms of process. Review supports the journal system as a means “to provide quality certification” (Ellison 2011, 640)[15]. However, there is no single accepted procedural model for peer review. Among other works, Lee et al. (2013)[16] and Mulligan, Hall and Raphael (2013)[17] are able to list single-blind and double-blind peer review, open peer review, hybrid peer review, a priori and a posteriori peer review, and post-publication analysis of usage as methods to certify quality and importance. Scholarly opinion is divided about preferences and effectiveness among diverse methods: “Only 25 % of those surveyed preferred the single-blind review process; 56 % preferred double-blind reviews, 13 % favored open review, and 5 % supported post-publication review. [...] Fifty-two percent felt that single-blind review was effective, with 37 % acknowledging post-publication review and 26 % recognizing open review as effective” (Sataloff 2011, 46)[18].

The operative activities involved in peer review also are hard to specify. “The peer review process itself is more or less a black box” (Bornmann and Daniel 2010, 11)[19]. Critics note that there seem to be no standardized rules controlling the journal editors who receive peer reviewer recommendations, and their decisions are private: “One challenge in studying peer review is simply getting the data: closely guarded peer-review reports submitted to biomedical journals, and the conversations that take place inside a journal’s offices” (Couzin-Frankel 2013, 1331)[20].

4 Validity and reliability in peer review

The stakes nevertheless are high. Defective peer review can disrupt the process by which manuscripts are certified as worthy of publication and dissemination.

Bornmann, who has written extensively on peer review, sums up the criticism in terms of reliability, fairness, predictive value, inefficiency, and personal stress:

“(1) reviewers rarely agree on whether to recommend that a manuscript be published [...]; (2) reviewers’ recommendations are frequently biased [...]; (3) the process lacks predictive validity because there is little or no relationship between the reviewers’ judgments and [...] the frequency of citations [...] in later scientific papers; (4) reviewing is inefficient because it [...] is time consuming and costly; and (5) reviewing can be [...] painful and distressing [...].” (Bornmann 2011, 203–204)[21]

Yet, the results of the process are accepted: “from this morass of conflicting advice comes a decision that fairly accurately discriminates high from low quality articles” (Jackson et al. 2011, 6).[22] At the same time, suggestions for improvement are widespread.

5 Finding more reviewers

Many scholars wish to expand the pool of reviewers in order to share the workload, and reduce bias. Because the statistical validity of review has been tied to the number of reviewer reports, open review and post-publication review have some appeal: “post-publication open review [...] could potentially improve the quality of reviews, by significantly increasing the number of reviewers and thus improving the statistical validity of a given manuscript’s score” (Herron 2012, 2276).[23] This could offset some of the uneven distribution of labor, in which “Active reviewers [...] completed an average of fourteen reviews per year. Hence, active reviewers are responsible for 79 % of all reviews” in one discipline (Sataloff 2011, 46).[24]

Limited pools of reviewers also can introduce bias, whether inadvertent or intentional. Souder (2011)[25] and Lee et al. (2013)[26] have compiled long lists of unfair influences including social bias, ad hominem bias, gender bias, affiliational bias, ideological bias, bias based on language or nationality, bias based on resistance to innovative or challenging ideas, bias against interdisciplinary work, and conflict of interest including financial conflict of interest. Among the most subtle factors at work is “Positive outcome bias (POB) [...] the increased likelihood that studies with a favorable or statistically significant outcome will be published than will studies of similar quality that show unfavorable or ‘no-difference’ results” (Emerson et al. 2010, 1934).[27] In the worst ethical instances, “a reviewer obstructs the publication of a manuscript because it competes with or questions his or her own work” (Casadevall and Fang 2009, 1274).[28]

Expanded reviewer pools can promote better representation of the gender, linguistic, ethnic, racial or national differences among global researchers, as noted by one journal: “Nature Chemical Biology is committed to expanding our referee database to ensure that our referee pool reflects the scientific, demographic and geographical diversity of our authors and readers” (Editors of Nature Chemical Biology 2010).[29]

Where bias is unintended, better training for new reviewers may be useful. As one researcher notes, “Most scientists learn the reviewing process by trial and error [...] Perhaps graduate students and postdocs should receive formal training in peer-reviewing manuscripts” (Gough 2009, tr2).[30] Training could include “the intellectual aspects of review, such as how to assess the aims and technical merit of scientific studies [and] training on the practical matters of how to express constructive criticism clearly in writing and [...] ethical dimensions of peer review” (Editors of Nature Chemical Biology 2010).[31] Professional associations (Gasparyan, Ayvazyan and Kitas 2011)[32] and graduate-level university courses (Squazzoni, Bravo and Tákacs 2013)[33] have been suggested as venues for training.

6 Recognition for reviewers

Authors also have suggested improved visibility for existing reviewers. “Some journals publish the reviewers’ names in their journals on an annual basis or send ‘thank you’ letters to reviewers” (Ling 2011, 233)[34]. This approach can include fuller explanation of the work of reviewers: “[...] we will publish a ‘peer-review statement’ that is intended to inform our readers of the rigorous review completed before publication” (Tolo 2014, 1).[35] Better integration of reviewers into the process builds on the social aspects of the process:

“1. We will establish a system by which reviewers are told the outcome of editorial decisions [...] 2. We will publish annually [...] the names of all the reviewers [...] 3. We believe that early-career scientists make excellent reviewers and [...] are therefore going to make a particular effort to more closely engage this community [...] 4. We will encourage reviewers [...] to provide feedback [...].” (Editors and Board of Antarctic Science 2014, 1)[36]

Such steps require no radical departure from current norms.

7 Rewards for reviewers

A more extreme approach proposes rewards beyond simple recognition. More formal status for reviewers could include “symbolic rewards for referees [...] and, more importantly, defining clear rules that link the admission and turnover of peers into their editorial boards also to excellence in reviewing” (Squazzoni, Bravo and Tákacs 2013, 293).[37] Ghazoul (2011)[38] considers rewards based on the quantity and quality of peer reviewer work, including small monetary prizes and reports that could be used in university tenure and promotion processes.

Several organizations have been founded to formalize and improve the situation for referees, while bringing greater efficiency to the process. One example is Peerage of Science, or PoS, which “aims to enhance the quality of reviewing by encouraging non-anonymous review, introducing ‘peer review of peer review’, providing the possibility of reviewers [...] to build a ‘referee factor’” (Hettyey 2012, 189).[39] Under the PoS plan, “authors no longer submit to a journal: instead they send their manuscript into the Peerage of Science peer-review process, which is accessible to editors of all participating journals” (Seppanen et al. 2012, 192).[40] SciRev (2014) is another project that hopes to use the Web to streamline reviewing activity.[41]

The Rubriq system not only hopes to rationalize the process, but adds the incentive of monetary compensation. “Rubriq’s emphasis is on speed. By paying peer reviewers $100 each, it hopes to get reviews back within a week” (Van Noorden 2013, 161).[42]

Paying reviewers may mean billing authors. Harnad explores how such a system might work as part of Open Access, through a “a no-fault refereeing charge for each round of peer review, irrespective of whether the outcome is acceptance or rejection [...]” (Harnad 2010).[43] If accepted widely, a centralized system could benefit authors, editors and publishers:

“[...] a peer review charge (a reasonable one, of the order of perhaps $200 or less per round of review) would not only be affordable, but it might even help further lower the overall expenses of the highly selective journals by discouraging unrealistic submissions [...] the author will get the referee reports and editorial recommendations either way, and if the author is conscientious, these will be helpful in revising for subsequent resubmission [...] No-fault refereeing will also reduce processing time and costs [...] because the submission will already have been refereed and revised [...].” (Harnad, 2010)[44]

Monetization of peer reviewing meets with some mistrust, conflicting as it does with social and community allegiances identified as motives for peer reviewers. One study cautions that “the presence of material incentives might undermine intrinsic pro-social motivations of individuals by transforming reviewing into a self-interest decision problem” (Squazzoni, Bravo and Tákacs 2013, 293).[45]

8 Alternative arenas

The information-sharing capabilities of online media allow new methods for scholarly communication and publishing. At a minimum, this can mean wider sharing of information derived from the traditional process. This could mean posting the text of anonymous peer review reports, or including reviewers’ identities (Henderson 2010).[46]

One nursing journal posts the full train of manuscript submissions, peer reviewer comments, and author revisions, offering a detailed look at the interaction between participants (Dougherty 2010).[47] BioMed Central journals routinely “publish all versions of a manuscript submitted along with peer-review comments and responses from authors” (Couzin-Frankel 2013, 1331).[48] Another medical journal found that “sharing reviewers’ signed reports [...] does not adversely affect the quality of these reports, nor does it improve it by any notable amount. It may, however, reduce the number of willing reviewers and increase the time taken to review” (Van Rooyen, Delamothe and Evans 2010, 1088).[49] While disclosing reviewer identity has not been found to impair results, it does appear to alter some behaviors: “when referees revealed their identities to authors, they provided better reviews, were more courteous, took longer to complete reviews, and were more likely to recommend publication” (Resnik 2011, 27).[50]

An extension of these approaches would open up review to the world at large.

“The ‘wisdom of crowds’ was first noted by the English scientist Francis Galton [...] While Surowiecki acknowledged that not all crowds may be wise, he suggested that if a crowd presents a diversity of independent opinions that draw on individual knowledge, this information may be aggregated into a common understanding of high accuracy. In this manner, a journal could use the collective intelligence of its readers [...].” (Herron 2012, 2278–2279)[51]

Some observers have speculated about the capacity of social media to replace traditional review. “Papers are increasingly being taken apart in blogs, on Twitter and on other social media within hours rather than years” (Mandavilli 2011, 286).[52] This kind of open review is not only rapid, but potentially useful. One study reports that “Tweets can predict highly cited articles within the first 3 days of article publication” (Eysenbach 2011, e123).[53] If social media can assess articles more rapidly than traditional peer review, with a wider distribution of labor and no loss in accuracy, then those media offer an intriguing alternative platform for open review.

9 Open review and Open Access

Given the importance of peer review to validate content, it is not surprising that “open” models are subject to skepticism. Open Access is not equivalent to “open review” despite some shared technologies, but speculation about the future of peer review often makes reference to Open Access, sometimes in a cautionary tone. Anxiety often revolves around the interplay between freely available content, article quality, income for publishers, and sustainability for scholarly journals (Björk and Solomon 2012).[54] In a future without the brand assurance of well-known publishers and journals, there is increased risk of fraud that may go undetected in the peer review and editing process (Henderson 2010).[55]

Several high-profile investigations of the potential for fraud have attracted widespread comment. In a deliberate test, one journalist submitted an obviously defective manuscript to a wide range of journals, triggering a mixed performance by the editorial and peer review system. “Some open-access journals that have been criticized for poor quality control provided the most rigorous peer review of all” (Bohannon 2013, 61).[56] At the same time, “Journals published by Elsevier, Wolters Kluwer, and Sage all accepted my bogus paper” (Bohannon 2013, 65).[57] Inadequate levels of screening and peer review seem to arise independent of the Open Access model, and instead should be of general concern.

10 Conclusion

These lively debates do not indicate that scholars wish to jettison peer review. On the contrary, scholars reaffirm the critical role of peer reviewers in the dissemination of reliable scholarly information. Most critics call for improvements, and often make specific suggestions.

Given the conservatism of academia and the decentralization of scholarly publication practices, peer review seems likely to continue to exist in multiple forms – perhaps even more forms than in the past. It seems unlikely that any single unifying entity, such as Peerage of Science, will be able to influence more than a small share of reviewing transactions. That same decentralization, however, allows experiments as interested parties try out variations. The impact of technology seems unavoidable, now that most scholarly publishing takes place through digital platforms. Thanks to Twitter and other social media, rapid open post-publication review is already taking place, even if it does so in an unofficial manner. It seems likely that conversations about peer review will remain dynamic as participants explore new opportunities with an eye to preserving the valued core of the process.

Published Online: 2015-6-17
Published in Print: 2015-6-22

© 2015 Walter de Gruyter GmbH, Berlin/München/Boston

Downloaded on 28.2.2024 from https://www.degruyter.com/document/doi/10.1515/bfp-2015-0024/html
Scroll to top button