Peer review underlies the present system of scholarly publication. Peer review itself – the process of scrutinizing and validating research findings prior to dissemination – is itself the topic of numerous research studies, as well as opinion pieces and commentaries by researchers, journal editors, and peer reviewers (themselves almost always researchers in their own right).
The habit of science leads scholars to observe, critique and offer ideas when they think about peer review. The impact of the digital revolution on information-sharing – including publishing and scholarly communication – also promotes reflection and investigation of potential improvements.
There is an extensive literature on peer review. Representative of past summaries of the literature are a two-part article by Campanario in 19981, books by Weller in 20012 and Shatz in 20043, and a white paper by Ware in 20084 (updated by a review article in 2011)5. Interest is extensive in the health sciences, with conferences such as the International Congress on Peer Review and Biomedical Publication (Rennie et al. 2012)6. Scholars in the social sciences and humanities also are exploring the future of peer review. Kathleen Fitzpatrick has shared monographs for pre- and post-publication review through the Media Commons web site (Fitzpatrick 2014)7. An equally interesting part of the literature examines peer review of grant applications and data but falls outside the scope of this essay.
Today’s writing about peer review looks at the motivations of reviewers, the mechanics of the process, the outcomes of review, problems in the system, and solutions that range from minor enhancements to revolutionary changes. The present essay examines what authors, reviewers and editors are saying about peer review and its future.
2 Why take part in peer review?
Peer reviewers devote substantial time to their work, without payment and – at least under the traditional model – without recognition. A variety of formal surveys and informal inquiries have elicited consistent explanations about motivation. Sataloff reports that “By far, ‘playing your part as a member of the academic community’ was the most common reason [...]” (Sataloff 2011, 47)8. Ghazoul (2011)9 and Francis (2013)10 cite the same motives. Simons-Morton also points to the prestige and improved writing skills assumed to accrue to reviewers (Simons-Morton et al. 201211).
This positive attitude is rooted in appreciation of the journal-based scholarly communications system. As researchers and scholars who submit their own manuscripts for examination, peer reviewers understand the need for such a system. A recent survey found that “the vast majority of reviewers believe that reviewing is an important part of being a researcher” (Mulligan, Hall and Raphael 2013, 149)12.
These comments indicate a shared perception that peer review is worthwhile, and is a mark of membership in the scholarly community. The social component often is cited: one study quotes a researcher who says that peer review “‘should be looked at as more of a process, a social activity’” (Couzin-Frankel 2013, 1331)13. Other analysts point to “the reciprocity motives that typically drive human behavior in many social interactions” (Squazzoni, Bravo and Tákacs 2013, 293)14.
3 The mechanics of peer review
While the motives for taking part are generally agreed upon, “peer review” can mean very different things in terms of process. Review supports the journal system as a means “to provide quality certification” (Ellison 2011, 640)15. However, there is no single accepted procedural model for peer review. Among other works, Lee et al. (2013)16 and Mulligan, Hall and Raphael (2013)17 are able to list single-blind and double-blind peer review, open peer review, hybrid peer review, a priori and a posteriori peer review, and post-publication analysis of usage as methods to certify quality and importance. Scholarly opinion is divided about preferences and effectiveness among diverse methods: “Only 25 % of those surveyed preferred the single-blind review process; 56 % preferred double-blind reviews, 13 % favored open review, and 5 % supported post-publication review. [...] Fifty-two percent felt that single-blind review was effective, with 37 % acknowledging post-publication review and 26 % recognizing open review as effective” (Sataloff 2011, 46)18.
The operative activities involved in peer review also are hard to specify. “The peer review process itself is more or less a black box” (Bornmann and Daniel 2010, 11)19. Critics note that there seem to be no standardized rules controlling the journal editors who receive peer reviewer recommendations, and their decisions are private: “One challenge in studying peer review is simply getting the data: closely guarded peer-review reports submitted to biomedical journals, and the conversations that take place inside a journal’s offices” (Couzin-Frankel 2013, 1331)20.
4 Validity and reliability in peer review
The stakes nevertheless are high. Defective peer review can disrupt the process by which manuscripts are certified as worthy of publication and dissemination.
Bornmann, who has written extensively on peer review, sums up the criticism in terms of reliability, fairness, predictive value, inefficiency, and personal stress:
“(1) reviewers rarely agree on whether to recommend that a manuscript be published [...]; (2) reviewers’ recommendations are frequently biased [...]; (3) the process lacks predictive validity because there is little or no relationship between the reviewers’ judgments and [...] the frequency of citations [...] in later scientific papers; (4) reviewing is inefficient because it [...] is time consuming and costly; and (5) reviewing can be [...] painful and distressing [...].” (Bornmann 2011, 203–204)21
Yet, the results of the process are accepted: “from this morass of conflicting advice comes a decision that fairly accurately discriminates high from low quality articles” (Jackson et al. 2011, 6).22 At the same time, suggestions for improvement are widespread.
5 Finding more reviewers
Many scholars wish to expand the pool of reviewers in order to share the workload, and reduce bias. Because the statistical validity of review has been tied to the number of reviewer reports, open review and post-publication review have some appeal: “post-publication open review [...] could potentially improve the quality of reviews, by significantly increasing the number of reviewers and thus improving the statistical validity of a given manuscript’s score” (Herron 2012, 2276).23 This could offset some of the uneven distribution of labor, in which “Active reviewers [...] completed an average of fourteen reviews per year. Hence, active reviewers are responsible for 79 % of all reviews” in one discipline (Sataloff 2011, 46).24
Limited pools of reviewers also can introduce bias, whether inadvertent or intentional. Souder (2011)25 and Lee et al. (2013)26 have compiled long lists of unfair influences including social bias, ad hominem bias, gender bias, affiliational bias, ideological bias, bias based on language or nationality, bias based on resistance to innovative or challenging ideas, bias against interdisciplinary work, and conflict of interest including financial conflict of interest. Among the most subtle factors at work is “Positive outcome bias (POB) [...] the increased likelihood that studies with a favorable or statistically significant outcome will be published than will studies of similar quality that show unfavorable or ‘no-difference’ results” (Emerson et al. 2010, 1934).27 In the worst ethical instances, “a reviewer obstructs the publication of a manuscript because it competes with or questions his or her own work” (Casadevall and Fang 2009, 1274).28
Expanded reviewer pools can promote better representation of the gender, linguistic, ethnic, racial or national differences among global researchers, as noted by one journal: “Nature Chemical Biology is committed to expanding our referee database to ensure that our referee pool reflects the scientific, demographic and geographical diversity of our authors and readers” (Editors of Nature Chemical Biology 2010).29
Where bias is unintended, better training for new reviewers may be useful. As one researcher notes, “Most scientists learn the reviewing process by trial and error [...] Perhaps graduate students and postdocs should receive formal training in peer-reviewing manuscripts” (Gough 2009, tr2).30 Training could include “the intellectual aspects of review, such as how to assess the aims and technical merit of scientific studies [and] training on the practical matters of how to express constructive criticism clearly in writing and [...] ethical dimensions of peer review” (Editors of Nature Chemical Biology 2010).31 Professional associations (Gasparyan, Ayvazyan and Kitas 2011)32 and graduate-level university courses (Squazzoni, Bravo and Tákacs 2013)33 have been suggested as venues for training.
6 Recognition for reviewers
Authors also have suggested improved visibility for existing reviewers. “Some journals publish the reviewers’ names in their journals on an annual basis or send ‘thank you’ letters to reviewers” (Ling 2011, 233)34. This approach can include fuller explanation of the work of reviewers: “[...] we will publish a ‘peer-review statement’ that is intended to inform our readers of the rigorous review completed before publication” (Tolo 2014, 1).35 Better integration of reviewers into the process builds on the social aspects of the process:
“1. We will establish a system by which reviewers are told the outcome of editorial decisions [...] 2. We will publish annually [...] the names of all the reviewers [...] 3. We believe that early-career scientists make excellent reviewers and [...] are therefore going to make a particular effort to more closely engage this community [...] 4. We will encourage reviewers [...] to provide feedback [...].” (Editors and Board of Antarctic Science 2014, 1)36
Such steps require no radical departure from current norms.
7 Rewards for reviewers
A more extreme approach proposes rewards beyond simple recognition. More formal status for reviewers could include “symbolic rewards for referees [...] and, more importantly, defining clear rules that link the admission and turnover of peers into their editorial boards also to excellence in reviewing” (Squazzoni, Bravo and Tákacs 2013, 293).37 Ghazoul (2011)38 considers rewards based on the quantity and quality of peer reviewer work, including small monetary prizes and reports that could be used in university tenure and promotion processes.
Several organizations have been founded to formalize and improve the situation for referees, while bringing greater efficiency to the process. One example is Peerage of Science, or PoS, which “aims to enhance the quality of reviewing by encouraging non-anonymous review, introducing ‘peer review of peer review’, providing the possibility of reviewers [...] to build a ‘referee factor’” (Hettyey 2012, 189).39 Under the PoS plan, “authors no longer submit to a journal: instead they send their manuscript into the Peerage of Science peer-review process, which is accessible to editors of all participating journals” (Seppanen et al. 2012, 192).40 SciRev (2014) is another project that hopes to use the Web to streamline reviewing activity.41
The Rubriq system not only hopes to rationalize the process, but adds the incentive of monetary compensation. “Rubriq’s emphasis is on speed. By paying peer reviewers $100 each, it hopes to get reviews back within a week” (Van Noorden 2013, 161).42
Paying reviewers may mean billing authors. Harnad explores how such a system might work as part of Open Access, through a “a no-fault refereeing charge for each round of peer review, irrespective of whether the outcome is acceptance or rejection [...]” (Harnad 2010).43 If accepted widely, a centralized system could benefit authors, editors and publishers:
“[...] a peer review charge (a reasonable one, of the order of perhaps $200 or less per round of review) would not only be affordable, but it might even help further lower the overall expenses of the highly selective journals by discouraging unrealistic submissions [...] the author will get the referee reports and editorial recommendations either way, and if the author is conscientious, these will be helpful in revising for subsequent resubmission [...] No-fault refereeing will also reduce processing time and costs [...] because the submission will already have been refereed and revised [...].” (Harnad, 2010)44
Monetization of peer reviewing meets with some mistrust, conflicting as it does with social and community allegiances identified as motives for peer reviewers. One study cautions that “the presence of material incentives might undermine intrinsic pro-social motivations of individuals by transforming reviewing into a self-interest decision problem” (Squazzoni, Bravo and Tákacs 2013, 293).45
8 Alternative arenas
The information-sharing capabilities of online media allow new methods for scholarly communication and publishing. At a minimum, this can mean wider sharing of information derived from the traditional process. This could mean posting the text of anonymous peer review reports, or including reviewers’ identities (Henderson 2010).46
One nursing journal posts the full train of manuscript submissions, peer reviewer comments, and author revisions, offering a detailed look at the interaction between participants (Dougherty 2010).47 BioMed Central journals routinely “publish all versions of a manuscript submitted along with peer-review comments and responses from authors” (Couzin-Frankel 2013, 1331).48 Another medical journal found that “sharing reviewers’ signed reports [...] does not adversely affect the quality of these reports, nor does it improve it by any notable amount. It may, however, reduce the number of willing reviewers and increase the time taken to review” (Van Rooyen, Delamothe and Evans 2010, 1088).49 While disclosing reviewer identity has not been found to impair results, it does appear to alter some behaviors: “when referees revealed their identities to authors, they provided better reviews, were more courteous, took longer to complete reviews, and were more likely to recommend publication” (Resnik 2011, 27).50
An extension of these approaches would open up review to the world at large.
“The ‘wisdom of crowds’ was first noted by the English scientist Francis Galton [...] While Surowiecki acknowledged that not all crowds may be wise, he suggested that if a crowd presents a diversity of independent opinions that draw on individual knowledge, this information may be aggregated into a common understanding of high accuracy. In this manner, a journal could use the collective intelligence of its readers [...].” (Herron 2012, 2278–2279)51
Some observers have speculated about the capacity of social media to replace traditional review. “Papers are increasingly being taken apart in blogs, on Twitter and on other social media within hours rather than years” (Mandavilli 2011, 286).52 This kind of open review is not only rapid, but potentially useful. One study reports that “Tweets can predict highly cited articles within the first 3 days of article publication” (Eysenbach 2011, e123).53 If social media can assess articles more rapidly than traditional peer review, with a wider distribution of labor and no loss in accuracy, then those media offer an intriguing alternative platform for open review.
9 Open review and Open Access
Given the importance of peer review to validate content, it is not surprising that “open” models are subject to skepticism. Open Access is not equivalent to “open review” despite some shared technologies, but speculation about the future of peer review often makes reference to Open Access, sometimes in a cautionary tone. Anxiety often revolves around the interplay between freely available content, article quality, income for publishers, and sustainability for scholarly journals (Björk and Solomon 2012).54 In a future without the brand assurance of well-known publishers and journals, there is increased risk of fraud that may go undetected in the peer review and editing process (Henderson 2010).55
Several high-profile investigations of the potential for fraud have attracted widespread comment. In a deliberate test, one journalist submitted an obviously defective manuscript to a wide range of journals, triggering a mixed performance by the editorial and peer review system. “Some open-access journals that have been criticized for poor quality control provided the most rigorous peer review of all” (Bohannon 2013, 61).56 At the same time, “Journals published by Elsevier, Wolters Kluwer, and Sage all accepted my bogus paper” (Bohannon 2013, 65).57 Inadequate levels of screening and peer review seem to arise independent of the Open Access model, and instead should be of general concern.
These lively debates do not indicate that scholars wish to jettison peer review. On the contrary, scholars reaffirm the critical role of peer reviewers in the dissemination of reliable scholarly information. Most critics call for improvements, and often make specific suggestions.
Given the conservatism of academia and the decentralization of scholarly publication practices, peer review seems likely to continue to exist in multiple forms – perhaps even more forms than in the past. It seems unlikely that any single unifying entity, such as Peerage of Science, will be able to influence more than a small share of reviewing transactions. That same decentralization, however, allows experiments as interested parties try out variations. The impact of technology seems unavoidable, now that most scholarly publishing takes place through digital platforms. Thanks to Twitter and other social media, rapid open post-publication review is already taking place, even if it does so in an unofficial manner. It seems likely that conversations about peer review will remain dynamic as participants explore new opportunities with an eye to preserving the valued core of the process.
Ware, M.: Peer Review: Benefits, Perceptions and Alternatives. In: PRC Summary Papers 4. London: Publishing Research Consortium 2008. http://www.publishingresearch.org.uk/documents/PRCsummary4Warefinal.pdf (last downloaded 2015-01-10).
Fitzpatrick, K.: MediaCommons Press: Open Scholarship in Open Formats. 2014. http://mcpress.media-commons.org/ (last downloaded 2015-01-10).
Mulligan, A.; Hall, L.; Raphael, E.: Peer Review in a Changing World: An International Study Measuring the Attitudes of Researchers. In: Journal of the American Society for Information Science and Technology 64 (1) (2013) pp. 132–161.
Bornmann, L.; Daniel, H.-D.: The Manuscript Reviewing Process: Empirical Research on Review Requests, Review Sequences, and Decision Rules in Peer Review. In: Library & Information Science Research 32 (1) (2010) pp. 5–12.
Herron, D. M.: Is Expert Peer Review Obsolete? A Model Suggests that Post-publication Reader Review May Exceed the Accuracy of Traditional Peer Review. In: Surgical Endoscopy 26 (8) (2012) pp. 2275–2280.
Emerson, G. B.; Warme, W. J.; Wolf, F. M.; Heckman, J. D.; Brand, R. A.; Leopold, S. S.: Testing for the Presence of Positive-outcome Bias in Peer Review: A Randomized Control Trial. In: Archives of Internal Medicine 170 (21) (2010) pp. 1934–1939.
Editors of Nature Chemical Biology: Editorial: Expanding Peer Review. In: Nature Chemical Biology 6 (5) (2010) p. 307. http://www.nature.com/nchembio/journal/v6/n5/full/nchembio.366.html (last downloaded 2015-01-10).
SciRev.: About SciRev – Background. 2014. http://scirev.sc/about/background/ (last downloaded 2015-01-10).
Harnad, S.: No-fault Peer Review Charges: The Price of Selectivity Need Not Be Access Denied or Delayed. In: D-Lib Magazine 16 (7/8) (2010), http://www.dlib.org/dlib/july10/harnad/07harnad.html (last downloaded 2015-01-10).
Van Rooyen, S.; Delamothe, T.; Evans, S. J. W.: Effect on Peer Review of Telling Reviewers that Their Signed Reviews Might Be Posted on the Web: Randomized Controlled Trial. In: BMJ 341 (7782) (2010) p. 1088.
Eysenbach, G.: Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact. In: Journal of Medical Internet Research 13 (4) (2011) p. e123.
Björk, B.-C.; Solomon, D.: Open Access versus Subscription Journals: A Comparison of Scientific Impact. In: BMC Medicine 10 (2012) p. 73. http://www.biomedcentral.com/1741-7015/10/73 (last downloaded 2015-01-10).
About the article
Published Online: 2015-06-17
Published in Print: 2015-06-22