Jodi Cusack, Oxford PharmaGenesis, Oxford, UK; Tim Koder, Oxford PharmaGenesis, Oxford, UK; Sara Schroter, The BMJ, London, UK
Peer review has been described as “a process in which academic research is subjected to critical evaluation by other academics. More precisely, it is the impartial and independent assessment of research by others working in the same or a related field.” [1]
Double-blind peer review, in which neither the authors or reviewers know each other’s identities, is one of the most commonly used models in medical publishing – considered to provide an unbiased review and validation of a manuscript by other experts in the field. For the pharmaceutical industry, peer review is by convention seen as a safe harbor for scientific discussions on new drugs and new indications for drugs (outside of scientific congresses). Any information shared outside of this safe harbor, it is feared, may be viewed as off-label promotion.
There are, however, problems with the peer review process, which can be seen as “…expensive, slow, subjective and biased, open to abuse, patchy at detecting important methodological defects, and almost useless at detecting fraud or misconduct.”[2] These issues are widely discussed by academics, publishers, and funders of research at forums, such as the International Congress on Peer Review and Scientific Publication, which has been encouraging research and discussion on peer review since 1989.
Image reproduced under CCBY license [1]
A 2014 survey by the Nature Publishing Group of over 30,000 of its researchers found that 70% were frustrated about the time taken to complete peer review, and 67% felt that publishers should look into alternative peer review methods.[3]
So, how can the peer review process be improved? A variety of peer review models have been introduced and adopted by journals and other organizations over the past several years – some of which may be more familiar to medical publication professionals, though others perhaps less so.
Single-blind peer review, used by PLOS journals among many others, is when reviewers know the authors’ names, but authors do not know the reviewers’ names. This carries a potential risk that reviewers will be influenced by information known about the author, such as their gender [4] or institution [5], and can also be viewed as unfair, with reviewers able to hide behind a cloak of anonymity while authors are known.
Open peer review, used by The BMJ and BioMed Central journals, is when reviewers and authors know each other’s identities. It was implemented by The BMJ following a trial showing that open peer review had no editorially significant impact on the quality of reviews, recommendation for publication, or time taken compared with blinded review. However, reviewers were more likely to decline to review.[6] Reviewers, especially junior researchers, may be reluctant to be seen to criticize more senior figures in their field. For publications supported by the pharmaceutical industry, reviewers may be wary of being seen as either too critical or too sympathetic.
Publishing the pre-publication history, as seen in The BMJ, takes open peer review a step further by making all peer review reports available alongside the published article. The BMJ implemented this practice to increase transparency and accountability, and encourage more constructive reviews, following a trial that found no important difference in review quality when reviewers were told their signed reviews would be publicly available.[7] Reviewers may, however, take longer to complete their reviews and be more likely to refuse requests to review.
Post-publication peer review, for these purposes defined as peer review after publication without any pre-publication peer review having been carried out, is a form of open peer review, with review reports being posted online and attributed to the reviewers who wrote them. It represents a more formalized extension of the preprints model widely used in other academic fields, in which manuscripts subjected only to quick basic checks are posted online so that data can be shared more rapidly and to enable comment and discussion, prior to any potential submission to a peer-reviewed journal.
In the F1000Research model, which uses post-publication peer review, submitted articles that have passed certain basic checks are published online for immediate viewing and citation, followed by peer review by invited referees. Once a prescribed number of recognized experts have reviewed and approved the article, it is confirmed as peer reviewed and indexed in PubMed and other bibliometric databases.
Wellcome Open Research, which is based on the F1000Research model, had fears about trolling with post-publication peer review, but anecdotal evidence suggests that reviewers are more careful when engaging in open reviews than they are in blinded reviews. On the other hand, considered online reviews may require a suitably considered response from the authors, which could represent an off-putting time commitment for them.
Preprints, while not technically a method of peer review, do provide the opportunity to gather feedback before submitting a manuscript for formal peer review. The BMJ ran a preprint server for several years before withdrawing it due to lack of uptake. However, with the growth of bioRxiv and the planned launch of medRxiv, preprints are becoming increasingly popular.[8]
Preprints are not without their own issues, though. Within medical publishing, there are fears that publishing non-peer reviewed findings may lead to harm to patients.[9] There is also the concern that journals will consider a preprint to be prior publication of the data; though for the majority of journals, this is not the case. In fact, Nature Communications currently highlights preprint versions of any papers under review/revision with their journals, strengthening the relationship between preprints and journal publication.[10]
Rewarded peer review, whether the reward is financial to the reviewer or by just giving credit for the time and effort spent reviewing papers, could improve the speed and quality of peer review.
The idea of payment for peer review may lead to perceptions of conflict of interest, and research suggests that it is time, not money, that is the main reason for potential reviewers declining to review.[11]
With Publons, PeerJ, and F1000Research, review reports have digital object identifiers (DOIs) and can be individually cited and tracked in the reviewer’s publications record (for example, via ORCID), as a means to provide recognition to the reviewer. Crossref, too, has recently extended its infrastructure to manage DOIs for peer reviews.
Patient involvement in peer review is a new development aiming to democratize research to ensure it is acceptable, appropriate, and relevant to patients. It is currently only regularly seen in The BMJ [12] and Research Involvement and Engagement [13] journals, alongside conventional peer review. It will be very interesting to see how many other journals follow their lead.
Results-free peer review, currently offered by BioMed Central journals, aims to reduce selection bias towards publishing positive results by initially only showing reviewers the rationale and methods of a manuscript. If reviewers determine the rationale and methods are relevant, the full paper is then reviewed to ensure that the results and conclusions are in line with the original research question and methodology.
The Registered Reports project set up by the Centre for Open Science has a similar aim as results-free peer review; but, in this case, the study protocol is peer reviewed before any experiments have taken place, and associated manuscripts are provisionally accepted for publication based on the quality of the protocol.
The Future of Peer Review in Medical Publishing
The peer review landscape is complex, and there is no single solution. Publishing companies will continue to develop and trial new models, and the drive for openness will continue.
Open peer review is gathering steam in the open science community, but still faces barriers. Concerns about its feasibility for smaller specialty and society journals are pervasive, and it may be a long time before open peer review is adopted by a broad group of prestigious journals.
Giving credit for peer review may also become more common, as it provides recognition for the time and effort reviewers spend evaluating others’ research for the benefit of the scientific community.
Some groups within medical publishing, especially those involved with industry-funded research, may be slow to respond to these innovations, as this sector is risk-averse and the cost of mistakes can be high.
The way forward may be to explore the risks and benefits of the different peer review models by experimenting with them, using carefully chosen manuscripts reporting suitable types of research.
References
- Hooper M, Bradbury S, Walsh A, Callan P, Steele A, Tate M & Barbour V. (2018, January 19). Journal Peer Review (Version 1). figshare. https://doi.org/10.6084/m9.figshare.5782815.v1
- Godlee F, Jefferson T. Introduction. In Godlee F, Jefferson T, editors. Peer review in health sciences. Second edition. London: BMJ Books, 2003:xiii-xv.
- http://blogs.nature.com/ofschemesandmemes/2015/03/27/further-experiments-in-peer-review (Accessed 09 February 2018)
- https://www.nature.com/news/gender-bias-distorts-peer-review-across-fields-.1.21685 (Accessed 09 February 2018)
- Peters D, Ceci S. Peer-review practices of psychological journals: the fate of submitted articles, submitted again. Behav Brain Sci 1982;5: 187-255.
- Van Rooyen S et al. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 1999;318:23.
- Van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 2010;341:c5729.
- http://www.prepubmed.org/monthly_stats/ (Accessed 14 February 2018)
- http://www.sciencemag.org/news/2017/09/plan-new-medical-preprint-server-receives-mixed-response (Accessed 12 February 2018)
- Nature Communications 8, Article number: 553 (2017)
- Tite L, Schroter S. Why do peer reviewers decline to review? A survey. Journal of Epidemiology & Community Health 2007;61:9-12.
- Richards T, Godlee F. The BMJ’s own patient journey. BMJ 2014;348:g3726.
- Stephens R, Staniszewska S. Research Involvement and Engagement: reflections so far and future directions. Research Involvement and Engagement 2017 3:24.