Restoring confidence in science – tinkering in the margin is not enough

Piero Olliaro, Professor of Poverty-related Infectious Diseases, Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK
 
Josephine Bourner, Clinical Trial Manager, Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK
 
Lakshmi Manoharan, Research Assistant (Medical Epidemiologist), Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK
 
The COVID-19 pandemic has laid bare long-standing, fundamental flaws in the way knowledge is generated, shared, and used. Traditionally, the central step is publication in scientific journals. However, this system is collapsing under astronomical numbers of articles and volumes of information, unable to cope with the conflicting needs for speed and quality of information; the question now is how to achieve a decent compromise between these contradictory wants.
 
The recent Peer Review Week, “the annual celebration of the importance of peer review”, implies we have the solution. We know, however, the system is far from watertight, but band-aiding the existing state of affairs in not the solution. We need a system reset, not tinkering in the margin. Celebrating the peer-review model as-is, without addressing its long-known shortcomings, is short-sighted and inadequate. The system was already overstretched and imperfect – the peer review process is often slow, opaque, unaccountable, biased and sometimes unprofessional; its deficiencies have been amplified by the overflow of COVID-19 papers and the political and economic interests around the pandemic, calling into question whether research publication is meant to inform public health decisions, or to make money instead.
 
How do editors decide if a manuscript is worth their attention? This decision is made more challenging with the avalanche of submitted manuscripts – rejection rates inevitably shoot up, and much potentially valuable research will remain unknown, while unworthy papers are also published. This raises again questions about the motivations and criteria applied to the selection of certain papers over others, and the level of scrutiny to which they are subjected.
 
The publication and subsequent retractions of the Surgisphere Corporation data in the Lancet and New England Journal of Medicine are the highest profile examples from this pandemic; international trials of hydroxychloroquine were halted and national COVID-19 treatment guidelines were altered based on this paper. It was only after the wider scientific community vocalised its concerns and the findings were widely publicised in the media that the database upon which the research was based on was questioned and subsequently discredited. With such editorial decisions being often made in haste during this pandemic, retractions might be too little, too late, after damage is done, especially when from high-ranking journals which have an inordinate influence and can sway case management and policy decisions.
 
Publishing – whether peer-reviewed or not – is just one step in the knowledge generation-sharing-use process; we need a rethink of the system as a whole. There has been plenty of innovation during this pandemic as publishers negotiate the need for timely information, while recognising the deficiency of human resources to review a proliferation of COVID-19 research. This has led to an expansion of pre-print servers, blogs and social media to disseminate new information as efficiently as possible, avoiding or ahead of peer-review. An alternative response by some publishers has been to mobilise early-career researchers to provide peer-review and extending review and resubmission deadlines.
 
Yet none of these responses are entirely satisfactory. One response shortcuts peer-review – potentially leading to harmful clinical and public health consequences – and the other compromises the need for efficiency and timeliness.
 
Fundamentally, the decision as to which piece of new information is worth dissemination should be based on its quality, novelty, and overall contribution to knowledge. In reality, editorial decisions are the result of a largely opaque process. To ensure greater transparency and accountability, journals should consider open peer-review and publication of their review processes and metrics. Additionally, greater transparency of journals’ peer review processes may encourage the necessary innovation and research required to determine optimal review methods. Accountability can be strengthened through the development of a code of ethics and professional standards – but this must have widespread buy-in from journals and reviewers alike.
 
While these are long-term investments and are perhaps not helpful during the current pandemic, or impactful in the short-term, the best time to invest in and re-think peer-review is right now. Worth considering is a model adopted by f1000Research, which prioritises rapid publication using a combination of post-publication open peer-review and user commenting. This model addresses the current need for rapid information dissemination, transparency and the human resource crisis facing peer-review. The model also encourages important public review from the wider scientific community, which is equally as valuable.
 
With the release of crucial vaccines on the horizon, we are at a critical stage in the pandemic. So much public trust has been lost in science – the anti-vax movement in particular has gained steam and there has been resistance to national COVID-19 scientist-advised control directives issued by governments. So much is at stake for the reputation of science and the public’s trust in decisions arising from our work.
 
The weaknesses of peer-review are clear. It is now time to focus on tangible improvements – making transparency our top priority. This will allow publishing to once again become a well-thought-of, integral part of the whole knowledge generation-sharing-use system in not only the eyes of the scientific community, but also in the eyes of the public.

Reply