For Peer review week 2019 Sense about Science have asked for a series of blogposts, to begin a Research ‘TO DO’ list. What can researchers, universities, funders and governments do to improve the quality of peer review and research? This is just the start, we need your ideas too.
Imagine a world where society has the high-quality evidence it needs to make informed
decisions about crime, health and education. What action can researchers, universities, funders and governments take to get us there?
How can we improve the quality of peer reviewed publications?
By Professor Gary Collins @GSCollins and Patricia Logullo @patlogullo on behalf of the EQUATOR Network @EQUATORNetwork
It has been estimated that more than 80% of
articles in biomedical journals lack important information. They are published
without the details that would be necessary for their findings to help
researchers and health professionals improve people’s lives. Because publications
are frequently incomplete or biased, this wastes money and human resources and can
lead to patient harm. These are the findings of key systematic reviews on
studies published in the last 20 years.
Since the early 1990s, researchers and journal
editors have tried to tackle this problem by creating reporting guidelines:
these are tools that remind authors to provide a minimum list of information
needed to ensure that:
– the article can be understood;
– the study can be replicated by another
– the results can be used by health professionals
– the study can be included in a systematic review
(to inform guidelines for example).
Research has shown that the quality of scientific
reporting has improved modestly with the increased use of reporting guidelines.
However, there is still a long way to go.
Another source of avoidable harm to patients is
when the studies are not reported at all: non publication of research (also
called publication bias) is a known cause of research waste. Since 2005, the
International Committee of Medical Journal Editors (ICMJE) has mandated prior
registration of trial protocols before clinical trials findings are published, Clinical
trial registration makes it easier to identify those studies that have completed
but not yet published.
The EQUATOR Network (www.equator-network.org) is a
global initiative dedicated to improving the quality and transparency of health
research. We conduct research on reporting, manage a collection of more than
400 reporting guidelines (which are currently being audited), and train authors
and editors how to use reporting guidelines. Our research experience allows us
to draft the following “to-do list” about improving reporting of health care
How to fix the problem of poor-quality reporting
Based on the EQUATOR Network experience of
conducting and evaluating research-on-research, and helping authors, editors
and librarians, these are the activities, procedures, approaches or policies
thatto improve the completeness of research reporting. Some of these issues are
currently being addressed by organisations and researchers, some might not be.
Researchers (research-on-research investigators) TO
1. Updating reporting guidelines – The
first reporting guideline, the CONSORT Statement, designed to guide the
reporting of clinical trials, was published in 1996. Since then, numerous reporting
guidelines have been developed for other study designs, including observational
studies (STROBE), systematic reviews (PRISMA), diagnostic test accuracy (STARD),
prognostic model studies (TRIPOD) studies and many more. CONSORT was updated in
2001 and 2010. However, many reporting guidelines have been available for a
long time now without having been updated. Periodic updating and revising is important
to ensuring their ongoing credibility, reflect advancements made in study
methodology and hopefully increase their use by researchers when writing up the
findings from their study.
2. Investigating poor adherence – Many
reporting guidelines have been published, but evaluations have shown they are
often not fully adhered to, with many key details often omitted — even in reports published in high ranking
journals. We therefore need to investigate and understand why researchers are
not fully following the recommendations contained in reporting guidelines or do
it poorly (incompletely), and why journals are allowing that to happen.
3. Testing new interventions — What could help
authors use reporting guidelines? What would be alternative approaches to
improving reporting quality? Does training help? We should test them as
interventions, in controlled and carefully planned trials.
Publishers and journal editors TO DO list:
1. Knowing reporting guidelines well — Editors
working for journals that endorse or claim to enforce the use of reporting
guidelines are not always sufficiently familiar with the contents of all the
main reporting guidelines and the corresponding checklists. This creates the
situation where the editors can ask authors to fill in the wrong checklist as a
mandatory step of submission. Training editors on the key reporting guidelines
is important to understand whether the relevant guideline and checklist has
been correctly followed and completed.
2. Standardise the instructions for authors —
The journals’ instructions for authors vary substantially, even between
journals that share the same publisher: this means they have quite different
expectations of how authors should write and format their papers. This
variation persists even in journals that have adopted ICMJE
(International Committee of Medical Journal Editors) recommendations. In order
to achieve greater consistency and quality, journals should consistently endorse
the key reporting guidelines (namely those listed on the EQUATOR homepage).
3. Check usage — Many journals endorse or
recommend following reporting guidelines, expecting authors to adhere to them
and submit a completed checklist (providing page numbers where key information
are reported in the submitted article). However, journals seldom check if
reporting guidelines were followed, or indeed if the completed checklist is
accurate. Effective solutions are required during the submission and peer
review process to identify key information, recommended in the reporting
guideline, that has not been reported before acceptance of the article.
4. Increase peer reviewers awareness — Ideally,
peer reviewers should be aware of the relevant reporting guidelines for the
study design of the article they are reviewing and consult them during the peer
review. Peer reviewers have the opportunity to help to improve reporting, by requesting
that authors provide details on aspects of the study that are missing.
5. Create the citizen box — Journals could start
requiring authors to complete a small paragraph of the manuscripts with
information directly relevant to patients and citizens in general, about the
importance of the study. This could make manuscripts more understandable to the
Systematic reviewers TO DO list:
1. Provide risk of bias feedback — One of
the daily tasks of systematic reviewers is to evaluate the risk of bias of
studies. For that, reviewers need detailed information about how the study was
done and what it found. When there is “risk of bias due to poor reporting”, we
don’t know what happened next. We don’t know what information was asked from
authors, because little is reported about this communication between reviewers
and triallists. Also, there should be a way to publish information that was
collected through direct contact with the authors.
2. Improve reporting — While systematic
reviews suffer from a lack of information in the papers they review, not all
systematic reviews are well reported, understandable by everybody or useful. Therefore,
there is room for improvement in the quality of the systematic reviews reports.
This goes from reporting a minimum set of information following the PRISMA
reporting guideline to help describe the included studies, to improving the
quality of language and format. This is critical for the abstracts and plain
language summaries that are frequently used by policymakers and patients.
Government, policymakers and funders TO-DO list:
1. Get to know and require the minimum sets
— Funders should mandate the use of reporting guidelines, so that policymakers
can make informed decisions on whether or not to adopt a new treatment. This
means they need information on harms (an item on several reporting guidelines),
feasibility of implementation (providing sufficient detail on the interventions,
e.g. by following the TiDIER reporting guideline) and cost effectiveness.
2. Examine protocol methods using reporting
guidelines — Funders routinely evaluate applications for funding. These do not
always provide all the important details on how authors intend to conduct the
research (where reporting guidelines such as SPIRIT and PRISMA-P for clinical
trials and systematic reviews can help). If funders start to require adherence
to reporting guidelines, at least for the methods section (which can then be
readily evaluated during peer review), this might help make better informed
decisions on whether to fund studies.
Patients TO-DO list
1. Involve! – Patients should be actively involved
in research. This is increasingly happening in Europe, but still in clinical
areas only. Patients could also be encouraged to participate in
research-on-research projects. We need to know their view on what is essential
or important to be reported in a manuscript.
2. Review! –Patients and members of the
public could and should participate in the revision process of new reporting
guidelines. Research is done with them and for them. So why not ask them to
verify the draft items of reporting guidelines checklists, even when they did
not participate in their building?
extend our thanks to David Tovey @DavidTovey for
his work editing these blogposts.
Please tweet your ideas #ResearchTODOlist #PeerRevWk19 and #QualityInPeerReview or email us at [email protected].