Review
Reproducible research practices are underused in systematic reviews of biomedical interventions

https://doi.org/10.1016/j.jclinepi.2017.10.017Get rights and content

Abstract

Objectives

To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research.

Study Design and Setting

We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

Results

We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses.

Conclusion

Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

Introduction

Biomedical researchers are increasingly encouraged to use reproducible research practices, which allow others to recreate the findings of studies, given the original data [1], [2], [3]. Such practices include providing a detailed description of the data collected and used for analysis, along with descriptive metadata, clearly reporting the analysis methods and results, and sharing the data set and statistical code used to perform analyses [1], [4]. There are many benefits of performing such practices in the context of systematic reviews (SRs) of studies. For example, users can check for possible data-entry errors when summary statistics for each included study are reported in sufficient detail. Transparent reporting of meta-analyses also makes it possible for others to reanalyze published meta-analyses using different inclusion criteria or statistical methods or to perform additional analyses that address secondary research questions [5]. For example, readers may reanalyze a published meta-analysis by restricting it to the subset of studies conducted in the setting where they work. In addition, sharing of data sets and statistical analysis code allows other researchers to cumulatively add new data that are published, thus keeping meta-analytic effect estimates up-to-date [6], [7].

The limited data on use of reproducible research practices in SRs come from studies that have recorded how well SRs adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The PRISMA statement includes an item recommending that for all outcomes considered, systematic reviewers report, for each study, “simple summary data for each intervention group and effect estimates and confidence intervals” [8]. However, not all studies evaluating PRISMA adherence have provided data on adherence to this item, opting to present a total score summed across all PRISMA items instead [9]. Furthermore, many studies that have identified low adherence to this item have assessed SRs in a single clinical specialty (e.g., [10], [11], [12], [13], [14]), which limits generalizability of the findings. To our knowledge, no study has quantified how often systematic reviewers report the data needed to recreate all meta-analytic effect estimates in an SR (including subgroup meta-analytic effects and sensitivity analyses) nor investigated whether completeness of reporting varies by type of outcome (i.e., primary or other).

Efforts to increase transparent reporting of SR articles have existed for many years (e.g., the PRISMA statement was disseminated in top medical journals in 2009); however, little attention has been given to the sharing of data collected as part of SRs [15]. For example, since 2015, the BMJ encourages authors of all research articles to link their articles to the raw data from their studies but requires data sharing on request as a minimum for clinical trials only [16]. No study has investigated how often sharing of data sets and statistical analysis code is done by authors of SRs.

We investigated how often research practices that facilitate reproducibility of analyses were used in a cross-sectional sample of SRs of therapeutic interventions. We also explored whether the use of such reproducible research practices was associated with whether an SR was a Cochrane review and with the systematic reviewers' reported use of the PRISMA statement.

Section snippets

Methods

We conducted this project in accordance with a study protocol, which is available on the Open Science Framework (RRID:SCR_003238): https://osf.io/523bq/. This study was conducted concurrently with another project evaluating the application and interpretation of statistical methods in SRs. The results of the other project will be described elsewhere.

General characteristics of SRs

We evaluated 110 SRs of therapeutic interventions. Nearly all (97/110 [88%]) were published in late 2013, and 68/110 (62%) were led by systematic reviewers based in China, UK, Canada, or USA (Table 1). Most SRs (78/110 [71%]) were not Cochrane reviews. The SRs focused on interventions for one of 19 different conditions, with diseases of the digestive system, diseases of the circulatory system, infectious and parasitic diseases, and neoplasms being the most common. The interventions studied were

Discussion

The use of reproducible research practices in SRs of therapeutic interventions was suboptimal in our sample. Systematic reviewers reported the data needed to recreate all meta-analytic effect estimates in the SR, including subgroup meta-analytic effects and sensitivity analyses, in only 65% of SRs. This percentage was higher in Cochrane than in non-Cochrane SRs (94% vs. 54%). In contrast, the data needed to recreate the index (primary or first reported) meta-analysis were available in nearly

Conclusions

Reproducible research practices in SRs of therapeutic interventions are suboptimal. Strategies are needed to facilitate the provision of detailed descriptions of data gathered and data used for analysis, transparent reporting of the analysis method and results, and sharing of data sets and statistical analysis code so that others can recreate the findings or perform secondary analyses.

Acknowledgments

The authors thank Sean Harrison (University of Bristol) for assistance with data analysis.

Authors' contributions: All authors declare to meet the ICMJE conditions for authorship. M.J.P., D.G.A., and D.M. conceived the study design. L.S. and J.E.M. provided input into the study design. M.J.P., D.G.A., L.S., J.E.M., and D.M. selected items for inclusion in the data-collection form. M.J.P., N.A., D.W., and F.Y. collected data. M.J.P. undertook the statistical analyses. M.J.P. wrote the first draft

References (39)

  • D. Moher et al.

    Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

    PLoS Med

    (2009)
  • A. Stevens et al.

    Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review

    BMJ

    (2014)
  • W.W. Tam et al.

    Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study

    BMJ Open

    (2017)
  • J.P. Peters et al.

    Reporting quality of systematic reviews and meta-analyses of otorhinolaryngologic articles based on the PRISMA statement

    PLoS One

    (2015)
  • Y. Liu et al.

    Reporting quality of systematic reviews/meta-analyses of acupuncture

    PLoS One

    (2014)
  • A.S. Tunis et al.

    Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?

    Radiology

    (2013)
  • J.J. Gagnier et al.

    Reporting and methodological quality of systematic reviews in the orthopaedic literature

    J Bone Joint Surg Am

    (2013)
  • L. Wolfenden et al.

    Time to consider sharing data extracted from trials included in systematic reviews

    Syst Rev

    (2016)
  • E. Loder et al.

    The BMJ requires data sharing on request for all trials

    BMJ

    (2015)
  • Cited by (0)

    Funding: There was no direct funding for this study. M.J.P. is supported by an Australian National Health and Medical Research Council (NHMRC) Early Career Fellowship (1088535). J.E.M. is supported by an NHMRC Australian Public Health Fellowship (1072366). F.C.-L. is supported by the Generalitat Valenciana (PROMETEOII/2015/021). A.C.T. is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. D.M. is supported in part by a University Research Chair, University of Ottawa. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the article.

    Conflicts of interest: The authors have read the journal's policy and the authors of this article have the following competing interests: A.C.T. is an associate editor for Journal of Clinical Epidemiology but had no involvement in the peer review process or decision for publication. M.J.P. and J.E.M. are affiliates of Cochrane Australia. M.J.P. is a co-convenor of the Cochrane Bias Methods Group. J.E.M. is a co-convenor of the Cochrane Statistical Methods Group. A.C.T. is an author of two of the systematic reviews included in this study but was not involved in eligibility assessment or data collection. D.G.A. is a senior investigator of National Institute for Health Research.

    Data availability: The study protocol, data-collection form, and the raw data and statistical analysis code for this study are available on the Open Science Framework: https://osf.io/523bq/.

    View full text