Systematic Review
A low proportion of systematic reviews in physical therapy are registered: a survey of 150 published systematic reviews

https://doi.org/10.1016/j.bjpt.2017.09.009Get rights and content

Highlights

  • A low proportion of systematic reviews (SRs) in the physical therapy field are registered.

  • More than one third of registered SRs showed discrepancies between protocol and publication.

  • Strategies to encourage prospective registration of SRs in the physical therapy field are needed.

Abstract

Background

Systematic reviews provide the best evidence about the effectiveness of healthcare interventions. Although systematic reviews are conducted with explicit and transparent methods, discrepancies might occur between the protocol and the publication.

Objectives

To estimate the proportion of systematic reviews of physical therapy interventions that are registered, the methodological quality of (un)registered systematic reviews and the prevalence of outcome reporting bias in registered systematic reviews.

Methods

A random sample of 150 systematic reviews published in 2015 indexed on the PEDro database. We included systematic reviews written in English, Italian, Portuguese and Spanish. A checklist for assessing the methodological quality of systematic reviews tool was used. Relative risk was calculated to explore the association between meta-analysis results and the changes in the outcomes.

Results

Twenty-nine (19%) systematic reviews were registered. Funding and publication in a journal with an impact factor higher than 5.0 were associated with registration. Registered systematic reviews demonstrated significantly higher methodological quality (median = 8) than unregistered systematic reviews (median = 5). Nine (31%) registered systematic reviews demonstrated discrepancies between protocol and publication with no evidence that such discrepancies were applied to favor the statistical significance of the intervention (RR = 1.16; 95% CI: 0.63–2.12).

Conclusion

A low proportion of systematic reviews in the physical therapy field are registered. The registered systematic reviews showed high methodological quality without evidence of outcome reporting bias. Further strategies should be implemented to encourage registration.

Introduction

Systematic reviews (SRs) provide the best evidence to contribute to decision-making about the implementation of healthcare interventions.1 Although these studies are conducted with explicit and transparent methods, discrepancies might occur between the protocol and the publication. For example, authors might adapt the methods so that the SR generates more positive and statistically significant results, especially because there is a tendency for some scientific journals to preferentially publish manuscripts with statistically significant results.2 This may affect the validity of the results by introducing bias, such as outcome reporting bias.3, 4, 5 Outcome reporting bias is defined as the selective reporting from a subset of original outcomes, based on results.3 One of the strategies suggested to reduce this bias is the prospective registration of protocols for SRs.6

Protocol registration has been increasingly recommended for clinical trials7 and SRs.8 A registry for protocols of SRs was first proposed by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement in 2009,8 which resulted in subsequent development and implementation of the International Prospective Register of Systematic Reviews (PROSPERO). A protocol provides transparency and makes explicit the hypotheses, methods and analysis of the SR that is to be conducted. According to the Cochrane Handbook for Systematic Reviews of Interventions,1 a prospectively registered protocol reduces authors’ biases by publicly documenting the a priori planned methods. When necessary, changes may occur between protocol and publication. However, any changes should be: decided upon without calculating their effect on the results, applied as an amendment to the registered protocol at the time of the decision, and reported with explanation in the manuscript.1

A previous study has demonstrated that nearly one-third of a sample of the SRs registered on PROSPERO show discrepancies between the primary outcomes registered in the protocol and the primary outcome reported in the publication.9 Other studies in several specific fields have also revealed discrepancies between protocols and published SRs.10, 11, 12 As the prevalence of discrepancies differs between different fields of research, it is important to assess this issue in other disciplines, such as physical therapy. Discipline-specific data may also indicate which strategies might be most beneficial to control these discrepancies.

Launched in 1999, the Physiotherapy Evidence Database (PEDro) indexes published practice guidelines, SRs and randomized controlled trials to support an evidence-based approach in physical therapy. PEDro is one of the most complete databases for physical therapy publications.13, 14 Pinto et al.15 analyzed 200 randomized controlled trials sampled from PEDro and identified that many: were not prospectively registered; were not registered at all; and/or had discrepancies between the registered protocol and the published report.15 However, a search of GoogleScholar using the search terms regist-, systematic review, and physiotherapy or physical therapy did not identify any studies addressing the extent of registration of SRs in physical therapy. Therefore, the primary aims of the study were: (a) to estimate the proportion of SRs of physical therapy interventions that are registered, (b) to assess the methodological quality of (un)registered SRs of physical therapy interventions, and (c) to investigate whether outcome reporting bias is present in those SRs that have a registered protocol. As a secondary aim, we explored whether registration is associated with characteristics of SRs, including the geographical location of the authors, the impact factor of the journal, funding, and spin.

Section snippets

Methods

This study was a survey of SRs of physical therapy interventions. PEDro was used as the source of the SRs because it is considered one of the most complete database of SRs of physical therapy interventions.13 From the total sample of SRs indexed in 2015, we randomly selected 150 reports using a random number function in Microsoft Excel software. The full texts were restricted to publications written in English, Italian, Portuguese and Spanish. The full-text published report of each SR was

Results

At the time this study was conducted (September, 2016), PEDro indexed 646 SRs that had been published in 2015. We selected a random sample of 150 published SRs, which equated to 23% of the SRs. Cochrane reviews accounted for 15 of the 150 reviews. The prevalence of registration among the SRs was 19% (29 SRs). The characteristics of the whole sample and of the registered and unregistered SRs specifically are presented in Table 1.

Fig. 2 describes the association between the SRs’ characteristics

Discussion

This survey found a low prevalence of registration of SRs in the physical therapy field, with only one fifth being registered. Registration was more likely among funded SRs and among SRs published in journals with a high impact factor. Importantly, registered SRs had significantly higher quality compared to unregistered SRs. Our findings also indicate that although a third of the registered SRs demonstrated discrepancies between protocol and publication, no evidence of outcome reporting bias

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors declare no conflicts of interest.

Acknowledgements

C.B.O., I.R.L., D.O.S. and R.V.B. are funded by São Paulo Research Foundation (FAPESP) [grant numbers 2016/03826-5, 2015/17777-3, 2015/11534-1, 2015/00406-2].

References (33)

  • A.C. Tricco et al.

    A third of systematic reviews changed or did not specify the primary outcome: a PROSPERO register study

    J Clin Epidemiol

    (2016)
  • M.J. Page et al.

    Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions

    Cochrane Database Syst Rev

    (2014)
  • N. Pandis et al.

    Discrepancies in outcome reporting exist between protocols and published oral health cochrane systematic reviews

    PLoS ONE

    (2015)
  • B. Tendal et al.

    Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

    BMJ

    (2009)
  • Z.A. Michaleff et al.

    CENTRAL, PEDro, PubMed, and EMBASE are the most comprehensive databases indexing randomized controlled trials of physical therapy interventions

    Phys Ther

    (2011)
  • R.Z. Pinto et al.

    Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials

    Phys Ther

    (2013)
  • Cited by (18)

    • Aerobic training reduces blood pressure and waist circumference and increases HDL-c in metabolic syndrome: a systematic review and meta-analysis of randomized controlled trials

      2018, Journal of the American Society of Hypertension
      Citation Excerpt :

      The protocol for this systematic review was previously registered in an international database of systematic reviews in health and social care (registration number CRD42016033862). Protocol registration provides explicit hypotheses, methods and analysis of the systematic review.23 In addition, it reduces authors’ biases by publicly documenting the planned methods.24

    • Updating systematic reviews

      2018, Journal of Physiotherapy
    • Overall confidence in the results of systematic reviews on exercise therapy for chronic low back pain: a cross-sectional analysis using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) 2 tool

      2020, Brazilian Journal of Physical Therapy
      Citation Excerpt :

      In contrast to our findings, the study by Martins et al.75 reported that, only 22.5% of the reviews were rated as ‘very poor’. Oliveira et al.76 evaluated the methodological quality of systematic reviews of physical therapy interventions using the AMSTAR tool, most of the reviews were rated as ‘low’ (median 3, 0–11 scale) for the orthopaedics PEDro subdiscipline. The three systematic reviews in which the overall confidence in the results were rated as ‘High” in our study evaluated the effectiveness of physical conditioning exercises,64 Yoga11 and Pilates10 in LBP patients.

    View all citing articles on Scopus
    View full text