Review ArticleStrategies to improve the credibility of meta-analyses in spine surgery: a systematic survey
Introduction
Meta-analyses are systematic reviews in which statistical tests combine the results from similar studies to produce best estimates of the underlying treatment effects [1]. Pooling multiple studies increases sample size, increasing both the accuracy and precision of the results, and provides guidance for clinical care. Meta-analyses also provide unique opportunities to evaluate differences between studies and detect publication bias and to direct future investigations by identifying knowledge gaps.
Publication rates of spine surgery meta-analyses have increased by approximately fivefold over the last 15 years, but the methodological quality, also known methodological credibility, with which they are being conducted has not been evaluated [2], [3]. Along with transparent reporting, high methodological credibility is necessary to avoid misleading conclusions. Flawed meta-analyses are at risk of compromising clinical decision making, and limitations have been documented in a variety of other surgical and medical specialties [3], [4], [5], [7].
Clinicians, researchers, and other evidence users may evaluate the credibility of a meta-analysis by applying the Users’ Guide to the Medical Literature [1], [8]. According to the Users’ Guide, the credibility of a meta-analysis depends on the extent to which it addresses a sensible clinical question, includes an exhaustive literature search, addresses possible explanations of between-studies differences, presents results in a clinically useful manner, reproducibly selects and assesses primary studies, and addresses confidence in the pooled effect estimates. Credibility is conceptually distinct from completeness of reporting, which describes the extent to which authors comprehensively report the items necessary for users to critically appraise strengths and weaknesses. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist is a 27-item guide that most major surgical and medical journals have adopted by for this purpose [9].
Our primary objectives were to evaluate the credibility of spine surgery meta-analyses and to propose strategies to improve future research. Our secondary objectives were to evaluate completeness of reporting and identify factors associated with higher credibility and completeness of reporting.
Section snippets
Eligibility criteria
We performed a systematic survey of all meta-analyses of therapeutic interventions related to spine surgery published since 1990. We defined meta-analyses as any summaries of research that attempted to address a focused clinical question in a systematic and reproducible manner and included a quantitative synthesis to yield a best estimate of treatment effect [1], [8]. We excluded systematic reviews that summarized the available literature but did not include quantitative syntheses of results
Results
Our results are summarized in Table 1, Table 2, Table 3 and Fig. 1, Fig. 2, Fig. 3. Further results are also available in Appendices 2–4.
Discussion
In 132 eligible meta-analyses of spine surgery interventions, we found that the mean number of satisfactory items for credibility was 3 of 7 (SD, 1.4; ICC, 0.86) and the mean number of satisfactory items for completeness of reporting was 18 of 27 (SD, 4.4; ICC, 0.94). Each of more recent publication, increasing Journal Impact Factor, increasing number of databases, inclusion of RCTs, and inclusion of non-English studies were associated with higher credibility and completeness of reporting.
Conclusions
Meta-analyses are powerful tools that can synthesize existing research, inform clinical practice, and directly support evidence-based care. They can be profoundly impactful when high-quality primary evidence and high-quality methodology align, but the credibility of many current spine surgery meta-analyses is limited. Researchers can improve the credibility of future meta-analyses by performing sufficiently exhaustive literature searches, addressing possible explanations of heterogeneity,
References (90)
- et al.
Spine surgery research: on and beyond current strategies
Spine J
(2012) - et al.
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement
J Clin Epidemiol
(2009) - et al.
GRADE guidelines: 2. framing the question and deciding on important outcomes
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 4. rating the quality of evidence—study limitations (risk of bias)
J Clin Epidemiol
(2011) - et al.
GRADE guidelines: 7. rating the quality of evidence—inconsistency
J Clin Epidemiol
(2011) - et al.
Single data extraction generated more errors than double data extraction in systematic reviews
J Clin Epidemiol
(2006) - et al.
GRADE guidelines: 1. introduction-GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
(2011) - et al.
Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis
J Clin Epidemiol
(2013) - et al.
Level of evidence of clinical spinal research and its correlation with journal impact factor
Spine J
(2013) - et al.
Cervical arthroplasty: a critical review of the literature
Spine J
(2014)
Motion analysis of single-level cervical total disc arthroplasty: a meta-analysis
Orthop Surg
A meta-analysis of cervical arthroplasty compared to anterior cervical discectomy and fusion for single-level cervical disc disease
J Clin Neurosci
Comparison of vertebroplasty and balloon kyphoplasty for treatment of vertebral compression fractures: a meta-analysis of the literature
Spine J
Kyphoplasty versus vertebroplasty for painful osteoporotic vertebral compression fractures—which one is better? A systematic review and meta-analysis
Int J Spine Surg
A meta-analysis of balloon kyphoplasty compared to percutaneous vertebroplasty for treating osteoporotic vertebral compression fractures
J Clin Neurosci
Both unilateral and bilateral pedicle screw fixation are effective for lumbar spinal fusion—A meta-analysis-based systematic review
J Orthop Translation
Unilateral versus bilateral pedicle screw fixation in lumbar interbody fusion: a meta-analysis of complication and fusion rate
Clin Neurol Neurosurg
An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods
J Clin Epidemiol
Validation of an index of the quality of review articles
J Clin Epidemiol
AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews
J Clin Epidemiol
Effectiveness of local vancomycin powder to decrease surgical site infections: a meta-analysis
Spine J
Is double-row rotator cuff repair clinically superior to single-row rotator cuff repair: a systematic review of overlapping meta-analyses
Arthroscopy
Differences between systematic reviews/meta-analyses of hyaluronic acid/hyaluronan/hylan in osteoarthritis of the knee
Osteoarthritis Cartilage
How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature
JAMA
Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?
J Bone Joint Surg Am
Meta-analyses in orthopaedic surgery. A systematic review of their methodologies
J Bone Joint Surg Am
Meta-analyses in joint arthroplasty: a review of quantity, quality, and impact
J Bone Joint Surg Am
Methodological quality of systematic reviews addressing femoroacetabular impingement
Knee Surg Sports Traumatol Arthrosc
Errors in the conduct of systematic reviews of pharmacological interventions for irritable bowel syndrome
Am J Gastroenterol
The process of a systematic review and meta-analysis: users’ guide to the medical literature
Methodology and reports of systematic reviews and meta-analyses: a comparison of cochrane reviews with articles published in paper-based journals
JAMA
The quality of reporting methods and results in network meta-analyses: an overview of reviews and suggestions for improvement
PLoS One
The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Health Technol Assess
OE—OrthoEvidence [homepage on the Internet]
Handsearching versus electronic searching to identify reports of randomized trials
Cochrane Database Syst Rev
Should unpublished data be included in meta-analyses? Current convictions and controversies
JAMA
Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses
BMJ
Subgroup analyses: when should we believe them?
J Bone Joint Surg Am
A methodological survey of the analysis, reporting and interpretation of absolute risk ReductiOn in systematic revieWs (ARROW): a study protocol
Syst Rev
Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units
Health Qual Life Outcomes
Science citation index [homepage on the Internet]
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Phys Ther
Conflict of interest in orthopaedic research. An association between findings and funding in scientific presentations
J Bone Joint Surg Am
The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review
BMJ
Cited by (14)
Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: A cross-sectional study
2021, American Journal of Clinical Nutrition
FDA device/drug status: Not applicable.
Author disclosures: NE: Nothing to disclose. LvdW: Nothing to disclose. MB: Consultancy: Smith & Nephew (C), Stryker (C), Amgen (C), Zimmer (C), Moximed (C), Bioventus (C), Merck, Eli Lilly, Sanofi (F); Grants/grants pending: Smith & Nephew (E, paid directly to institution), DePuy, Eli Lily, Bioventus (C), Stryker (F, Paid directly to institution), Zimmer, Amgen (F, Paid directly to institution). MG: Grant: Canadian Institutes of Health Research (G, Research grant), Canadian Cancer Society (F, Research grant); Personal Fees: Wright Medical (C, Consultant ad hoc). IA: Nothing to disclose. BD: Nothing to disclose. GG: Nothing to disclose.
The disclosure key can be found on the Table of Contents and at www.TheSpineJournalOnline.com.
The authors certify that they, or a member of their immediate families, have no funding or commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, and so forth) that might pose a conflict of interest in connection with the submitted article. This study was not funded. Dr Bhandari is funded in part by a Canada Research Chair. This study did not require local institutional ethics board approval.