Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy

doi:10.1016/j.jclinepi.2009.04.007

Journal of Clinical Epidemiology

Volume 63, Issue 3, March 2010, Pages 289-298

https://doi.org/10.1016/j.jclinepi.2009.04.007 Get rights and content

Abstract

Objective

This study assessed the impact of systematic review and data extraction experience on the accuracy and efficiency of data extraction in systematic reviews.

Study Design and Setting

We conducted a prospective cross-sectional study from October to December 2006. Participants were classified as having minimal, moderate, or substantial experience in systematic reviews and data extraction. Three studies on insomnia treatment were extracted. Our primary outcome was the accuracy of data extraction. Data sets of each experience level were analyzed for errors in data extraction and results of meta-analyses. Additionally, the time required for completion of data extraction was compared.

Results

Error rates were similar across the various levels of experience and ranged from 28.3% to 31.2%. Mean rates for errors of omission (11.3–16.4%) were generally lower than those for errors of inaccuracy (13.9–17.9%). There were no significant differences in error rates or accuracy of meta-analysis results between groups. Time required approached significance, with minimally experienced participants requiring the most time.

Conclusion

Overall, there were high error rates by participants at all experience levels; however, time required for extraction tended to decrease with experience. These results illustrate the need to develop strategies aimed at mastery of data extraction, rather than reliance on previous data extraction experience alone.

Section snippets

Background

There is currently no recommended standard for data extraction in systematic reviews with respect to the experience level of reviewers in systematic reviews and data extraction. To our knowledge, there is no empirical evidence regarding the types and magnitude of errors accompanying data extraction conducted by reviewers with various levels of experience in systematic reviews and data extraction, the impact of these errors on the results of meta-analysis, or the efficiency in data extraction

Participant recruitment

The participants of this study were recruited through The Cochrane Collaboration, the Evidence-based Practice Center program of the US Agency for Healthcare Research and Quality, and relevant departments at the University of Alberta. A letter of invitation was sent to the members and students of these respective entities, which directed them to an online screening questionnaire. Individuals with prior knowledge of the systematic review process by education and/or experience were eligible for

Participants

Two hundred and forty individuals responded to the invitation to participate in the study and completed the screening questionnaire (Fig. 1). One hundred and fifty-four individuals were eligible to participate based on their completion of the screening questionnaire and were categorized according to their level of experience in systematic reviews and data extraction. One hundred and twenty-one individuals began the data extraction process with variable completion rates across studies.

Discussion

This is one of the first studies to examine, in a controlled manner, the effect of systematic review and data extraction experience on the accuracy of data extraction in systematic reviews. Overall, we found that level of experience did not result in measurable differences in error rates. Of note is the high level of errors in general with an overall error rate of 28.7%, which is higher than that found in previous research [4]. We found that the errors were more often because of inaccuracies

Conclusion

We found that data extraction did not vary significantly by level of data extraction and systematic review experience in terms of error rates or results of the meta-analysis. Overall, we found high error rates by all experience level groups, which underscores the importance of adequate instruction, training, and care in data extraction in systematic reviews. The familiarity of the data extractors with the terminology and outcomes specific to a field of research may play a role in the accuracy

Acknowledgments

The authors would like to acknowledge Joseph Lau, David Moher, and Lina Santaguida (on behalf of Parminder Raina) who provided expert input on the development of the participant experience classification scheme. We also thank Marilyn Josefsson and Kelley Bessette for their assistance in handling participant inquiries and records.

We gratefully acknowledge the Canadian Agency for Drug and Technologies for Health who provided the funding for this research.

Author contributions: J.H. participated in

References (12)

D. Moher et al.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement
Lancet
(1999)
A. Jones et al.
High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews
J Clin Epidemiol
(2005)
N. Buscemi et al.
Single data extraction generated more errors than double data extraction in systematic reviews
J Clin Epidemiol
(2006)
L. Vase et al.
A comparison of placebo effects in clinical analgesic trials versus studies of placebo analgesia
Pain
(2002)
A. Hróbjartsson et al.
Unsubstantiated claims of large effects of placebo on pain: serious errors in meta-analysis of placebo analgesia mechanism studies
J Clin Epidemiol
(2006)
W. Strang et al.
Inter-reader variation

There are more references available in the full text version of this article.

Cited by (50)

Resource use during systematic review production varies widely: a scoping review
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Because the goal of this scoping review was to descriptively map the resource use of SR production steps rather than to derive cause–effect relationships, we did not apply a formal certainty of evidence or risk of bias (RoB) assessment. We included 34 studies (32 quantitative primary studies, 1 qualitative primary study, 1 SR) that were published in 38 publications [9,23–59] (Fig. 2: PRISMA study flow). In the following sections, we first summarize the characteristics of the included studies.
We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.
We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.
We included 34 studies. Thirty-two reported on the resource use—mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.
Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.
The effectiveness of psychological interventions for loneliness: A systematic review and meta-analysis
2021, Clinical Psychology Review
Chronic loneliness is associated with a range of mental health difficulties. Previous theory and research indicate that psychological interventions show promise for reducing loneliness, however, there have been no systematic reviews or meta-analyses to ascertain the efficacy of these interventions across the lifespan. The aim of this study was to synthesise, meta-analyse and explore the heterogeneity in RCTs of psychological interventions for loneliness in order to establish their efficacy.
Five databases (Ovid Embase, Ovid Medline, PsycINFO, Web of Science and CINAHL) were systematically searched in order to identify relevant studies. Included studies were required to be peer-reviewed RCTs examining psychological interventions for loneliness. Two independent coders examined the abstracts of the 3973 studies and 103 full texts, finding 31 studies that met inclusion criteria, 28 of which contained sufficient statistical information to be included in the meta-analysis. The quality of included studies was assessed using the Cochrane Risk of Bias Tool.
The 31 studies (N = 3959) that were included in the systematic review were conducted with participants from a diverse range of cultures, age groups and populations. The interventions were of mixed quality and were mostly face to face, group-based and delivered weekly. The most common type of intervention was Cognitive Behavioural Therapy (CBT).
28 studies (N = 3039) were included in a meta-analysis which found that psychological interventions significantly reduced loneliness compared to control groups, yielding a small to medium effect size (g = 0.43). Subgroup analysis and meta-regressions were conducted in order to explore heterogeneity and found that type of psychological intervention was approaching significance as a moderator of the effectiveness of psychological interventions for loneliness.
In conclusion, psychological interventions for loneliness across the lifespan are effective. This finding should inform policy makers, researchers and clinicians going forward, especially in the context of increased loneliness due to the COVID-19 pandemic. There was considerable heterogeneity in the effectiveness of the interventions, suggesting that future research should also explore what works for whom and consider personalising psychological treatment.
The influence of industry sponsorship and conflict of interest on results and conclusions of systematic reviews regarding treatment of knee osteoarthritis
2021, Osteoarthritis and Cartilage Open
To assess the influence of industry sponsorship and authors’ conflicts of interest on the favorability of results and conclusions of systematic reviews regarding osteoarthritis of the knee.
We searched MEDLINE, Embase, and the Cochrane Database of Systematic Reviews on June 26, 2020 for systematic reviews with or without meta-analyses focusing on treatment of osteoarthritis of the knee. Author COI were collected from the systematic review’s COI disclosure statement, the CMS Open Payments Database, Dollars for Profs, Google Patents, the United States Patent and Trademark Office (USPTO), as well as previously published COI disclosure statements. Study sponsorship was determined using information provided in each systematic review’s funding statement.
Our study included 53 systematic reviews conducted by a total of 279 authors. Thirty-one authors (of 279; 11.1%) had one or more COI. Fourteen systematic reviews (of 53; 26.4%) had one or more conflicted authors. Of these reviews, only two (of 14; 14.3%) reported narrative results favoring the treatment group, whereas five (of 14; 35.7%) reported conclusions favoring the treatment group. Our results showed no statistically significant association between the presence of COI and the favorability of results (P= 0.11) and conclusions (P= 0.73). Because our sample only included one industry-sponsored systematic review, we were unable to adequately assess for a relationship between industry-sponsorship and favorability of results and conclusions.
We found no association between the favorability of systematic review results and conclusions with either the presence of author’s conflicts of interest or industry-sponsorship.
Does evidence support “banking/extending sleep” by shift workers to mitigate fatigue, and/or to improve health, safety, or performance? A systematic review
2019, Sleep Health
Sleep deprivation is common in shift work occupations, including safety-sensitive occupations. While extending sleep prior to scheduled shifts (i.e., “banking sleep”) may be an intuitive strategy for fatigue mitigation, the evidence behind this strategy is unclear.
We performed a systematic review of literature retrieved in searches of four databases. We examined agreement between two independent screeners, abstracted key findings, reviewed and synthesized findings, and evaluated the quality of evidence using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) methodology. The Cochrane Collaboration's risk of bias tool was used to evaluate bias of individual studies. We reported findings as prescribed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.
Of the 3007 records screened, five met inclusion criteria. The inter-rater agreement for inclusion/exclusion was high (κ = 0.87). One study addressed patient safety outcomes. Four studies assessed the impact of banking sleep on performance, five assessed measures of acute fatigue, and three evaluated banking sleep on indicators of health. All five studies presented a very serious risk of bias and the quality of evidence was very low. Given these caveats, the findings, in aggregate, support banking sleep as a strategy to improve indicators of performance and acute fatigue.
This systematic review identifies gaps in research of shift workers on the efficacy of banking sleep as a fatigue risk management strategy. The available evidence supports banking sleep prior to shiftwork as a strategy for improved patient safety, performance, and reducing acute fatigue.
Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review
2019, Journal of Clinical Epidemiology
Citation Excerpt :
Mathes et al. [16] recently published a methodological review of data extraction errors and methods to increase data extraction quality. Our study identified the same three methods studies as Mathes et al. [35–37] and several others covering other extraction approaches. Notably, our findings and conclusions are consistent.
The aim of the article was to identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.
A systematic review was conducted, searching MEDLINE, EMBASE, and the Cochrane Library from inception to September 1, 2016. Quality appraisal of included studies was undertaken using a modified Quality Assessment of Diagnostic Accuracy Studies 2, and key results on accuracy, reliability, efficiency of a methodology, or impact on results and conclusions were extracted.
After screening 5,600 titles and abstracts and 245 full-text articles, 37 studies were included. For screening, studies supported the involvement of two independent experienced reviewers and the use of Google Translate when screening non-English articles. For data abstraction, studies supported involvement of experienced reviewers (especially for continuous outcomes) and two independent reviewers, use of dual monitors, graphical data extraction software, and contacting authors. For quality appraisal, studies supported intensive training, piloting quality assessment tools, providing decision rules for poorly reported studies, contacting authors, and using structured tools if different study designs are included.
Few studies exist documenting common systematic review practices. Included studies support several systematic review practices. These results provide an updated evidence-base for current knowledge synthesis guidelines and methods requiring further research.
Meta-analysis of Nutrition Studies
2019, Analysis in Nutrition Research: Principles of Statistical Methodology and Interpretation of the Results
The meta-analysis of nutrition studies combines results from multiple studies on single foods or nutrients or several of them, and dietary patterns or specific dietary plans and supplements in relation to a health outcome, and provides an overall estimate of the effect. This chapter offers a framework to assist researchers in understanding the rationale behind performing a meta-analysis in nutrition science. It addresses in detail the proper implementation and interpretation of meta-analytic procedures and discusses the main aspects of a systematic review and meta-analysis, such as framing a research question, searching for relevant studies, applying appropriate meta-analytical techniques to combine the data, and presenting them in a comprehensive manner.

View all citing articles on Scopus

View full text

Systematic Reviews and Meta AnalysisSystematic review data extraction: cross-sectional study showed that experience did not increase accuracy

Abstract

Objective

Study Design and Setting

Results

Conclusion

Section snippets

Background

Participant recruitment

Participants

Discussion

Conclusion

Acknowledgments

Lancet

J Clin Epidemiol

J Clin Epidemiol

Pain

J Clin Epidemiol

Inter-reader variation

Systematic Reviews and Meta Analysis
Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy