Original Article
Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement

https://doi.org/10.1016/j.jclinepi.2011.11.014Get rights and content

Abstract

Objective

In the course of performing systematic reviews on the prevalence of low back and neck pain, we required a tool to assess the risk of study bias. Our objectives were to (1) modify an existing checklist and (2) test the final tool for interrater agreement.

Study Design and Setting

The final tool consists of 10 items addressing four domains of bias plus a summary risk of bias assessment. Two researchers tested the interrater agreement of the tool by independently assessing 54 randomly selected studies. Interrater agreement overall and for each individual item was assessed using the proportion of agreement and Kappa statistic.

Results

Raters found the tool easy to use, and there was high interrater agreement: overall agreement was 91% and the Kappa statistic was 0.82 (95% confidence interval: 0.76, 0.86). Agreement was almost perfect for the individual items on the tool and moderate for the summary assessment.

Conclusion

We have addressed a research gap by modifying and testing a tool to assess risk of study bias. Further research may be useful for assessing the applicability of the tool across different conditions.

Section snippets

Introduction and background

What is new?

Key findings

  1. We developed a risk of bias tool for prevalence studies based on a review of the literature, expert consensus, pilot testing of draft items, and refinement of the tool.

  2. The tool consists of 10 items addressing four domains of bias plus a summary risk of bias assessment.

  3. Raters found the tool easy to use, and we demonstrated high interrater agreement of the tool in assessing risk of bias of prevalence studies of low back and neck pain (overall agreement: 91%; Kappa statistic: 0.82).

What this adds to what was known?

Methods

The process we followed is shown in Fig. 1. To establish face validity of the tool, we (1) examined checklists that were regarded as good quality in previous reviews [12], [13], [14]; (2) listed key recommendations from previous reviews of study assessment tools [4], [6], [12], [15]; (3) searched the peer-reviewed literature for tools for assessing risk of study bias, particularly those relevant to low back and neck pain prevalence studies [3], [8], [9], [10], [11], [13], [14], [15], [16], [17]

Results

The final risk of bias tool comprises 10 items plus a summary assessment (see Appendix on the journal's Web site at www.jclinepi.com). Items 1 to 4 assess the external validity of the study (domains are selection and nonresponse bias), and items 5 to 10 assess the internal validity (items 5 to 9 assess the domain of measurement bias, and item 10 assesses bias related to the analysis). We included the item assessing whether the results of the study were representative of the national population

Discussion

Assessing the risk of study bias is an important step in performing and interpreting systematic reviews of the literature. Risk of bias tables are now routinely included in Cochrane systematic reviews that assess the efficacy and safety of treatment interventions [3]. Reviews have found that tools used for assessing quality of nonrandomized studies are poorly developed or have been specifically designed to assess randomized controlled trials and thus fail to include key criteria important for

Conclusions

We have developed a risk of bias tool, which builds on previous work, to assess risk of bias of studies measuring disease prevalence. This tool was found to be easy to apply and demonstrated high interrater agreement. We have now applied our risk of bias tool for assessing prevalence studies of gout, osteoarthritis, and rheumatoid arthritis for GBD 2010. Further research is needed to assess the reliability of the tool for assessing prevalence studies of other conditions.

References (28)

  • C.E. Dionne et al.

    Does back pain prevalence really decrease with increasing age? A systematic review

    Age Ageing

    (2006)
  • Q.A. Louw et al.

    The prevalence of low back pain in Africa: a systematic review

    BMC Musculoskelet Disord

    (2007)
  • J.J. Deeks et al.

    Evaluating non-randomised intervention studies

    Health Technol Assess

    (2003)
  • S.H. Downs et al.

    The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

    J Epidemiol Community Health

    (1998)
  • Cited by (1683)

    View all citing articles on Scopus
    View full text