Original Article
Development quality criteria to evaluate nontherapeutic studies of incidence, prevalence, or risk factors of chronic diseases: pilot study of new checklists

https://doi.org/10.1016/j.jclinepi.2010.08.006Get rights and content

Abstract

Objective

To develop two checklists for the quality of observational studies of incidence or risk factors of diseases.

Study Design and Setting

Initial development of the checklists was based on a systematic literature review. The checklists were refined after pilot trials of validity and reliability were conducted by seven experts, who tested the checklists on 10 articles.

Results

The checklist for studies of incidence or prevalence of chronic disease had six criteria for external validity and five for internal validity. The checklist for risk factor studies had six criteria for external validity, 13 criteria for internal validity, and two aspects of causality. A Microsoft Access database produced automated standardized reports about external and internal validities. Pilot testing demonstrated face and content validities and discrimination of reporting vs. methodological qualities. Interrater agreement was poor. The experts suggested future reliability testing of the checklists in systematic reviews with preplanned protocols, a priori consensus about research-specific quality criteria, and training of the reviewers.

Conclusion

We propose transparent and standardized quality assessment criteria of observational studies using the developed checklists. Future testing of the checklists in systematic reviews is necessary to develop reliable tools that can be used with confidence.

Introduction

What is new?

Checklists for reporting and methodological qualities of observational studies that addressed the incidence of chronic conditions or disease risk factors were developed with transparent and standardized quality assessment in systematic reviews and evidence-based reports.

The prevalence and incidence of chronic conditions have implications for policy and health care utilization. Valid information about risk factors is important in reducing the burden of chronic diseases [1], [2]. Although systems to rank the strength of the recommendations about effective interventions consider all evidence from observational studies as low [3], [4], prevalence and risk factors for chronic diseases can be evaluated only in observational studies [5]. Public policy decisions should be based on applicable and unbiased results from high-quality studies [6], [7], [8]. Assessing the quality of observational studies is an important part of evidence-based reports made for the Agency for Healthcare Research and Quality [9].

An extensive review of all available systems for rating the strength of scientific evidence led to the conclusion that future efforts need to identify valid and reliable quality ratings for observational studies [9], [10]. Different methodological aspects, including selective treatment assignment, access to health care, or provider characteristics, may have different importances for studies that examine treatment effects and prevalence of chronic conditions or the association of disease risk factors with patient mortality and morbidity [9], [10]. Therefore, quality evaluation that is part of grading of a body of evidence must be tailored to the methodological aspects and quality standards of nontherapeutic observational studies.

The present collaborative project sought to develop valid and reliable quality criteria of observational studies that examine the incidence or prevalence of chronic conditions and risk factors for diseases. We propose criteria for the design, reporting standards, and assessment of nontherapeutic observational studies in systematic reviews and evidence-based reports.

Section snippets

Methods

We developed two checklists, one for studies of incidence or prevalence and another for risk factors, based on our literature review and in collaboration with experts from other Evidence-based Practice Centers (EPC) and the Centers for Disease Control and Prevention (CDC).

The protocol to construct the checklist was based on a conceptual model of the development of indices, rating scales, or other appraisals, to describe and measure symptoms, physical signs, and other clinical phenomena in

Results

We identified 84 publications that described 96 tools [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85],

Discussion

We present the results of our pilot collaborative project to develop checklists for quality assessment of observational nontherapeutic studies. Comparing our checklists with previously published ones, we could find only one previously published tool that was developed for studies of incidence or prevalence of health conditions [50]. The scale has three domains, including the validity of study methods and interpretation and applicability of the results. A maximum score of 8 does not

References (127)

  • J.A. Borghouts et al.

    The clinical course and prognostic factors of non-specific neck pain: a systematic review

    Pain

    (1998)
  • S. van Rooyen et al.

    Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts

    J Clin Epidemiol

    (1999)
  • S. Zaza et al.

    Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services

    Am J Prev Med

    (2000)
  • E.P. Steinberg et al.

    Methods used to evaluate the quality of evidence underlying the National Kidney Foundation-Dialysis Outcomes Quality Initiative Clinical Practice Guidelines: description, findings, and implications

    Am J Kidney Dis

    (2000)
  • T.V. Macfarlane et al.

    Systematic review of population-based epidemiological studies of oro-facial pain

    J Dent

    (2001)
  • G.G.M. Scholten-Peeters et al.

    Prognostic factors of whiplash-associated disorders: a systematic review of prospective cohort studies

    Pain

    (2003)
  • S.J. Rangel et al.

    Development of a quality assessment scale for retrospective clinical studies in pediatric surgery

    J Pediatr Surg

    (2003)
  • M. Pavia et al.

    Association between fruit and vegetable consumption and oral cancer: a meta-analysis of observational studies

    Am J Clin Nutr

    (2006)
  • A.C. Tricco et al.

    Few systematic reviews exist documenting the extent of bias: a systematic review

    J Clin Epidemiol

    (2008)
  • A. Conde-Agudelo et al.

    Effects of birth spacing on maternal health: a systematic review

    Am J Obstet Gynecol

    (2007)
  • D.M. Fox

    Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions

    Health Aff (Millwood)

    (2005)
  • J. Lavis et al.

    Towards systematic reviews that inform health care management and policy-making

    J Health Serv Res Policy

    (2005)
  • D. Atkins et al.

    Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches of The GRADE Working Group

    BMC Health Serv Res

    (2004)
  • D. Atkins et al.

    Systems for grading the quality of evidence and the strength of recommendations II: pilot study of a new system

    BMC Health Serv Res

    (2005)
  • A. Aschengrau et al.

    Essentials of epidemiology in public health

    (2003)
  • L.A. Bero et al.

    How consumers and policymakers can use systematic reviews for decision making

    Ann Intern Med

    (1997)
  • K.S. Chan et al.

    Systematic reviews for evidence-based management: how to find them and what to do with them

    Am J Manag Care

    (2004)
  • West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology...
  • T. Shamliyan

    Prevention of urinary and fecal incontinence in adults

  • A.R. Feinstein

    Clinimetrics

    (1987)
  • How to read clinical journals: IV. To determine etiology or causation

    Can Med Assoc J

    (1981)
  • C.L. Krogh

    A checklist system for critical review of medical literature

    Med Educ

    (1985)
  • M.J. Gardner et al.

    Use of check lists in assessing the statistical content of medical studies

    Br Med J (Clin Res Ed)

    (1986)
  • C.D. Mulrow et al.

    Blood glucose and diabetic retinopathy: a critical appraisal of new evidence

    J Gen Intern Med

    (1986)
  • M.P. Longnecker et al.

    A meta-analysis of alcohol consumption in relation to risk of breast cancer

    JAMA

    (1988)
  • J.S. Reisch et al.

    Aid to the evaluation of therapeutic studies

    Pediatrics

    (1989)
  • W.O. Spitzer et al.

    Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking

    Clin Invest Med

    (1990)
  • J.A. Berlin et al.

    A meta-analysis of physical activity in the prevention of coronary heart disease

    Am J Epidemiol

    (1990)
  • S.R. Stock

    Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: a meta-analysis

    Am J Ind Med

    (1991)
  • F.G. Fowkes et al.

    Critical appraisal of published research: introductory guidelines

    BMJ (Clin Res Ed)

    (1991)
  • S.G. Carruthers et al.

    Report of the Canadian Hypertension Society Consensus Conference: 1. Introduction

    CMAJ

    (1993)
  • C.A. Carson et al.

    Quality of published reports of the prognosis of community-acquired pneumonia

    J Gen Intern Med

    (1994)
  • M. Avis

    Reading research critically. II. An introduction to appraisal: assessing the evidence

    J Clin Nurs

    (1994)
  • T.W. Gyorkos et al.

    An approach to the development of practice guidelines for community health interventions

    Can J Public Health

    (1994)
  • M.K. Cho et al.

    Instruments for assessing the quality of drug studies published in the medical literature

    JAMA

    (1994)
  • M. Levine et al.

    Users’ guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group

    JAMA

    (1994)
  • S.N. Goodman et al.

    Manuscript quality before and after peer review and editing at Annals of Internal Medicine

    Ann Intern Med

    (1994)
  • D. Campos-Outcalt et al.

    The effects of medical school curricula, faculty role models, and biomedical research support on choice of generalist physician careers: a review and quality assessment of the literature

    Acad Med

    (1995)
  • B.M. Margetts et al.

    Development of a scoring system to judge the scientific quality of information from case-control and cohort studies of nutrition and disease

    Nutr Cancer

    (1995)
  • D.E. Cowley

    Prostheses for primary total hip replacement. A critical appraisal of the literature

    Int J Technol Assess Health Care

    (1995)
  • Cited by (89)

    • Quality assessment of prevalence studies: a systematic review

      2020, Journal of Clinical Epidemiology
      Citation Excerpt :

      Table 1 summarizes the main characteristics of these tools. Among these tools, seven (87.5%) were new tools [11–13,15–19], and one (12.5%) was an adaptation of an existing instrument [14]. Four tools (50.0%) were developed to assess studies of prevalence of any clinical condition [13–15,18,19], whereas the other four tools (50.0%) were developed to appraise prevalence studies of specific medical fields [11,12,16,17]; however, with some adaptations, they could all be applied to any clinical condition.

    • MicroRNA and retinoic acid

      2020, Clinica Chimica Acta
      Citation Excerpt :

      After selection of desired articles, following data were extracted: author, journal and population information, all statistical association information like odds ratio (OR), relative risk (RR) or hazard ratio (HR). Evaluation of the quality of articles, were done by using The Guidelines of Methodological Evaluation of Observational Research [8]. Design bias (scored 0–3), Selection bias (scored 0–4), Information bias, (scored 0–5), Confounding (scored 0–3), and Analysis bias were assessed and scores from 0 to 17 points were given to each article.

    • Association between night work and prostate cancer: a systematic review and meta-analysis

      2024, International Archives of Occupational and Environmental Health
    View all citing articles on Scopus

    This project was funded under Contract No. 290-02-0009, from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The authors are responsible for its content. Statements in the article should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

    View full text