Original Article
Effects of FRAX® Model Calibration on Intervention Rates: A Simulation Study

https://doi.org/10.1016/j.jocd.2011.03.007Get rights and content

Abstract

The WHO fracture risk assessment tool (FRAX®) estimates an individual’s 10-yr major osteoporotic and hip fracture probabilities using a tool customized to the fracture epidemiology of a specific population. Incorrect model calibration could therefore affect performance of the model in clinical practice. The current analysis was undertaken to explore how simulated miscalibration in the FRAX® tool would affect the numbers of individuals meeting specific intervention criteria (10-yr major osteoporotic fracture probability ≥20%, 10-yr hip fracture probability ≥3%). The study cohort included 36,730 women and 2873 men aged 50 yr and older with FRAX® probability estimates using femoral neck bone mineral density. We simulated relative miscalibration error in 10% increments from −50% to +50% relative to a correctly calibrated FRAX® model. We found that small changes in model calibration (even on the order of 10%) had large effects on the number of individuals qualifying for treatment. There was a steep gradient in the relationship between relative change in calibration and relative change in intervention rates: for every 1% change in calibration, there was a 2.5% change in intervention rates for women and 4.1% for men. For hip fracture probability, the gradient of the relationship was closer to unity. These results highlight the importance of FRAX® model calibration, and speak to the importance of using high-quality fracture epidemiology in constructing FRAX® tools.

Introduction

Although reduced bone mass is an important and easily quantifiable measurement, studies have shown that most fractures occur in individuals with a bone mineral density (BMD) T-score above the operational threshold for osteoporosis 1, 2, 3, 4. The use of clinical risk factors (CRFs) has been shown to enhance the performance of BMD in the prediction of hip and major osteoporotic fractures (5). In addition to a prior fragility fracture, other important CRFs include age, sex, body mass index, prolonged use of glucocorticoids, rheumatoid arthritis, parental history of hip fracture, current smoking, alcohol intake of 3 or more units/d, and secondary osteoporosis (6). These elements are integrated in the WHO fracture risk assessment tool (FRAX®) for estimation of individual 10-yr major osteoporotic and hip fracture probabilities (6).

Meta-analyses have confirmed that there is an improvement in the fracture prediction using BMD and CRFs together compared with using either BMD alone or CRFs alone 4, 5. This has led to broad endorsement of FRAX® and its integration into several clinical practice guidelines 7, 8, 9, 10, 11, 12, 13, 14, 15. Fracture and mortality rates are known to vary widely between countries (16). Therefore, population-specific FRAX® tools can be customized to the fracture and mortality epidemiology in that specific region (6). At present, more than 35 FRAX® models are available, and others are being developed.

Fundamental features of a predictive model’s performance are discrimination and calibration (17). Model discrimination addresses the question: “How well did the model perform in terms of risk stratification?” Of equal importance is model calibration, which asks the question: “Was the observed fracture risk consistent with the predicted fracture risk?” Although FRAX® has been independently shown in multiple cohorts to provide fracture discrimination comparable to the initial derivation cohorts (5), assessment of calibration is less frequently considered but incorrect calibration could affect performance of the model in clinical practice.

A 10-yr major osteoporotic fracture probability of greater than 20% is considered high risk and an indication for intervention according to the National Osteoporosis Foundation (NOF) of the United States and Osteoporosis Canada 7, 8, 18. The NOF also recommends that a 10-yr hip fracture probability of 3% or greater be considered for intervention, in addition to those with any BMD measurements in the osteoporotic range and those with prior spine or hip fractures. The current analysis was undertaken to explore how simulated miscalibration in a previously validated FRAX® tool would affect the numbers of individuals meeting these intervention criteria.

Section snippets

Subjects and Setting

In the Province of Manitoba, Canada, health services are provided to virtually all residents through a single public health care system. Bone density testing with dual-energy X-ray absorptiometry (DXA) has been managed as an integrated program since 1997; criteria and testing rates for this program have been published (19). The program maintains a database of all DXA results, which can be linked with other population-based computerized health databases through an anonymous personal identifier.

Results

The study cohort included 36,730 women and 2873 men. Baseline characteristics of the population are summarized in Table 1. Men were slightly older than women (mean age 68.2 ± 10.1 vs 65.7 ± 9.8, p < 0.001). Women had a lower mean femoral neck T-score (−1.5 ± 1.0 vs −1.2 ± 1.1, p < 0.001). FRAX® risk estimates for major osteoporotic fracture were greater for women than men (p < 0.001), but were similar for hip fracture probability. Using the reference calibration, 4137 (11.3%) women and 83 (2.9%) men exceeded

Discussion

This analysis found that relatively small changes in model calibration (even on the order of 10% relative miscalibration) can have large effects on the number of individuals qualifying for treatment. This effect is particularly evident with major osteoporotic fracture probability where a 50% relative miscalibration could more than double (overcalibration) or virtually eliminate (undercalibration) treatment of osteoporosis. Hip fracture probability is also affected by miscalibration, but the

Acknowledgments

The authors are indebted to Manitoba Health for the provision of data (HIPC File No. 2007/2008−49). The results and conclusions are those of the authors, and no official endorsement by Manitoba Health is intended or should be inferred. This article has been reviewed and approved by the members of the Manitoba Bone Density Program Committee. We would like to thank Ms. Helena Johansson and Dr. John Kanis for generating the FRAX® results for the Manitoba cohort.

Disclosures: William D. Leslie:

References (35)

  • J.A. Kanis et al.

    The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women

    Osteoporos Int

    (2007)
  • B. Dawson-Hughes

    A revised clinician’s guide to the prevention and treatment of osteoporosis

    J Clin Endocrinol Metab

    (2008)
  • B. Dawson−Hughes et al.

    Implications of absolute fracture risk assessment for osteoporosis practice guidelines in the USA

    Osteoporos Int

    (2008)
  • J.A. Kanis et al.

    FRAX and the assessment of fracture probability in men and women from the UK

    Osteoporos Int

    (2008)
  • J.A. Kanis et al.

    Case finding for the management of osteoporosis with FRAX((R))-assessment and intervention thresholds for the UK

    Osteoporos Int

    (2008)
  • K. Lippuner et al.

    FRAX assessment of osteoporotic fracture probability in Switzerland

    Osteoporos Int

    (2010)
  • J.A. Kanis et al.

    European guidance for the diagnosis and management of osteoporosis in postmenopausal women

    Osteoporos Int

    (2008)
  • Cited by (16)

    • Circulating serum microRNAs including senescent miR-31-5p are associated with incident fragility fractures in older postmenopausal women with type 2 diabetes mellitus

      2022, Bone
      Citation Excerpt :

      As DXA scans were not performed in the AGES-Reykjavik study, and as the FRAX website and the International Society of Clinical Densitometry (ISCD) and the International Osteoporosis Foundation (IOF) in their joint official positions do not recommend to calculate FRAX using a BMD measurement obtained with other technologies than DXA [62], we decided to not input our CT-derived DXA-equivalent areal femoral neck BMD into the FRAX calculator and therefore could not compute BMD containing FRAX major osteoporotic fracture ten year probabilities in the conventional way. However, in order to take advantage of our available CT-derived areal BMD information and as meta-analyses have shown improved fracture prediction using BMD and clinical risk factors together compared with using either BMD or CRFs alone [63,64], we created via Cox-Proportional hazard modelling (with clinical FRAX and DXA-equivalent femoral neck aBMD as covariates) a combined FRAXaBMD variable that was used as surrogate FRAXaBMD for all statistical analyses. In 17 subjects, baseline CT could not be performed, and one woman was too heavy for clinical FRAX calculations, leaving a total of 149 subjects in whom FRAXaBMD could be calculated.

    • Worldwide Fracture Prediction

      2017, Journal of Clinical Densitometry
      Citation Excerpt :

      Such errors would, however, impact absolute FRAX-derived fracture probabilities, and thus daily practice at the individual and public health level, the latter because of a major impact on cost-effectiveness analyses (15). Indeed, Leslie et al showed that small changes in FRAX calibration, for example, as little as 10%, had a large effect on the number of individuals qualifying for treatment (162). The importance of high-quality data that best represents country-specific epidemiology was further underscored in a study directly comparing 8 national FRAX tools for fracture prediction and treatment qualification in 36,730 Canadian women, mean age 65.7 ± 9.8 yr, enrolled in the Manitoba BMD program.

    • Calibration of FRAX: A Journey, not a Destination

      2021, Calcified Tissue International
    View all citing articles on Scopus
    View full text