Exploring methods for comparing the real-world effectiveness of treatments for osteoporosis: adjusted direct comparisons versus using patients as their own control

Several classes of agent are available for the prevention and treatment of osteoporosis, with the primary aim of reducing fracture risk [1]. Bisphosphonates have been the mainstay of treatment in Europe [2] and are available as oral (e.g., alendronate, risedronate and ibandronate) and intravenous (e.g., ibandronate and zoledronate) formulations. In Europe, oral bisphosphonates (OBPs) are commonly recommended as first-line treatment (most patients being prescribed alendronate), while intravenous treatments are typically reserved for second-line therapy [3]. With several osteoporosis treatment options available, it is important to develop and evaluate methods for comparing their effectiveness in real-world settings.

Bisphosphonates have been shown to reduce osteoporotic fracture risk in several randomised controlled trials (RCTs) [4‐6]. Although considered the gold standard for evaluating drug safety and efficacy, RCTs often have strict inclusion and exclusion criteria, and, consequently, enrolled patients may not accurately reflect the patient population treated in clinical practice with regard to comorbidities, concomitant medication use, age or disease severity. Moreover, owing to the complexity of identifying fractures and the requirement for large sample sizes and long follow-up periods, conducting RCTs to assess the effectiveness of osteoporosis treatments is difficult and costly [7].

Retrospective observational studies using large healthcare databases may prove useful for comparing the real-world effectiveness of osteoporosis treatments [8]. While gathering such data is less costly and more time-efficient than conducting RCTs, the lack of randomisation could lead to indication bias (e.g., for treatments indicated in patients at high risk of osteoporotic fracture, the real-world patient population may be biased towards those at high fracture risk). Confounders may be known and characterised, such as age, comorbidities or previous fractures, or may be unknown or unmeasurable factors [9]. It is important to consider the possibility of confounding when interpreting data from observational studies.

Several studies have utilised administrative databases to compare the real-world effectiveness of different OBPs; Cox proportional hazard models adjusting for risk factors for fracture, including through the use of propensity score methods, have been used to estimate effect differences between OBPs [10‐13] and to compare compliant vs. non-compliant patients [14]. However, these methods adjust only for known confounders. Abelson and colleagues estimated bisphosphonate effectiveness by measuring the change in fracture incidence over time and using each treatment group as its own control [9]. This method is similar to a self-controlled case series (SCCS) method in which individuals act as their own control and patient characteristics that remain constant over the observation period are accounted for [15, 16]. This method assumes that reduction in fracture incidence is not immediate following osteoporosis treatment initiation; bone mineral density (BMD) can take up to 2 years to reach its maximum level following treatment initiation [17].

No single study has used different methods for evaluating the real-world effectiveness of osteoporosis therapies that attempt to adjust for potential biases. This study evaluated two methods of comparing the real-world effectiveness of OBPs and zoledronate that attempt to adjust for observed and unobserved confounding.

We have assessed different methods for comparing the effectiveness of osteoporosis treatments in women initiating zoledronate or OBPs using retrospective data from Sweden and the Netherlands. The real-world effectiveness of these treatments was also compared; this analysis was performed on data from Sweden only, owing to the small zoledronate sample size in the Netherlands. This small sample size may be the result of the PHARMO database network linking data and not including the entire population of the Netherlands, as well as the fact that zoledronate had been available for only a short time when patients were being identified for inclusion in this study.

In both countries, baseline characteristics known to influence fracture risk differed between treatment groups. In each country, more than 40% of patients initiating zoledronate had received another osteoporosis treatment in the 12 months prior to zoledronate initiation. In addition, compared with patients initiating OBPs, a higher proportion of patients initiating zoledronate had experienced a fracture in the 12 months prior to treatment initiation. These results are not surprising, considering zoledronate is a second-line treatment given to patients who have a high fracture risk, or who found administration of OBPs complex, or who were intolerant to first-line therapies [34].

The 3-year cumulative fracture incidence in the Swedish sample was estimated at 18 and 14% for the zoledronate and OBPs groups, respectively. For both treatments, this is higher than that observed in two major multinational trials investigating their efficacy (the Fracture Intervention Trial [5] and the HORIZON trial [4]). This is not surprising, however, given that Swedish fracture risks are among the highest in the world [35] and that patients in real-world clinical practice are often older and more frail than those included in RCTs [8]. In addition, several studies have shown that persistence with bisphosphonates, particularly those administered orally, is suboptimal [36‐39]. In our study, no consideration was given to patients terminating treatment, so it is likely that several patients were not persistent with treatment. Given that persistence has been shown to affect anti-fracture efficacy [36, 40‐43], this may also explain some of the differences between the results in our study and those in the clinical trials. Moreover, patients in trials are likely to be monitored more closely than those in observational studies, which may itself encourage persistence.

RCTs have demonstrated that zoledronate and OBPs have similar efficacies [6, 18]. However, zoledronate is commonly used at second line [34], and patients prescribed zoledronate tend to be more frail than those prescribed OBPs. It is therefore not surprising that our analysis found a higher fracture incidence in patients initiating zoledronate than in those initiating OBPs. However, given that compliance and persistence with osteoporosis treatment are associated with efficacy [44], the high persistence seen with yearly IV zoledronate infusions [45], which guarantees 100% persistence and compliance for at least 12 months, may be expected to result in improved effectiveness compared with OBPs, which require frequent self-administration and have lower persistence and compliance. However, it should be noted that factors other than good compliance and persistence also play a role in real-world effectiveness.

The crude HRs for fracture incidence for the zoledronate group relative to the OBPs group ranged from 1.32 to 1.35 (p < 0.001). When adjusting for baseline covariates (ADC method), and using different models and propensity score adjustments, this was reduced to a non-significant estimate of approximately 1.10 (range 1.09–1.21, p > 0.05). This implies that, while the data registries used in our study included a number of known risk factors for fracture, information on additional potential risk factors (e.g., BMD, smoking, body mass index, fall propensity and socioeconomic variables) may be needed to successfully adjust for differences in baseline fracture risk. Information on several potential risk factors could be obtained from electronic medical records or hospital databases, enabling further retrospective research on the extent to which the differences in baseline risk can be accounted for.

OCA is the only conceivable method whereby the patient group acts as its own control, making it unnecessary to capture all the relevant risk factors for a fracture. The results from the OCA showed that the incidence of any fracture was lower in the treatment exposure period than in the baseline risk period. When including all fractures in all patients, the fracture incidence reductions were similar in the two treatment groups. However, in the analysis of fractures in hospitalised patients only and in the subgroup analysis of treatment-naïve patients, there was a trend towards a greater reduction in fracture incidence in the zoledronate than in the OBPs group. In the OBPs group, results remained unchanged when considering only fractures in hospitalised patients compared with the primary analysis which included fractures identified in both the hospital and outpatient settings. The reason for this pattern is unknown and warrants further study.

For zoledronate, the IRRs estimated using the OCA method were similar to those reported by Black and colleagues [4] for zoledronate vs. placebo (relative risk 0.67) and by Cummings and colleagues [5] for alendronate vs. placebo (relative risk 0.86). Our estimates for OBPs were slightly higher than those reported by Abelson and colleagues (IRR 0.72 and 0.79 for alendronate and risedronate, respectively) [9]. When considering only treatment-naïve patients, the IRR decreased from 0.91 to 0.59 in the zoledronate group. This is likely because over 50% of zoledronate-treated patients had previously received an OBP and benefited from some fracture protection and thus had a reduced fracture risk at baseline compared with those patients who were OBP-treatment-naïve. With zoledronate used as a second-line treatment in Sweden, it may be more appropriate to compare only treatment-naïve patients in the two treatment groups in this study, even though the sample size of the zoledronate group is notably smaller than that of the OBPs group.

In this study, time of onset of treatment effect was assumed to be 90 days. The exact time of onset is not known and may vary between different osteoporosis treatments; this is a factor to which the OCA method is sensitive, by definition. While more research is needed to identify the time to onset of treatment effect for different antiresorptive treatments, it is unlikely that onset of effect occurs at a specific time point (e.g., 90 days). Consequently, this type of approach should be used only to estimate comparative effectiveness, rather than effectiveness of individual treatments. Furthermore, it should be noted that in many cases, treatment is initiated because of a fracture. In such cases, the elevated risk of subsequent fracture in the period immediately after a fracture may be related to this first fracture rather than to a high long-term fracture risk. The differences between the two patient groups in fracture risk at treatment initiation could also affect the possible size of the treatment effect and limit comparisons between treatments [9]. It should also be noted that the treatment exposure period was limited to 1 year to take into account the possibility that a patient’s fracture risk may change over time independently of treatment effectiveness [31‐33]. Hence, this approach considers only a limited time period and does not investigate the long-term effectiveness of the different treatments. This approach does not account for variables that change over time, such as concomitant medications and comorbidities.

The strength of both methods used to assess effectiveness is that real-world persistence is accounted for. However, studies suggest persistence differs between patients receiving zoledronate and those receiving OBPs; this was not adjusted for in our study [39]. Compared with RCTs, the less intensive monitoring of patients in real-world clinical practice could be expected to accentuate differences in persistence/compliance profiles across treatments. A limitation of this study is the relatively small sample of patients receiving zoledronate, which made it hard to draw definite conclusions about the analytical approaches explored.

The present exploration of methods for assessing real-world effectiveness provides useful information regarding the challenges in estimating the real-world effectiveness of osteoporosis treatments.

The Swedish and Dutch retrospective data sources used appear suitable for obtaining robust estimates of fracture incidence. Given that the efficacy of zoledronate has been demonstrated to be similar to or higher than that of OBPs in RCTs, any failure to show that zoledronate is as effective as OBPs in clinical practice is likely a result of unobserved differences in patient characteristics (e.g., BMD), particularly when adopting an ADC approach which does not account for unmeasured confounders. The ADC between these treatments was deemed not to account sufficiently for differences in underlying fracture risk at treatment initiation. While we found the method whereby patients acted as their own controls to have potential because it adjusts for both measured and unmeasured confounders, it does not account for factors that change over time, and the necessarily short baseline risk period (only 90 days) meant that large patient samples would be needed to accurately estimate risk of fracture in this time window.

Overall, owing to the methodological challenges involved in estimating real-world and comparative effectiveness, these types of analyses should be regarded as a potential complement to treatment efficacy observed in RCTs. Each method has advantages and disadvantages; so, it may therefore be advisable to consider using both approaches in order to provide a broad overview of treatment efficacy in real-world practice.