Using meta-regression analyses in addition to conventional systematic review methods to examine the variation in cost-effectiveness results – a case study

A systematic literature search was performed to identify all English-language (online or print) publications (at any time before January 2012) of CEAs using decision analytic models to compare the costs and consequences of DES (sirolimus-eluting stent (SES), paclitaxel-eluting stent (PES), everolimus or zotarolimus-eluting stent (ZES)) versus BMS for patients who require a stent implantation due to an atherosclerotic lesion of the coronary artery. The effectiveness of the studies had to be expressed in quality adjusted life years (QALY) or in disease specific measures such as repeat revascularizations avoided, TLR (target lesion revascularization) and TVR (target vessel revascularization). Furthermore, studies were only included if they reported results in enough detail to enable separation of incremental costs from incremental effects. There was no restriction on the perspective used in the economic evaluation. Reviews, editorials and abstracts were not included in the review.

Studies were identified using electronic databases (PubMed, EMbase, NHS EED, Cochrane Library and INAHTA) and by scanning reference lists of eligible articles. The full search strategies for EMbase and PubMed are presented in Additional file 1. To ensure that all relevant publications were identified in the CRD (NHS EED and HTA) and Cochrane Library databases we limited the search terms to “stent” and “stents”. These terms were searched in “any field” for CRD and in “title, abstract, keywords” for Cochrane Library. We also included the relevant publications found in the reviews by Ligthart et al. [4], Hill et al. [2], and Neyt et al. [3].

One reviewer (LB) screened the titles and abstracts identified through the searches. The full text evaluation was performed by two reviewers (LB & FW) and discrepancies were discussed and resolved by consensus or by consulting a third reviewer (WR). Various parameters (Tables 1 and 3) were extracted from the relevant publications by one reviewer (LB). The parameters chosen in the regression analyses were the most likely general study characteristics (e.g., population, time horizon, funding) that are reported in conventional systematic reviews. In addition, we added the most important input parameters (e.g., cost of procedure, relative risk of repeat revascularization, probability of repeat revascularization, utilities) that are used in the model to estimate the cost-effectiveness. These key parameters are often varied in deterministic sensitivity analyses. Costs were converted to Euros [5] and corrected for inflation if necessary [6] to present the costs as 2012 Euros. Furthermore, we wanted to see if modelling assumptions (e.g., oculo-stenotic effect) were of influence on the incremental outcomes. All assumptions reported in the studies were monitored. Lastly, two reviewers (LB & FW) independently assessed the quality of the models using the Philips et al. checklist [7] for the assessment of model-based economic analyses. The Philips checklist is a framework based on existing guidelines on the use of decision analytic modelling in health technology assessments. The checklist is structured in three themes: a) structure, which focusses on the scope and mathematical structure; b) data, which examines data identification and uncertainty methods; and c) consistency, which assesses the overall quality of the model based on the publication. Both overall study quality and the quality per theme were given a score from 0-100 %, which was calculated by dividing the sum of the questions answered positively by the total number of relevant questions. Since some questions were not relevant for all studies (e.g., questions concerning quality-of-life values) the denominator could differ between studies.

The influence of modelling methods, the choice of parameters and the quality of the models on the main outcomes (incremental costs, incremental QALYs and absolute risk reduction repeat revascularizations) were analysed both quantitatively and qualitatively. Associations between parameters and the outcomes were assessed by identifying outliers found on cost-effectiveness planes. Furthermore, several bivariate linear regressions were estimated to confirm the associations and also to measure the influence of other parameters on the outcomes. Including associations that could be predicted beforehand (e.g., type of lesion, price stent) are included in the regression analyses since it could be seen as a validation check if the analyses also show these associations. Multivariate analyses with all of the parameters that were significant in the bivariate analyses could not be performed due to a high frequency of missing values caused by incomplete reporting.

We included every subgroup or sensitivity analysis found in a study as long as incremental costs or incremental effectiveness were provided or could be calculated. As a result, our meta-regression analyses were based on many more observations than the number of studies that were included. Since Hill et al. [2] provided more than 30 % of the observations used in our study; we incorporated study ID as a random effect in the regression models. Some studies reported both incremental effects and incremental QALYs for a specific analysis. Since the incremental costs associated with both outcomes is the same we only included one of the two analyses for the regression analyses on the incremental costs to avoid double counting. Data management and all statistical analyses were performed with SPSS 19.0 (SPSS Inc., Chicago, IL, USA). The level of measurement was ordinal or ratio, depending on the covariate. The model assumptions and study characteristics (e.g., funding) were measured at an ordinal scale. Input parameters such as the probability of repeat revascularization were measured at a ratio scale. Conclusions about statistical significant were based on an alpha level of 5 %.

In most analyses, DES was more expensive (88 % of analyses) and more effective in both QALYs and repeat revascularizations avoided (99 % of analyses) than BMS. Most of the 16 studies [2, 8, 10, 11, 14, 16, 21, 23] concluded that DES is not cost-effective for all subgroups since the incremental QALYs did not offset the incremental costs. However, many concluded that DES was more cost-effective in high-risk patients. The ICER varied considerably between and within studies: from DES being dominated by BMS [14, 21] to DES being dominant in specific analyses [2, 8, 10, 11, 15, 19, 22]. Figs. 2 and 3 present the variability of the incremental costs and effects of the studies using repeat revascularizations avoided or QALYs as an outcome measure, respectively. The mean values of input parameters stratified by the type of study outcome are presented in Table 2.

We also assessed the quality of the models of all studies using the Philips et al. [7] checklist. Studies appeared to score higher on the theme structure (63 % ± 16 %) than on the other two themes, data (57 % ± 22 %) and consistency (55 % ± 21 %). The average overall quality of the models was moderate (59 % ± 15 % of a maximum possible score of 100 %).

Based on 124 separate analyses (9 studies), the number of repeat revascularizations avoided (the absolute risk reduction in repeat revascularizations) with DES also varied considerably (Fig. 2) between and within studies (range: −0.0001, 0.19), which resulted in variation in the ICERs. The overall conclusions of most of the studies corresponded with the 124 separate analyses (Table 3). The regression analyses showed that the relative risk reduction of repeat revascularizations and the initial probabilities of restenosis were positively associated with repeat revascularizations avoided. Furthermore, a more complex vessel or lesion was associated with higher relative risk reduction and initial risk of restenosis after a percutaneous coronary intervention with BMS. Consequently, this leads to an increase in repeat revascularizations avoided and DES becomes more effective. Furthermore, the number of stents was also positively and significantly associated with repeat revascularizations avoided, probably because it is a proxy for subgroups who have a higher risk of developing restenosis due to diabetes, lesions and vessels characteristics. These factors could have been predicted beforehand since subgroup analyses and sensitivity analyses of the individual studies show the same conclusions.

Besides these factors that could be predicted beforehand, with the meta-regression analyses we were able to find a negative association between overall quality of a model and repeat revascularizations avoided. Furthermore, the theme data was also negatively associated with this incremental outcome. Consequently, models with a higher quality led to less favourable results for DES.

Figure 3 presents the incremental QALYs and incremental costs for 384 separate cost-effectiveness analyses (11 studies). This Figure shows that Shrive et al. [17] and Remak et al. [20] clearly found a larger incremental QALY gain than the other studies.

Again, the meta-regression analyses found associations with incremental QALYs that were expected (Table 4). Relative risk reduction of repeat revascularizations and the initial probability of restenosis after BMS were associated with a greater QALY gain, as seen in individual sensitivity analyses [2, 14, 15, 21, 22, 24]. Furthermore, analyses showed that non-elective patients, patients with a high risk of a repeat revascularization, patients with complex vessels or lesions or older patients will benefit more from DES, something that was also recognised in the individual studies [2, 12, 17, 21, 24]. In addition, we found a significant positive association between time horizon (continuous) and incremental QALYs. This was also found by Hill et al. [22] and Ekman et al. [15] who varied the time horizon in the sensitivity analyses.

Studies [2, 17] that have explicitly mentioned that they have assumed that the occurrence of repeat revascularizations within the time horizon is the result of restenosis and studies assuming that repeat revascularization rates are based on angiographic follow-up have estimated significantly higher incremental QALYs. Angiographic follow-up leads to inflated estimates of clinical effectiveness compared with clinical follow-up since not clinically significant restenosis results in “unnecessary” repeat revascularizations when angiographic follow-up is performed. Consequently, the difference in repeat revascularizations will be overestimated (oculo-stenotic effect) [25]. Some studies use “real-world” [8, 10, 11, 21] follow-up data and consequently report lower estimates (visible in Figs. 2 and 3) than other studies such as, Remak et al. [20] that used angiographic follow-up [12, 15, 17, 23]. This phenomenon is described earlier by Eisenberg et al. [26], who concluded that cost-effectiveness studies using angiographic follow-up overestimate the cost-effectiveness of DES.

The meta-regression analyses showed that studies using real-world evidence compared with angiographic follow-up leads to a reduction in incremental QALY gain. The added value of meta-regression analyses is limited in explaining the variation in incremental QALYs, although it identified modelling assumptions that were significantly associated with incremental QALYs.

Figures 2 and 3 show that there was large variation in incremental costs (range: €-4070 to €3506). Regression analyses (Table 5) confirmed associations (cost parameters and population characteristics) that were seen in the individual studies [2, 8, 12, 17, 20, 21, 24]. The analyses showed that probability of restenosis after BMS, the reduction in restenosis risk by DES, the difference in stent price, and the number of stents used were important parameters influencing the incremental costs. Both input parameters varied considerably between the analyses: the difference in stent costs ranged from €0 [8, 19] to €2685 [17] and the number of stents varied between 1 [22] and 2.6 [19] stents per procedure depending on the type of patient.

On a meta-level we were able to conclude that funding and the type of cost-effectiveness analysis was associated with incremental costs. Funding was provided by the stent manufacturer in five [15, 17‐20] of the 16 studies and three of them [15, 17, 20] concluded that DES was cost-effective compared with BMS. Of the studies that were not funded by a manufacturer (N = 8) only one [10, 11] of them concluded that DES could be cost-effective. Studies that were not funded estimated on average higher incremental costs than studies that were (p < 0.05). Furthermore, some associations with incremental costs are recognised from scenario analyses performed by studies. The directions of the following associations are confirmed by the regression analysis but not significant. According to Jahn et al. [10, 11] it is important to incorporate wait time into the model since it leads to a decrease (−734, 95 % CI:-1690;223) in incremental costs. A time horizon shorter than 12 months was associated with higher incremental costs (479, 95 % CI: −1024;65); Hill et al. [22] estimated costs and effects for different time horizons and showed that a longer time horizon led to lower incremental costs. This is likely because of the continuing treatment effect of DES in the subsequent years which would increase in the number of repeat revascularizations avoided compared with BMS. This increase in reduction of repeat revascularization would further offset the cost of the initially more expensive DES.

Meta-regression analyses showed also that the number of repeated revascularizations avoided explained a large proportion of variation (R² = 0.53). As shown in Fig. 2, there appeared to be a linear association between incremental costs and repeat revascularizations avoided. Incremental QALYs (Fig. 3), on the other hand, was not associated with incremental costs (R² = 0.001), probably since incremental QALYs are determined by several factors including repeat revascularizations avoided, life-years gained and quality of life values.

Despite the fact that the quality of the models was assessed by two independent reviewers it was difficult to judge the quality due to subjectivity of the questions; this problem was been recognized in the past [28]. Furthermore, to provide studies with a score between 0 and 100 % we needed to assume that all questions of the checklist were equally important. Thus studies could obtain a reasonably high score if less informative/important questions were fulfilled. In addition, the quality of the models was based on the documentation of the model and therefore it is possible that studies that scored low did not transparently present model details, however the actual model could be of high quality. Regardless of these limitations, we found a statistically significant association between quality and the outcome repeat revascularization.

Furthermore, title abstract screening was performed by one reviewer which could be seen as a limitation of the study. However, checks of whether the studies included in previously published reviews were also identified with the search, increased the sensitivity of the search and thereby reduced the chance of missing relevant publications. Full assessment and assessing the quality of the model using the Philips checklist was performed by two reviewers independently.

Another limitation of our study is that all 508 analyses were analysed as independent observations even though in reality these 508 analyses were based on 16 studies. We have used study identification number as a random effect in the regression models to address this problem.

In this case study, linear regression models were used to estimate the associations of study characteristics on the outcomes (incremental costs, incremental QALYs and repeat revascularizations avoided) since the number of observations was large. However, regression models could be improved by first considering if the dependent variable (e.g., incremental costs) can best be modelled using a different function (e.g., gamma).

Moreover, meta-regression analyses (bivariate or multivariate) help to explain variation in outcomes, however it also identified associations that were not expected a priori. For example, type of study was associated with the incremental costs, which is not logical since the type of study mainly influences the incremental effects. Covariates that are on beforehand implausible (e.g., type of study and incremental costs) should not be included in future meta-regression analyses since it leads to false positive outcomes.

In addition, transparency in documentation is a major issue leading to a high frequency of missing values that made it impossible to perform multivariate analyses with all of the parameters that were significant in the bivariate analyses. Consequently, we were unable to: 1) take into account interaction effects, 2) determine the most influential covariates, and 3) create a prediction model. A solution could be to include a smaller number of input parameters with only common input parameters (e.g., cost of procedure, time horizon etc.). However, this will lead to fewer associations between outcomes and covariates.

Transparent reporting is crucial in this field and would solve the problem of missing values for systematic reviews such as this. A recently published review on the challenges of modelling the cost-effectiveness of cardiovascular disease interventions has recognized the same problem [29].

Lastly, we did not include the studies published after January 2012. However, we expect that including newer studies that met inclusion criteria (i.e., estimating the cost-effectiveness of DES versus BMS using modelling methods) do not have an impact on the results of our case study showing that using meta-regression analyses could be useful method in addition to conventional systematic reviews.

To improve this case study lessons can be learned from meta-regression analyses and meta-analyses that are performed for the clinical effectiveness of interventions. More specifically, it could provide guidance in how to handle missing data [30], how to treat study heterogeneity, how to include covariate interaction [31]. In addition, it shows limitations of the methods [1].