Reassessing the Evidence Hierarchy in Asthma: Evaluating Comparative Effectiveness

The Global Initiative for Asthma (GINA) publishes management and therapeutic recommendations that draw heavily on cRCT evidence [3]. To be included in cRCTs, patients must fulfill strict GINA-based diagnostic criteria to ensure that observed effects are not obscured by other actors (“confounders”) introduced, for example, by inclusion of patients with obstructive lung disease other than asthma (eg, chronic obstructive pulmonary disease [COPD]). GINA stipulates that when evaluating a patient’s response to asthma therapy, the following characteristics are consistent with an asthma diagnosis: an increase in forced expiratory volume in 1 s (FEV₁) of ≥12% (and 200 mL) after administration of a bronchodilator (ie, indicative of reversible airflow limitation), an improvement in peak expiratory flow (PEF) of 60 L/min (or ≥20% of the prebronchodilator PEF) after inhalation of a bronchodilator, or diurnal variation in PEF of more than 20% (with twice-daily readings, >10%) [3].

The generalizability of criteria used to inform the GINA therapeutic recommendations was evaluated through a survey of respiratory health carried out in Wellington, New Zealand [18]. The New Zealand researchers identified the eligibility criteria common to all cRCTs published in the past 30 years that were cited by GINA and were designed to evaluate asthma drug efficacy in a minimum of 400 adult patients (n = 17 of the 215 cRCTs cited as level A or B evidence by GINA). Common eligibility criteria in these trials included a diagnosis of asthma, age older than a lower age limit, and bronchodilator reversibility. Other inclusion criteria were a specified FEV₁ range, inhaled corticosteroid (ICS) use, specified symptoms or use of rescue drugs, an upper age limit age, and peak flow variability. The proportion of Wellington survey participants with current asthma and full questionnaire responses and pulmonary function testing (n = 127) who met the eligibility criteria is summarized in Table 2; only 29% met the GINA-recommended reversibility diagnostic criterion, and 44% met the PEF variability criterion [18].

A similar study was conducted in Norway to evaluate the extent to which a real life obstructive lung disease population met criteria commonly used in cRCTs [20]. A minority of patients met the following commonly used asthma eligibility criteria: FEV₁ 50% to 85% of predicted (37.1% eligible), reversibility of 12% within the past year (14.9% eligible), absence of comorbidity (9.6% eligible), and nonsmoker (or ex-smoker with a nicotine burden <10 pack-years; 5.4% eligible). If patients were also required to be symptomatic and to have regular ICS usage, the percentage of eligible patients fell further to 3.3% (Table 2) [20]. Thus, in asthma, the typical cRCT population represents a small minority of patients with asthma treated in real world everyday clinical practice.

This finding holds true in other areas of respiratory disease. For example, a recent prospective cohort study found that of patients treated for allergic rhinitis in everyday practice, only 7.4% of the 311 patients examined would have been eligible for major placebo-controlled RCTs of persistent and intermittent allergic rhinitis [21]. The most common reasons for which the real world patients with allergic rhinitis would have been excluded from a cRCT were the absence of an allergy diagnosis based on skin testing and/or serum-specific IgE testing, insufficient disease severity, and the presence of comorbidities.

Concerns surrounding the limited representative nature of cRCTs arise where there are data to suggest that cRCT findings may not hold true in particular subgroups of the population. Subgroups of particular note within the larger asthma population include those with poor inhaler technique, current smokers, obese patients, those with comorbid conditions (eg, rhinitis), and those with low adherence to therapy [11‐14, 22‐28]. When considering inhaler handling, for example, data suggest a link between poor technique and poor asthma control, as improper use of pressurized metered-dose inhalers (pMDIs) for the delivery of ICS is associated with decreased asthma control [13].

Smoking and severity of rhinitis are also important determinants of asthma control, as patients with severe rhinitis and/or higher average cigarette use exhibit poorer control [11]. Cigarette smoking is known to reduce the effect of ICS therapy [27‐30]. Other factors that have been shown to affect patients’ response to asthma therapy are obesity, likely through inflammatory mechanisms [23], and presence of comorbidities such as COPD and heart failure [15]. It is often difficult in practice to differentiate between asthma and COPD, whereas only patients with a clear-cut diagnosis of asthma are included in cRCTs.

Therefore, the importance of real world factors in asthma (eg, smoking, comorbidities, patient adherence, and inhaler technique) should not be overlooked, as they may explain the wide gap between the level of asthma control that can be achieved in cRCTs [6] and the frequently disappointing results observed in observational studies carried out among less selected populations [31]. However, data on the comparative effectiveness of therapies in these subgroups remain lacking. Appropriately designed studies to address these unanswered research questions may reveal differential effectiveness of therapeutic options in particular patient subgroups and could be used to help guide more tailored, individualized asthma management.

The term pragmatic trials was first used in 1967 by Schwartz and Lellouch [36] to describe trials designed to help choose between care options. They contrasted this with explanatory trials that were designed to test causal research hypotheses (eg, that an intervention causes a particular biological change). Schwartz and Lellouch [36] considered there to be a continuum rather than a dichotomy between explanatory and pragmatic trials, and characterized pragmatism as an attitude to trial design rather than a characteristic of the trial itself—an attitude that has come to be understood as one that maximizes applicability of trial results to usual care settings, relies on unequivocally important outcomes (eg, mortality and severe morbidity), and is tested in a wide range of participants [37‐40].

A trial using this approach for allergic rhinitis found that guideline-based treatment was more effective than free treatment choice [41]. Thus, pragmatic clinical trial designs offer a means of testing a hypothesis in a more naturalistic, real world setting than cRCTs by modeling and reflecting everyday clinical practice in their scheduling (eg, longer treatment exposure) and in their approach to patient recruitment (eg, including patients with relevant comorbidities). While consenting patients are still assigned randomly to predefined study arms, pragmatic trials have broader inclusion criteria [42] than cRCTs and tend to be longer in duration [43, 44].

The international UPLIFT trial was a pragmatic trial designed to evaluate the long-term effect of tiotropium compared with placebo on lung function in patients with COPD [43]. The UPLIFT design was more naturalistic than that used in cRCTs, as it allowed patients to continue taking their standard background therapy (any respiratory medications except anticholinergic drugs) throughout the 4-year trial. Moreover, UPLIFT used a true intention-to-treat approach when evaluating the mortality end point; thus, if patients discontinued study medication during the trial, they were still included in the final mortality analysis. It is unfortunate that the other trial end points were not treated in the same manner [43]. Interestingly, UPLIFT found the rate of lung function decline in the “placebo” group (in which two thirds of patients were receiving an ICS and/or LABA through continuation of their standard background therapy) to be similar to that observed in the ICS, LABA, or ICS/LABA groups of the TORCH study, another long-term trial comparing these treatments with placebo [45].

Another pragmatic trial of note, commissioned by the United Kingdom government, is ELEVATE, an equivalence trial evaluating leukotriene receptor antagonists in primary care at steps 2 and 3 of the national asthma guidelines [44]. Broad inclusion criteria were defined and effectiveness outcomes were measured over a 2-year outcome period; the primary outcome measure was asthma-related quality of life (QoL), a patient-oriented measure of effectiveness. Furthermore, the pragmatic trial design ensured continued patient participation in the study even if patients did not receive and complete the full prescribed regimen—a true intention-to-treat approach. As a result, the dropout rate was only 4% over 2 years in ELEVATE [44], which compares favorably with cRCT rates (eg, 25% in GOAL [46] and 16% in IMPACT [47]). Because patients continued to receive care at their usual practices, ELEVATE achieved high levels of complete data for the primary end point and high levels of clinical data from routine practice: more than 90% of patients supplied data at 2 years for the primary end point, and more than 95% for health care resource and asthma exacerbations. Previous cRCTs comparing step 2 and 3 asthma therapies have yielded inconsistent results on the relative efficacy of the treatment options available [48‐54]. In ELEVATE, leukotriene receptor antagonists were equivalent to the comparators at 2 months with regard to QoL, and although equivalence in QoL was not shown at 2 years, there were no significant differences in secondary measures at either time point [44].

For seasonal allergic rhinitis, a cluster randomized trial in primary care showed that patients receiving a treatment following the international consensus on rhinitis demonstrated a large improvement compared with those receiving free treatment choice [55]. In order to be closer to patients’ needs, the primary end point was QoL.

Pragmatic trials also offer the possibility of evaluating adherence in a more naturalistic setting than cRCTs. Not only do cRCTs often demand unrepresentatively high levels of treatment adherence, but their inherently interventional nature can also artificially drive adherence.

One approach to more naturalistic adherence data capture is illustrated by a pragmatic trial designed to evaluate the impact of dosing regimen on adherence to asthma medication. Price et al. [56] compared adherence rates to once-daily and twice-daily mometasone furoate by capturing adherence data using patient self-report and dose counters. Patients with poor adherence remained in the study and thus were included in the final analysis. Using this approach, a discernible difference in adherence rates was measured between treatment arms, with greater adherence recorded for the once-daily regimen [56].

Evaluating cost-effectiveness of health care interventions is complicated by the difficulties in placing a monetary value on improved QoL and reduced morbidity and mortality. Indeed, drug treatment to prevent morbid events is rarely cost-saving or cost-neutral, and the ultimate questions that have to be addressed are as follows:

Understanding the most appropriate and meaningful data on which to base cost-effectiveness evaluations can also be challenging. A study designed to evaluate the external validity of published cost-effectiveness studies compared the data used in the published studies (typically based on cRCTs) with observational data from actual clinical practice. The authors concluded that cost-effectiveness evaluations based solely on cRCT data lack external validity and, as they do not represent patients in actual clinical practice, should not be used to inform prescribing policies [58]. In light of this, treatment and health technology assessments should move away from analyses in carefully screened populations toward actual cost-effectiveness trials using real world clinical data [57].

Nonetheless, however naturalistic the design of a pragmatic trial, it still requires patient consent and the involvement of “trial-minded” physicians. As such, pragmatic trials will still deal with a defined subgroup of the overall patient and physician populations, and their protocols still require closer monitoring of clinical and biological parameters and more frequent contact with health care professionals than occurs in standard clinical practice [9, 59‐61].

As pragmatic trials are designed to study real world practice, they are less effective than efficacy trials, sacrificing internal validity to achieve generalizability [62]. Poor design and/or execution (true also of some cRCTs) can bias results toward similar efficacy across treatment arms and therefore bias the study toward a finding of equivalence. Better designed pragmatic trials often include objective outcome measures (eg, survival, test results) and subjective measures (eg, QoL surveys), which, if broadly consistent, can diminish concerns about potential bias [62].

Additional challenges arise from the fact that the very characteristics of real world practice that pragmatic trials are designed to capture—including variable adherence, use of concomitant therapies, presence of comorbidities, changeable symptoms over time—tend to reduce measureable differences between therapies. Such concerns highlight the benefit of including both an intention-to-treat and a per-protocol analysis to allow regression to equivalence [63, 64]. Some of these shortcomings can be addressed—to varying degrees—by observational studies.

As defined by the National Centre for Biotechnology Information, an observational study is a “type of nonrandomized study in which the investigators do not seek to intervene, instead simply observing the course of events” [65]. As such, observational studies using clinical databases offer another method of studying the comparative effectiveness of outcomes as evaluated in real world patients in a noninterventional, naturalistic setting. They also provide a means to study, characterize, and better understand real world prescribing practices and adherence to guidelines in clinical practice. Observational studies involve accessing, collating, and analyzing information held in patient records and can be cross-sectional or longitudinal in design [9]. Although they are limited by the lack of treatment randomization and potential bias through subjectivity of treatment choice, their use of routine clinical data gives them high external validity.

The General Practice Research Database and the Doctors Independent Network (DIN-Link) Database in the United Kingdom are clinical databases that have been extensively used for research [66‐69]. They comprise patient records collected over years, not months, thereby allowing investigation of longitudinal treatment effects. These features are invaluable in hypothesis generation and testing and can also help refine the design and powering of RCTs [9].

Observational studies can help evaluate guideline implementation in everyday clinical practice. In the United States, an observational study was carried out using an integrated managed care database to characterize the patterns of care observed in patients prior to emergency department (ED) treatment of acute asthma [70]. The study was motivated by the limited data on resource use prior to ED attendance and the recognition that better understanding of treatment patterns prior to an ED visit may help identify opportunities for improved interventions. The study explored adherence to guideline recommendations through evaluation of ICS therapy in the year prior to the ED visit and through quantification of short-acting β₂-agonists and oral corticosteroids and rescue medications in the year before and in the month after the ED visit. Also investigated was the impact of the acute care intervention in the ED on altering the prescription of ICS and other asthma medications in the 2 months after the ED event. The study demonstrated a high dependence on rescue medications—short-acting β₂-agonists and oral corticosteroids—in this population prior to ED attendance, and that the ED event resulted in only an incremental short-term improvement in ICS-containing controller treatment. Such characterization of prescribing patterns in real world clinical practice requires observational, noninterventional methods and cannot be achieved through cRCTs [14].

Another area of asthma management in which study alternatives to cRCTs can provide useful information is in the evaluation of inhaler type. Asthma cRCTs typically train recruited patients in inhaler technique and often require trial participants to be able to demonstrate and maintain proper inhalation technique throughout. However, in real world practice, many patients use their inhaler devices incorrectly, and proper inhaler technique is infrequently reinforced [14, 71, 72].

The REALITY study used the General Practice Research Database to evaluate the comparative effectiveness of different inhaler types as used in everyday asthma management [73‐75]. Participating patients commenced or increased ICS therapy via a range of different inhaler types (pMDIs, breath-actuated metered-dose inhalers [BAIs], and dry powder inhalers). No requirements were placed on patients’ inhaler training beyond routine standard care. Significant differences in the odds of achieving successful asthma control were found for both BAI- and dry powder inhaler–treated patients compared with patients using pMDIs [74]. Moreover, these differences had significant health economic implications, with BAIs being on average more cost-effective than pMDIs [73].

Although well-designed observational studies have good external validity, they lack the internal validity of cRCTs. One of the main challenges of observational studies is ensuring adequate treatment group comparability, but expertise is growing and methodologies are continually evolving that assess baseline comparability of the study cohorts and also validate outcomes for consistency across multiple subgroups [76]. When study cohorts are similar at baseline, outcome analysis can proceed with suitable statistical adjustments being made for any characteristics that are statistically or clinically significantly different between cohorts or are strongly predictive of the outcome [77, 78]. Where cross-sectional baseline data reveal substantial differences between groups, a matched cohort analysis (or other suitable methodology, such as propensity-based matching) should be used. Patients are matched on key demographic and clinical baseline characteristics to minimize any differences in baseline disease severity and to ensure that strongly confounding baseline effects are comparable across treatment groups, thus allowing study outcomes to be appropriately interpreted [79].

In this regard, Price et al. [77, 78] employed a matched cohort approach to an effectiveness and cost-effectiveness comparison of extra-fine hydrofluoroalkane beclomethasone dipropionate (EF HFA-BDP; QVAR [Teva Respiratory, Horsham, PA]) pMDI, fluticasone propionate (FP) pMDI, and chlorofluorocarbon-BDP pMDI therapies. To ensure similarity of asthma severity at baseline, patients in the EF HFA-BDP and FP [77], and in the EF HFA-BDP and BDP [78] groups were matched on important demographic and asthma-related baseline characteristics prior to outcome evaluation (Table 3). The two separate matched cohort analyses found that in a real world setting, patients receiving EF HFA-BDP had a similar or better chance of achieving asthma control at lower prescribed doses than with FP [77] or chlorofluorocarbon-BDP (Table 4, Table 5, and Fig. 1) [78]. These findings were reinforced by similar outcomes in the unmatched cohort analyses and the consistency of subanalysis results for age group and smoking status.