Adding non-randomised studies to a Cochrane review brings complementary information for healthcare stakeholders: an augmented systematic review and meta-analysis

Asthma is a global health problem affecting over 300 million people worldwide [1] and a major cause of disability, poor quality of life, health resources utilisation, and costs [2‐4]. Despite the availability of effective therapies, asthma control remains suboptimal, calling for the implementation and evaluation of effective asthma management interventions [5]. These include chronic disease management (CDM) programmes, which are more than simple patient education, encompassing a set of coherent interventions that centres on the patients’ needs, focuses on education and self-management, on healthcare integration and coordination provided by various professionals, as well as on improvements in communication between patients and healthcare providers [6‐12].

Summarising the evidence of CDM programmes for asthma in a systematic review is not an easy task [13]. Most often, state-of-the art Cochrane systematic reviews of interventions include randomised controlled trials (RCTs) only [14]. However, RCTs are often not the most suitable design for evaluating complex and context-dependent interventions [15‐17]. Non-randomised studies, such as controlled and uncontrolled before-after studies, are often undertaken in the pragmatic context of CDM programme implementation in communities [16]. These non-randomised studies could be very informative for the various healthcare stakeholders involved in the development, implementation, conduct and long-term sustainability of such programmes, especially in terms of contexts, key components and implementation processes. Including non-randomised studies (NRS) or non-controlled studies (NCS) in systematic reviews is challenging, however, notably when assessing study quality and the potential for selection bias and its impact on study results [14, 18]. Yet, reviews and meta-reviews have shown that non-randomised studies do not systematically lead to biased results and that there is little evidence for significant effect estimate differences between observational studies and RCTs [14, 19].

The main objective of this systematic review was to examine what complementary information could be retrieved from the addition of NRS and NCS to a published Cochrane review on asthma CDM programmes, for healthcare stakeholders involved at various levels in the development, implementation, conduct and long-term sustainability of such programmes. We also assessed how the effectiveness of asthma CDM programmes varied across included study designs.

This systematic review (augmented review throughout the text) is an extension of a previously published Cochrane review [20] evaluating the effectiveness of CDM programmes in adults with asthma, that was conducted independently from Cochrane. We used the same methodology as in the Cochrane review, except for study designs considered, risk of bias assessments and some aspects of the statistical analyses. While major methodological elements are presented thereafter, details of the methodology of the published Cochrane review are available in the original text [20].

Of the 10,593 records identified in the Cochrane review, 20 studies (from 27 articles) met all inclusion criteria and 66 articles were excluded because of the study design. Of these 66 excluded articles, 29 were rejected because no outcomes were available, children represented more than 50 % of the sample, or the intervention did not fulfil the inclusion criteria after further examination, leading to 37 remaining articles to additionally consider in the augmented review. Ten other articles were found by hand searching the reference lists of included articles. We therefore included 27 Cochrane articles and 47 additional articles, for a total of 57 studies, from 74 articles, in this augmented review (see flow diagram in Fig. 1 and key components of all included studies in the supplementary table (see Additional file 2)). These 57 studies consisted of seven CBAs [31‐37], 22 BAs [38‐59], six ITSs [60‐65] and two XSs [66, 67], in addition to the 15 RCTs [68‐82], one QRCT [83, 84], and four CBA*s [85‐88] of the Cochrane review.

Including study designs other than RCTs, QRCTs and CBA*s allowed the addition of 37 studies to the 20 studies included in the Cochrane review. In terms of study and intervention characteristics, the newly added studies allowed the consideration of programmes taking place in proportionally more countries outside of North America that included population-wide interventions with greater number of patients in a greater variety of settings and patient pools, and were of longer duration. In addition, patients in the studies of this augmented review suffered from less severe asthma. In terms of effectiveness, results from the meta-analyses confirmed that CDM programmes were significantly favourable to patients. Indeed, they improved asthma-specific QoL, asthma severity scores and lung function across all designs, with the size of effect varying by study design: CBAs and BAs showing more often larger effects than RCTs. CDM interventions also significantly improved self-efficacy scores in CBAs and BAs, while the meta-analysis was inconclusive in RCTs. In addition, we were able to perform meta-analysis of healthcare use in this augmented review, which was not possible in the Cochrane review: while RCTs did not show significant improvement for healthcare use outcomes, BAs showed lower likelihood of hospitalisations and unscheduled visits after CDM interventions. Finally, the augmented review permitted the consideration of additional outcomes: CDM interventions were associated with decreased likelihood of patients missing days off work or school in BAs, and increased likelihood of using an action plan in BAs, CBAs, and one XS.

Adding NRS and NCS to the original Cochrane review increased the applicability of the Cochrane evidence, as the variety of settings and profile of patients in the newly added studies represent probably better the general setting and target patient population of most asthma CDM programmes. Previous studies have already reported that patient populations selected for recruitment in RCTs in the respiratory field, and more specifically in asthma, represent only about 5 % of the patient population being treated by clinicians in everyday practice [90]. Although RCTs are important and necessary to establish the efficacy of new therapies and interventions, their usefulness can be limited when evaluating how complex interventions will perform in routine care in the real world [91, 92]. In fact, new guidance on conducting systematic reviews of effectiveness interventions encourages the inclusion of non-RCTs to allow for in-depth descriptions of study components and the context and process of implementing the intervention [93]. A few years after that publication, a meta-review of study design criteria in systematic reviews of health systems interventions positively showed that only one third of reviews restricted study designs to RCTs [94].

Overall, evidence from all study designs was consistent in terms of the direction of the effect; because of the increased applicability, such results strengthen the conclusion of the Cochrane review that CDM programmes have a positive effect on quality of life and asthma severity, meaningful outcomes for the everyday life of patients with asthma. Adding non-randomised studies also allowed us to run new meta-analyses showing improvements associated with CDM programmes for five outcomes (hospitalisations, unscheduled visits, days off work, self-efficacy and action plan) that could not be performed in the Cochrane review. Despite the increased risk of bias, the results, which align with the hypothesized trends, suggest evidence of reduction of hospitalisations, unscheduled visits and days off work. Such results are encouraging in the context of healthcare spending reductions. According to Schünemann and colleagues’ terminology [95], the NRS and NCS evidence can be considered as complementary (i.e. NRS and NCS provide additional information on whether the intervention works in different populations) and sequential (i.e. information not (yet) obtained or available from RCTs).

When looking at the effect sizes of RCT, NRS and NCS, the proportion of outcomes reported by authors as improved in response to CDM was higher in the newly added studies compared with the studies in the Cochrane systematic review. The newly added studies (NRS and NCS) also showed both larger effect size than RCTs and larger confidence intervals, indicating uncertainty for the results in addition to high statistical heterogeneity. These results fuel the ongoing debate about the impact of study design and associated biases on study effect size. A number of reviews have been conducted comparing effect sizes and biases of RCTs with non-randomised studies and non-controlled studies. While some reviews found that lack of randomisation or inadequate randomisation was associated with selection bias, larger effects, smaller effects, or reversed direction of effects [14, 19, 96, 97], others found little to no difference in effect sizes between study designs or according to study quality [98, 99]. As suggested by Rockers and colleagues, non-experiments can produce valid and causal effect estimates, in settings that are more natural than RCTs [92]. More research is clearly needed on the issue of study design, risk of bias assessment and effect size. A special emphasis should be put on sound methods for measuring and dealing with confounding bias in NRS studies and other biases in NCS, as well as valid methods for pooling NRS and NCS [100, 101].

Overall, randomised, non-randomised and non-controlled studies included in this augmented review were of moderate to low quality with regards to reporting, statistical and methodology standards. While risk of presenting uncertain results without knowing the direction or magnitude of the effect holds true for both non-randomised and randomised controlled trials, risk of bias and confounding is probably higher in non-randomised trials [97]. However, a study at high risk of bias does not necessarily mean the study results are actually biased. In the case of asthma CMD programmes, other authors conducting similar systematic reviews have reported similar low quality of studies [102, 103]. One reason for this low quality may be the lack of widely accepted methods for evaluating CDM programmes, which are typical complex interventions, built up from a number of interacting components, aiming to modify both providers and patients behaviours, targeting multiple organisational levels and groups, including a large number and variety of outcomes and allowing a certain degree of flexibility (i.e. intervention tailored to patients) [93]. Such complex interventions are difficult to evaluate [104, 105], emphasizing the need for sound evaluation methods to evaluate routine practice in actual settings [106]. Recent guidance has been published to help authors in developing and evaluating complex interventions in general [93, 107] and CDM in particular [106, 108]. Guidelines to improve the completeness and precisions of reporting interventions and their effectiveness have also been published for a variety of study designs, such as the template for intervention description and replication (TIDieR) [109], SQUIRE 2.0 for quality improvement studies [110], the CONSORT statement for RCTs [111], and STROBE for cohort, case–control and cross-sectional studies [112]. Future CDM evaluations should follow these available guidelines to produce high-quality evidence that can then be synthesized in systematic reviews.

The main limitation of this review is that the initial search of the Cochrane review was restricted to quantitative studies, not including all types of available evidence on CDM programmes, such as evidence from implementation, qualitative and mixed methods research which could lead to a more comprehensive understanding of complex interventions.

Evidence from NRS and NCS complemented RCT evidence from the Cochrane review. In fact, adding NRS and NCS to the original Cochrane review both increased the directness of the evidence, as the variety of settings and profile of patients in the newly added studies probably better represent the general setting and target patient population of typical asthma CDM programmes, and also strengthened the message that CDM programmes have a positive effect on quality of life and asthma severity, which are meaningful outcomes for the everyday life of patients with asthma. The inclusion of study designs other than RCTs in systematic reviews investigating the effectiveness of healthcare interventions, especially when they are complex, is increasingly considered to allow for in-depth descriptions of study components and the context and process of implementing the intervention [93, 94, 97]. However, while NRS and NCS studies provide results from interventions in real-life settings that may be valuable to patients, healthcare providers, managers and policymakers, among others, the methodological quality of the studies remains suboptimal, calling for caution in effectiveness results’ interpretation, as well as for improvements in CDM evaluation methods and reporting by health researchers.