The cost-effectiveness of family/family-based therapy for treatment of externalizing disorders, substance use disorders and delinquency: a systematic review

A systematic literature search was performed in Pubmed, ERIC, Psycinfo and Cochrane reviews (including economic trials and clinical trials). These different search engines were used because of their high quality, coverage of large databases and their focus on economic trials.

Search terms encompassed the different types of systemic therapy (Functional Family Therapy, Multidimensional Family therapy, Multidimensional Foster Care, Multisystemic Therapy, Family Behavior Therapy and Brief Strategic Therapy) but also more general classifications (systemic therapy, substance abuse treatment, family based therapy, Family based intervention, Family system intervention, Family intervention program). These terms were searched for in title and abstract and were then combined with terms referring to economic evaluations searched for in title and abstract or a Medical Subject Headings (MeSH) term (economic evaluation, cost-effectiveness, cost-utility, cost benefit, cost analysis, cost measure) and in the title (costs). Costs were searched for only in the title, and not in the abstract, because the latter resulted in many irrelevant studies. This search term was included as we noticed that although in some studies both costs and effects were evaluated, the main focus of these studies was to evaluate the costs and a smaller part was referring to the effects. Consequently, when only terms referring to both the costs and effects were included, these studies would have been missed. The search term “Economic modeling” was not explicitly incorporated into the search strategy as the modeling should be part of a cost-effectiveness, cost utility, cost benefit or cost analysis (corresponding with our aim).

Abbreviations were also included. To improve our search, MeSH terms were used, see Appendix 2 for more details.

In Fig. 1 the selection criteria are described and numbered. The criteria were applied to the studies in chronological order and when a study was excluded based on a criterion the number as shown in Fig. 1 was noted. We considered studies from January 1990 until January 2016. The selected study types were clinical/randomized controlled trials (RCT), reviews, systematic reviews and meta-analyses. The treatment needed to consist of a family/family-based intervention, targeted at adolescents (10–20 years old) with a substance use disorder, externalizing disorder or delinquent behavior. The method needed to be a cost or cost- effectiveness/benefit/ utility analysis. When studies were assessed for eligibility based on their abstracts and it was likely that they only contained cost-outcomes and no effect-outcomes, they were also included. To determine the eligibility of the full text articles, the same selection criteria were used, except that accessibility of the study was a requirement (full text available) and studies that only contained costs-outcomes and no effect-outcomes were excluded. The selection of the articles was performed by two researchers independently. Differences in selections were discussed until consensus was reached.

The quality of the studies was assessed with the British Medical Journal Checklist for authors and peer reviewers of economic submissions [20] and the Consensus on Health Economic Criteria (CHEC) list for assessment of methodological quality of economic evaluations [21] as recommended by the Cochrane reviews handbook [18]. We also consulted the critical appraisal of the studies by the NHS Economic Evaluation Database (NHS EED) structured abstract [18]. This is a database from Cochrane library consisting of structured abstracts of economic evaluations of health care interventions. Full economic evaluations were identified from a variety of sources and assessed according to a set of quality criteria. Subsequently, detailed structured abstracts were produced. In addition to the checklists, information about the economic perspective of the study (health care, societal etc.), design, country, follow-up, type of disorder, sample size, study dropout, age, gender, type, duration and intensity of intervention, time horizon, currency and price year, key features of sensitivity analyses and the included cost types were collected for the economic evaluation described in the studies. In accordance with the suggestions in the Cochrane handbook [18] five different biases of the individual studies were addressed: selection bias, performance bias, detection bias, attrition bias and reporting bias [18]. They were respectively addressed by assessing if patients were properly balanced at baseline, patients and therapists were blinded, outcome assessors were blinded, the amount of dropout in the studies and by reading the protocols of the studies.

An overview of the characteristics of the studies, participants and the interventions is shown in Table 1. Ten of the eleven selected studies were published between 2003 and 2015 [22‐31] and one study was published in 1996 [32]. Eight of the studies originated from the United States (USA) [22‐24, 27, 29‐32]. Remaining studies were initiated in Sweden [26], England [28] and Mexico [25]. All studies were (based upon) randomized controlled trials. Two pairs of studies [22, 24, 27, 29] were each based on one sample. Most of the studies compared a family/family-based intervention with care as usual [23, 26, 28, 30‐32]. MST was the most researched intervention as it was investigated in eight studies [23, 24, 26‐28, 30‐32]. In the Study of Borduin et al. [31] Multisystemic Therapy for Problem Sexual Behavior (MST-PSB) was investigated. MST-PSB is an adaptation to MST aimed at the treatment of juvenile sexual offenders. A description of the (non- family/family-based) comparator interventions is shown in Table 2. The mean number of sessions of the family/family-based interventions was between 1 and 3 times a week and the mean duration of treatment was between 12 and 31 weeks. The average follow-up time was between 6 and 300 months (25 years); only four studies followed patients for more than 1 year [26, 28, 30, 31]. Two studies were outliers in respect to the time horizon they used (8 years and 25 years) [30, 31].

Six studies were aimed at adolescents with substance use disorder [22, 24, 25, 27, 29, 32], one study investigated adolescents with a conduct disorder [26], one study adolescents at risk for continuing criminal activity [28],one study adolescents who had experienced a psychiatric crisis [23], another study adolescents who were serious juvenile offenders [30] and one study aimed to investigate juvenile sexual offenders [31]. The average sample size of the 9 studies (with separate samples) was 178 (SD = 163) with a variation between 48 and 600 patients. Follow-up attrition, when registered, was low (not more than 30 %). Average age at baseline was 15 (Standard Deviation (SD) = 1) years and between 61 and 96 % of the individuals were males.

Types of economic analyses included cost-effectiveness analyses [23, 25, 27, 29], cost-benefit analyses [22, 26, 30, 31] and cost offset analyses [28]. The difference between a cost-offset and a cost-benefit analysis is often not well-explained. A cost-offset analysis compares the monetary value of resource use with the monetary value of costs reduced by the intervention (usually health care costs). In contrast to a cost-benefit analysis which also focusses on other outcomes that are translated in monetary outcomes (like translating number of life years gained to a monetary value). In reality, cost-offset analysis is a partial cost-benefit analysis because it compares the cost of a program with the monetary value of a single outcome (i.e., avoided future health care costs). In two studies, the economic evaluation was not explicitly classified [24, 32].

Details of the interventions and outcomes of our analyses are described in Tables 3 and 4. Costs were indexed until 2014.

Only for one study [23] commentary was available from the NHS-EED. We compared the commentary on the study with our quality assessment checklists to evaluate if all issues were addressed. The quality of the studies was not only assessed for the 7 unique studies but for the 9 studies. The argument for including all studies was to differentiate between methods (e.g. analysis), display of results and discussion even though they were based on the same study. The quality assessed with the BMJ checklist was between 52 and 86 % (Table 5). The quality assessed with the CHEC list was between 50 and 79 % (Table 5). Up to date, there are no thresholds (minimum number of criteria satisfied) for these checklists to determine the difference between bad and good quality economic evaluations [18]. Overall, the outcomes on the checklists matched although quality assessed with the CHEC list was consequently lower. The largest difference in quality percentages was 20 %. All studies clearly stated their primary outcome measures. Most studies did not report all relevant costs and effects.

All studies were RCTs [22‐32]. Two of these studies [23, 32] only included patients receiving Medicaid (an aid program regarding insurances for low income families in the United States). For these studies, the RCT of the effect study contained (due to randomization) balanced samples. However, these samples were not checked for balance after the selection of participants who received medicaid, so they were at risk for selection bias. All studies had a high risk of performance bias, as blinding of both therapist and patient is impossible. For two studies [23, 32] blinding was not necessary as both the cost and outcome data were extracted from existing data systems (The medicaid billing records). Although blinding of outcome assessors is possible to reduce detection bias, no study reported to have done so. Blinding is also necessary for pre-allocation assessment. All studies were based on randomized controlled trials where allocation concealment is necessary. The studies included in this review, did not explicitly refer to the allocation concealment. Three studies were at risk of attrition bias. These three studies did not describe the number of patients that dropped out from the study [24, 27, 32]. Two studies only described the overall attrition rate [22, 25]. For one study [29] however, overall attrition rate could be extracted by using the study of French et al. [22] as it was based on the same participants. Dropout in the effect-study of Sheidow et al. [23] was low and although no dropout was described for the economic evaluation, as the economic evaluation is based on the same participants, this is expected to be low. Overall, dropout rate (when measured) seemed low. Reporting bias was assessed by reading protocols from the studies and no bias was reported. Only for two studies [22, 29] a protocol existed. Other studies did not have such a protocol, although for three studies trial registrations were present [24, 27, 28]. There were no indications of deviations from the original design. The economic evaluations did not always include all clinical outcomes that were available [23‐26, 32] as there was often only interest in specific outcomes. One study [25] excluded clinical outcomes as there was no difference between treatments in terms of outcomes and so only costs were considered (costs minimization). The exclusion of outcomes was not related to possible negative impact on the results as effects in the studies were equally or more beneficial when compared to the effects of the comparator.

Uncertainty around treatment costs was not presented in four studies as averages of these costs were used [24, 27, 30, 31]. In six studies [22, 23, 25, 29‐32] uncertainty around the (other) estimates was not (fully) addressed. In seven studies, a simple one way sensitivity analysis was used to assess the impact that changes in a certain parameter will have on the conclusions [22, 23, 26, 28, 30‐32]. In two studies, sensitivity analysis was applied by imputing missing data in different ways. Outcomes proved to be robust [27, 28]. Two studies performed scenario analyses meaning that cost estimates (surrounded by uncertainty) were increased or decreased. Data proved to be robust [26, 32]. In another study a sensitivity analysis was carried out to assess the effect which outliers in each therapy group had on outcomes, but this did not have an effect the results. In the studies of Dopp et al. [31] and Borduin et al. [31] a sensitivity analysis was applied by using plausible minimum and maximum values (obtained from other studies) for offense categories, arrest rates and discount rates. French et al. [22] used different models which assessed the effect on using more or less covariates in the models but it did not affect the results. In six of the studies cost-effectiveness/utility/benefits were assessed based on models [22‐25, 28, 32]. Four of these studies used simple regression models [23‐25, 28] and two used a more advanced least squares random effect model [19, 26]. The remaining three studies did not integrate any model in the analysis. Three studies did not report their price year (the year to which costs are indexed) [23, 24, 32].

Authors of three studies indicated that a societal perspective was adopted, where not only health care costs but also other costs, for example those associated with lost or impaired ability to work, were taken into account [22, 26, 29]. However, this was only true for the study of Olsson [26], as this was the only study to assess costs outside the health care sector. In the studies of Dennis et al. [29] and French et al. [22], the societal part was defined as using market values for calculating the costs of goods and services used. Dopp et al. [30] and Borduin et al. [31] conducted cost-benefit analyses and did not explicitely mention their perspective. Both studies focused on taxpayer benefits and expressed intagible benefits in monetary values. Cary et al. [28] used a narrow perspective as only services that were recorded by a specific data-system were included (appointments with social workers, connexion workers (a United Kingdom (UK) governmental information, advice, guidance and support service for young people aged thirteen to nineteen), reparation workers (coordinates and supports a range of interventions and community reparation projects that young people will have to undertake as part of their Referral or Community Order), parenting workers, group workers and psychologists). Sheidow [23] adopted the perspective of an institution. Other studies did not explicitly state their perspective. Most of the studies only reported treatment costs. A summary of the costs and clinical outcomes measured in the studies is provided in Table 6.

Following Drummond et al. [33], full economic evaluations should not only report costs, but also health outcomes. Four studies were classified as cost-effectiveness analyses [23, 25, 27, 29]. Only one of these studies compared treatments using an incremental cost-effectiveness ratio [29] as described for instance by Drummond et al. [33]. The cost-effectiveness analysis of French et al. [25] was reduced to a simple cost minimization analysis as the effects of both treatments after analysis proved to be similar. Sheidow et al. [27] calculated average cost-effectiveness ratios (ACER), which means that there was no direct comparison between treatments but only between the before- and after treatment costs and effects of every participant. In four studies it was explicitly stated that cost-benefit analyses [22, 26, 30, 31] were performed. Olsson [26] considered psychosocial and behavioral effects, but as no difference was observed regarding these clinical measures between treatments, these effects were excluded from the analysis. French et al. [22] did not value the health outcomes on which the intervention was focused (like reduction in days of substance use) but did value the effects of treatment on education, employment and criminal activity. Dopp et al. [30] and Borduin et al. [31] conducted a cost-benefit analysis; the cost outcome were the treatment costs and the benefits were defined as taypayer benefits, tangible benefits and intangible benefits were expressed in monetary values. Cary et al. [28] classified his study as a cost-offset evaluation. He calculated the net-benefit, but stated that his study cannot be viewed as a cost-effectiveness study as he did not measure health outcome. Two studies did not state the type of economic analyses they performed [24, 32], but did consider both costs and benefits. Mcollister [24] indicated that her study was not a full economic evaluation, as she only considered treatment costs. This is also the case concerning the study of Sheidow et al. [27], however, this study was stated to be a cost-effectiveness analysis. Furthermore, Schoenwald et al. [32] did not classify their study explicitly but considered both costs of different health care services and monetary benefits so it can be considered a cost-benefit analysis.

Four studies commented on their generalizability [23, 25‐27]. Sheidow et al. [23] reported that as their sample only consisted of youths enrolled in Medicaid, which are generally economically less advantaged, findings cannot be generalized to a more economically advantaged population. The same is true, although not stated, for the study of Schoenwald et al. [32] who also analyzed Medicaid data. The study of Olsson [26] was conducted in Sweden, where MST is twice more expensive than in the USA and may play a different role in society. MST in Sweden may be used as an alternative to nonplacement interventions as opposed to an alternative to placement interventions as found in other studies. Also in the study of French et al. [25], which was conducted in Mexico, location and small sample size were indicated as limitations for generalizability. The same was true, although not stated, for the study of Cary et al. [28] which was conducted in the United Kingdom. Also an important limitation (but not mentioned as such) were the omissions of uncertainty around the estimates in the studies of Dopp et al. [30] and Borduin et al. [31], so the results should be interpreted with caution. Furthermore, the study of Borduin et al. [31] was based on a very small (the smallest one in this review) sample size (only 48 patients) so uncertainty around the estimates (not reported) is expected to be high. Sensitivity analysis is not a solution for this problem as significance of the results cannot be determined (as the estimates in the sensitivity analysis are also subjected to uncertainty). The juvenile drug court programs, analyzed in the study of Sheidow et al. [27] are not easily generalized to other settings as they show great variation due to absence of a strict format. In addition, other settings may have different populations and salaries implying differences in costs. Almost all studies were cautious with drawing conclusions on their data. They not only recognized limitations within their research but also recognized that the number of economic evaluations is very limited and more research is needed before being able to draw conclusions [22‐28, 32].

The data from the economic evaluations were not pooled as the population, setting, outcomes, costs and interventions were not comparable across studies.