Systematic review of economic evaluations of interventions for high risk young people

The aim of this systematic literature review is to identify and critique full economic evaluations of interventions for high risk young people with the purpose of informing the design of future rigorous economic evaluations of such intervention programs.

A PRISMA compliant systematic search of the literature was conducted with the assistance of an accredited librarian [17] (see Additional file 1). The search comprised two steps as per the guidelines provided in the Cochrane Collaboration Handbook on Systematic Reviews of Interventions [18]. First, a preliminary search was conducted to identify original articles in the following electronic databases: Econlit, PubMed Clinical Queries and Scopus. A full search was then conducted in CINAHL, Embase, Global Health, Medline, PsychInfo and Social Work Abstracts. Given that the literature often refers to the need for more economic evaluations in the area of youth programs [14, 19, 20], coupled with the lack of economic evaluations of interventions for high risk young people that address multiple domains in the literature [4], the terms used in the search strings to identify high risk young people and interventions were designed to generate as many relevant interventions as possible. Terms for full economic evaluations were then added to the search strings. Full economic evaluation is used here to refer to cost-benefit analysis (CBA), cost-effectiveness analysis (CEA), cost-utility analysis (CUA) and social return on investment (SROI), a type of CBA [16]. Searches were limited to English language and the years 2000-April 2018.

Second, a grey literature search was conducted to identify articles not located by the electronic database search. Search terms for high risk young people and full economic evaluations were applied to a search of the following websites: Centre for Aboriginal Economic Policy Research (CAEPR), National Institute for Health and Care Excellence (NICE), Health Technology Assessment, Washington State Institute for Public Policy (WSIPP), HealthInfoNet, Closing the Gap Clearinghouse, LIt.search, Australian Policy Online (APO), Virginia Commonwealth University Library, Google and Google Scholar. A hand search of reference lists of identified articles was also conducted.

Figure 1 shows the search process, from the databases searched to identify records to the exclusion criteria applied in the initial screening, and finally, the eligibility criteria used to assess full text articles for inclusion in the review.

Two reviewers (KE, RL) independently screened the titles and abstracts of retrieved articles following the PICOS criteria for inclusion & exclusion specified in Table 1.

After excluding articles that did not match the inclusion criteria, the full-text versions of the remaining articles were obtained and assessed for eligibility.

Despite all included economic evaluations being rated average or good (Additional file 3), many suffered from methodological limitations, emphasising the challenges often faced in the conduct of economic evaluations of public health/community interventions. A number of factors impact on the reliability of conclusions drawn in economic evaluations such as uncertainty in effect size, accuracy of cost information, scope, modelling and timing of the analysis, perspective and choice of discount rate. A considerable number of included studies failed to follow best practice for conduct of economic evaluations and did not adjust for differential timing, or perform incremental or uncertainty analysis. However, three key methodological challenges were identified from the literature as representative of the methodological issues more specific to the conduct of economic evaluations of interventions for youth populations: 1). attribution of effects; 2). measuring and valuing outcomes; and 3). identifying relevant costs and consequences.

Attribution of effects (1.) refers to the strength of the evidence around causality and is reflected in the range of study designs. RCTs are the preferred source of evidence to determine estimates of intervention effects, but these are often difficult to conduct in a public health environment (as opposed to more controlled clinical or laboratory settings), so alternative approaches are sometimes needed. In addition, outcomes are often measured in the short term whereas the impact of a public health intervention may occur in the long term. Measuring and valuing outcomes (2.) refers to the approach taken in an economic analysis. In health, for example, the outcome of interest may be quite narrow and directed at specific individuals or groups such as cost per life years gained for a treatment or cost per cases prevented for immunisation and require a CEA. Alternatively, many public health interventions have effects on individuals outside the target group and thus  need a broader measurement of outcomes, requiring a CBA. Finally, identifying relevant costs and consequences (3.) refers to which costs and benefits are included in an economic evaluation. Some studies may not have collected the data required by an economic evaluation thus limiting the cost and consequences that can be identified, or necessitate modelling based on assumptions derived from other sources or the literature. The impact of public health interventions is often broad and can have a ripple effect, so the costs and benefits may be related to a number of sectors and this needs to be addressed in the economic evaluation. It is these three methodological challenges that will form the focus of this review.

The methodological characteristics of the included economic evaluations are described below. Firstly, the methodology of the CBAs and SROI will be examined, followed by the CEAs and CUA, in order to best facilitate comparison across like studies.

The review identified 13 CBAs and one SROI of interventions for at risk young people [22‐34].

The review identified 17 CEAs and one CUA of interventions for at risk youth.

What is striking about the 29 economic analyses included in this review is the heterogeneity of approaches to economic evaluation represented. This is unsurprising, given that most were conducted retrospectively and had potentially not planned to conduct an economic evaluation. The results described above document the different analytic characteristics of the studies and the economic analyses conducted of them. Often the analyst is unable to predetermine the type of analysis required as this will depend on the results of the clinical trial or the quality or availability of the data. As some studies demonstrated, different approaches are sometimes employed together in an attempt to better address the question at hand [30, 34, 26]. Alternatively, the decision maker may determine the choice of economic evaluation and narrow the perspective to address only their concerns or interests [35]. All or any of these considerations will impact on the choice of analysis and the way in which the cost and consequence data is measured and valued.

As the results of the review demonstrate, there are three main challenges when conducting economic analyses of public health programs for at risk youth:

Study design is a fundamental consideration when attributing causality of an impact to a public health intervention. Ideally, economic evaluations should be an integral part of the planning process of any intervention, allowing for prospective costing data and impact data collection. This is particularly important for interventions designed to address the complex needs of high risk young people. Inclusion of a control group and longitudinal rather than cross-sectional data collection facilitates the attribution of causality to the program [22, 25]. While RCTs may not always be ethically appropriate for a high-risk population, where a no treatment control group is preferred and matched control groups are difficult to source, study designs like multiple baseline design can provide a partial solution to this problem, particularly for a program implemented in various locations.

Complex health and social problems require complex interventions that also require appropriate evaluation designs [53]. Economic evaluations of these interventions also need to capture this complexity. Most interventions and their evaluations included in this review addressed too few domains of risk, despite the mutually reinforcing nature of these domains. Interventions for at-risk youth that focus on a single outcome, as was the case with many CEAs, is contrary to the nature of the treatment, which will have multiple effects on clients and their communities [44, 54]. Quite a number of economic analyses suffered from small [24, 44, 45, 47, 48] or non-generalisable samples [22, 28, 35, 48, 50, 54, 55], missing sample data [37, 42, 50], and poor quality or lack of evidence-based evaluations [43, 50]. This requires assumptions to be made about effect or impact. Sheidow et al. [48] addressed the problem of a small heterogeneous sample by averaging data rather than using individual data; a problem better addressed by larger sample size. Similarly, a lack of costing data means estimations must be made about costs [34]. Limited evidence of effect in program evaluation data or limited costing data can result in an under or over-estimation of the impacts of a program, impacting on the quality and precision of the economic analysis. While modelled parameters can be used to address missing or incomplete data, often the results of such an analysis can only be suggestive of actual impact.

The time frames used to evaluate interventions are another source of weakness for economic evaluations. Only eleven economic evaluations utilised data from follow-up of greater duration than 12 months [22‐25, 30, 32, 33, 46, 49, 50], making it difficult to have confidence that an intervention would remain cost effective or cost beneficial in the long-term. For example, in an analysis of a smoking prevention intervention, without real data or longer term follow-up, there is no evidence to determine whether an intervention delayed or prevented smoking uptake [43], or whether cessation or any other behaviour of interest is maintained into the future [51]. Extrapolating effects into the future is particularly important in a youthful population. Youth is a time of transition and development, so program evaluations should strive to measure costs and outcomes across all time frames: short; medium and long term.

This review identified 29 empirical studies employing 32 economic evaluations comprising 13 CBAs (41%), 17 CEAs (53%), a CUA (3%) and a SROI (3%). As a consequence, outcomes in the economic evaluations varied: CEAs were based on a great diversity of outcomes, sometimes recorded in natural units such as number of incidences of substance abuse [48]; initiation of tobacco use, instances of delinquent behaviour [33]; number of (established) smokers prevented [31, 37]; days of abstinence, and so on (see Results 2b. above). Some analyses converted data into utility-based outcomes such as ICER per DALY avoided [26], life years saved [37], QALYs gained [37], criminal activity free years (CAFYs) [42], and depression free days (DFDs) [45].

Given the broader perspective usually associated with a CBA, in the 13 included CBAs and SROI, the number of outcomes measured varied from one to 19. However, for the nine analyses conducted from a societal perspective [22‐25, 29, 31, 33, 34], the broadest and most comprehensive perspective, only four measured and valued more than three outcomes [22, 24, 25, 29]. Outcomes varied considerably across the CBAs and SROI based on the domain of risk being addressed by the intervention or the outcome of interest (See Results 1b above).

The perspective chosen in an economic analysis influences the types of costs and consequences included in the evaluation. Economic evaluations included in this review were conducted from a number of perspectives ranging from a broad societal perspective to employer, government, public sector, healthcare or even a single school perspective. Some incorporated multiple perspectives. A broader societal perspective in a CBA allows intersectoral outcomes to be included, thus enabling a more comprehensive economic evaluation. For example, French et al. [29] monetised 19 different outcomes from a broad range of domains such as health services utilization, substance abuse treatment utilization, education and employment and criminal activity. Kuklinski et al. [33] also included tangible costs and consequences from a broad range of sectors in their CBA such as increased earnings, decreased medical expenditure, reduced criminal justice system costs, as well as intangible effects such as pain and suffering and quality of life. From a single outcome of teenage births averted, Rosenthal et al. [24] monetised numerous benefits related to increased productivity and earnings, reduced public assistance, reduced incarceration, improvements in educational and employment opportunities, decreased social and economic support needs, improved personal motivation, and improved peer group influence.

Other economic evaluations limited the considered costs and consequences by using a narrower perspective. For example, Guyll et al. [30]. conducted a CBA and acknowledged the full range of benefits of youth programs, but then used a narrower employer perspective that excluded many potential benefits from being included in the analysis. Sheidow et al. [48] acknowledged the limitations of a CEA of an intervention that potentially has benefits across multiple domains; in these circumstances a CBA was suggested as a more appropriate form of economic analysis for its ability to include a range of benefits.

Despite acknowledging the limitations of their analysis, Sheidow et al. [48] conducted a CEA based on the costs and consequences of substance abuse and crime, which produced complicated results that were unlikely to optimally inform policy in the area. Two other multi-program CEAs demonstrated the difficulty associated with a CEA in a multiple domain context. In the French et al. [44] CEA, none of the three more costly interventions were more effective than the usual care psycho-educational group program for marijuana use or delinquency outcomes, so usual care was the most cost effective intervention. Similarly, as was the case with the Swisher CEA [49] where, after 2 years, the only significant effect of the multiple substance program was fewer female smokers in the intervention group, this program becomes the most cost effective, regardless of cost. French et al. [44] raise the methodological difficulty of more complicated CEAs that include multiple outcomes, stating that cost-effectiveness ratios for multiple outcomes can produce conflicting implications. CBA measures and values the outcomes of a public health program across numerous sectors including employment, family, education or the criminal justice system. CBA provides an estimate of the value of the resources used by a program compared to the value of the resources the program might save or generate [16]. In reality, often a CBA will compare only those costs and consequences that can readily be valued in monetary terms. In contrast, however, CEAs and CUAs assume that one program alternative will be most preferred, regardless of net benefit. Therefore, a program may be adopted which involves a net cost; that is, it does not generate benefits that exceed cost [16]. Under these circumstances, society is better off by rejecting the program – an outcome that would be highlighted with a CBA. Despite some analysts objecting to the monetisation of health related benefits needed for a CBA, when considering the shortfalls associated with CEAs of complex interventions such as those mentioned above, CBA seems a preferable and more appropriate approach to economic evaluation of interventions for at-risk youth [44, 54, 56].

A limitation of this systematic literature review was that despite a rigorous literature search conducted by an accredited librarian, many relevant references were not identified. Of the 29 articles that met the inclusion criteria, 17 were identified via grey literature or hand search, so there is the possibility that not every economic evaluation of an intervention for at risk young people will have been captured. Others have referred to similar difficulties sourcing studies when conducting literature searches of economic evaluations [53, 57].

It is also possible that identified economic evaluations may have been inaccurately classified, however, given the clearly articulated exclusion criteria and the inter-rater agreement on classification and quality assessment, this is unlikely.

A further limitation is the use of the quality assessment rating scale [21]. Despite it being based on the Drummond checklist, a well-known economic tool, it is not a standardised rating scale.

This review has identified that CBAs are most appropriate for capturing the multiple impacts interventions for high risk young people have on participants and society, including the consequential downstream benefits. A societal perspective is the preferred approach for CBA because it allows a broader, more comprehensive analysis. In addition, from a theoretical perspective, CBA is consistent with welfare economics, where all costs and benefits are considered, regardless of who benefits [58]. Many of the methodological challenges highlighted by this review could potentially have been addressed if the economic analyses were conducted prospectively and economists were involved in discussions of study design, and cost and outcome data collection. Prospective long-term data collection and a strong study design that incorporates a control group contribute to the quality of economic evaluation. Where RCTs are not possible, other study designs may provide a partial solution to this problem. In addition, extrapolation of costs and impacts into the future is important for youthful populations in order to account for the time lag in effect of the many benefits arising from interventions for high risk young people. Interventions for high risk young people should target multiple domains of benefit and economic evaluations of such interventions need to capture this complexity [14, 15]. However, techniques for monetization of benefits need to be developed and agreed upon.