An overview of systematic reviews of economic evaluations of pharmacy-based public health interventions: addressing methodological challenges

We selected four policy documents that addressed the specifics and issued recommendations on the economic evaluation of public health interventions: Kelly et al. [11], Honeycutt et al. [16], MRC [2], and NICE [1]. Six academic papers were also selected: Cookson et al. [25], Weatherly et al. [15], Lorgelly et al. [26], Marsh et al. [27], Edwards et al. [28], and Alayli-Goebbels et al. [29].

This was a selected search on existing recommendations on the economic evaluation of public health interventions with the purpose of obtaining key insights for triangulation. This selected search was performed on MEDLINE® (via PubMed) in titles/abstracts till July 2017. A first search used the following terms in titles/abstracts: “review” and “economic evaluation” and “public health interventions” and “methods” which returned 15 titles. After reviewing title and abstract, we included five relevant articles [15, 25‐28]. A second similar search was performed replacing “public health interventions” for “behavior” and “interventions” which returned 22 titles of which, after reviewing title and abstract, we selected one [29]. We then performed snowballing from reference lists of more recent included articles [28, 29] and further selected four policy papers [1, 2, 11, 16].

We selected three academic publications that issued specific recommendations for conducting economic evaluations alongside clinical or pragmatic trials: O’Sullivan et al. [17], Petrou and Gray [18], and Ramsey et al. [30].

This was a selected search on existing recommendations on the economic evaluation alongside clinical or pragmatic trials with the purpose of obtaining key insights for triangulation. This selected search was first performed on MEDLINE® (via PubMed) till July 2017 using the term “economic evaluation” in title/abstract and the following terms in title: “alongside” and (“clinical trials” or “randomized controlled trials”). This first search returned 14 titles of which we just selected the two most recent relevant titles available for free [17, 18]. We then performed a search on ISPOR website under Good Practices for Outcomes Research on July 2017, and we selected the most recent report on economic evaluation alongside clinical trials [30].

This overview considered recommendations by Cochrane, by Smith et al., and by the Joanna Briggs Institute (JBI) [31‐33] for overviews and by the Centre for Reviews and Dissemination (CRD) of the University of York and Cochrane for systematic reviews for public health interventions and for economic evaluations [34‐36].

We have followed PRISMA Checklist in reporting this overview. See supplementary completed PRISMA Checklist, Additional file 1.

First, a systematic review or, at least, an in-depth review of the best available evidence of the effectiveness and cost-effectiveness should be performed prior to the economic evaluation [1, 2]. Second, economic appraisal should be linked to the appraisal of effectiveness [1, 2, 11, 12]. Third, although individual or cluster randomized controlled trials (RCTs) are the preferred study design, they may not always be feasible in the economic evaluation of public health interventions, and randomization may be difficult, in which case quasi-experimental designs could be an option [2, 11] since few RCTs exist or are not feasible or appropriate [15, 26‐29]. Weatherly et al. also suggests the use of econometric techniques [15]. Marsh et al. proposes alternative study designs [27]. Fourth, a societal perspective should be adopted [2, 16]. NICE mentions the public sector’s perspective but acknowledges the societal perspective, where appropriate [1]. Fifth, costs and outcomes should be collected during trial or through decision analytic models but may need further modeling or estimation procedures based on links between measurable outcomes and long-term outcomes [1, 2, 11, 16] reinforced by academic papers [15, 26, 29]. Sixth, public health interventions tend to generate broad outcomes, which may not be captured by quality-adjusted life years (QALYs). Hence, the types of economic evaluations preferred for public health interventions are cost-benefit analysis (CBA), to capture broader health and nonhealth benefits [1, 11, 16], and cost-consequence analysis (CCA), due to the frequent nature of multiple outcomes in public health interventions [1, 11]. Cost-utility analysis (CUA) is also required by NICE whenever health is the sole or predominant benefit [1]. Both CUA and cost-effectiveness analysis (CMA) may be used as well [11, 16]. Seventh, academic papers also report that broad costs and benefits of public health interventions may fall to nonhealth sectors and need to be captured in economic evaluations [15, 28, 29]. Weatherly et al. mentions the intersectoral impacts in CCA and the general equilibrium approach [15]. In addition to CCA, Weatherly et al. also suggests CBA, CUA, and multiple-criteria decision analysis (MCDA) [15]. Lorgelly et al. suggests a capability to encompass health and nonhealth dimensions [26]. Marsh et al. also suggests CBA, capabilities, subjective well-being, MCDA, and better modeling [27]. Eighth, recommended discounting rates for both costs and benefits in economic evaluations of public health interventions are 1.5% for NICE [1] and 3% for CDC [16]. Ninth, equity considerations need to be included [1, 2, 11]. Academic papers also report equity considerations [15, 25, 26, 28, 29]. Cookson et al. suggests possible approaches [25]. Some of these approaches were also identified by Lorgelly et al. and Edwards et al. [26, 28]. Tenth, the economic evaluation of public health interventions should include a wider spectrum of research methods, including qualitative and quantitative research, to understand the contextual and process indicators affecting behavior change [1, 2, 11, 16].

First, most authors consider that such economic evaluations should be based on well-designed pragmatic trials with fewer strict protocols [17, 18]. Second, selection of subjects and sites should also seek for proximity to real-world populations [17, 18]. Third, sample size should be based on important clinical outcomes believed to be correlated with economic outcomes [17, 18]. Fourth, an appropriate length of follow-up is required. Estimates beyond trial time horizon are important and require good modeling [17, 18]. Fifth, current practice or standard of care should be the comparator, although there may be different standards of care in the comparator [17, 18]. Sixth, outcome measures, if composite, should be disaggregated, and direct measures are preferred. Health state utilities should be collected directly from study subjects at regular intervals by instruments or mapping techniques [18, 30]. Petrou mentions that QALYs may sometimes be too restrictive or insensitive [18]. Seventh, most relevant or economically important resource use and cost measures should be collected together with clinical data recorded in case report forms (CRFs) or patient medical records, patient diaries, and/or interviews; computerized record linkage may also be an option in the future [17, 18]. Eighth, valuation in costs needs to be consistent with resource use, perspective, and time horizon. It may include microcosting, unit costing, and gross costing [17, 18]. Ninth, papers outline several additional recommendations concerning methods: plan of statistical analysis and hypothesis prior to trial; plan for on-going data quality monitoring; incremental analysis with an intention-to-treat approach; common time horizon for costs and outcomes; within-trial assessment of costs and outcomes; arithmetic mean cost differences for cost comparisons and bootstrapping, OLS or GLM to compare difference between groups; multivariable methods for analysis of outcomes; uncertainty through confidence intervals, p values, and incremental cost-effectiveness ratios (ICERs) on various time horizons; common discount rate; accounting for missing/censored data through imputation of missing data using multiple imputation approaches; and one or more summary measures (ratio, difference, and/or probability measures) [18, 30]. Tenth, Ramsey and Petrou mention that reporting should include the following: description of trial; major findings; economic data collected; missing and censored data; methods to project costs and outcomes; statistical methods; resource use, costs and outcomes; and results within and beyond time horizon of trial [18, 30].

Economic evaluations of pharmacy-based public health interventions seem to follow most economic quality criteria, but there are still some issues in certain key areas to improve.

This overview sought to include all relevant systematic reviews complying with inclusion criteria, with no restrictions on the period covered, populations, interventions, comparators, or outcomes, in attempt to be inclusive and to critique methods.

We have defined four of seven AMSTAR 2 critical domains for an overall critique on how well included reviews performed in these domains, as recommended. Risk of bias from individual studies being included in the review was not considered, since the most important instruments for nonrandomized studies included in a systematic review were released in 2016 and 2017. Appropriateness of meta-analytical methods was not considered, since included reviews were narrative. Assessment of presence and likely impact of publication bias was also not considered, since funnel plot asymmetry requires a minimum of ten studies and included reviews each have eight studies on average.

The findings of this overview are consistent with methodological issues described by Elliot et al., Perraudin et al., Malet-Larrea et al., Jommy, and Whitty for the economic evaluation of pharmacy-based interventions [24, 53, 55, 60, 61]. In addition, they are also consistent with the major methodological challenges described by Weatherly et al. for the economic evaluation of public health interventions [15] and with the methodological challenges described by O’Sullivan et al. for the economic evaluation conducted alongside clinical trials [17].

This review only included those studies containing the selected search terms. Hence, it is possible we might have missed relevant systematic reviews that may have used a different terminology.

We used a specific filter recommended by the Centre for Reviews and Dissemination (CRD) for economic evaluation in Embase via Ovid, as stated in Additional file 2.

We did not find a specific filter for economic evaluation for other databases used in this review published in ISSG Search Filter Resource. We restricted the search strategy to the definition of full economic evaluation, that is, comparing costs and health outcomes of two or more interventions in which three generic types of economic evaluation are used: cost-effectiveness, cost-utility, or cost-benefit analyses. We deliberately excluded search terms used in partial economic evaluations. However, this restriction might have implicated on the sensitivity of these searches.

In addition, it is possible we might have missed relevant recommendations on economic evaluations of public health interventions and/or alongside trials due to the selected exploratory approach used for these reviews with the sole purpose of obtaining key insights for triangulation.

In recent years, governments in various countries have introduced profound changes to pharmacy remuneration systems in the components related to equitable, safe, and quality access to medicines and to efficiency and/or quality incentives. These two components are the most important in pharmacy remuneration systems. However, the network of pharmacies provides a unique opportunity for governments to implement relevant public health interventions, that fit within national, regional and local health policies, in close collaboration with primary care. In several countries, governments have already contracted with pharmacies to pay for relevant interventions. However, as for any other health technology, public health interventions provided by community pharmacists must seek to demonstrate effectiveness and economic benefits to be reimbursed by public payers.

The economic evaluation of pharmacy interventions presents challenges.

We hope the findings of this overview may assist in improving the design, implementation, and assessment of pilot trials; hence, the robustness of evidence to justify payers’ investment, which requires the endorsement of community pharmacists to participate in trials and an informed understanding of policy makers in negotiations.

Based on the findings of this overview and in addition to the methodological considerations for economic evaluations of pharmacist interventions by Elliot et al. [24], we propose a methodological approach for the economic evaluation of pharmacy-based public health interventions (see Table 7).

As the research corpus continues to expand following practice and policy requirements, it will become important to build a multidisciplinary expert consensus around a specific guidance for the economic evaluation of pharmacy-based public health interventions.