Cost data in implementation science: categories and approaches to costing

Costs have been identified as a key outcome in implementation research [1, 2], yet there is little guidance on how to measure costs as distinct from more traditional cost-effectiveness analyses. Importantly, an implementation strategy may affect its target intervention and have further downstream impacts [2], all of which should be considered in the analysis [3]. Economic evaluations (comparing alternatives based on both costs and benefits) can inform resource allocation decisions and influence stakeholder acceptability, adoption, or sustainability of an implementation strategy or intervention [4‐6]. By measuring costs, implementation scientists can answer questions ranging from what are the costs of alternative implementation strategies in a particular context to what would it cost to replicate these strategies elsewhere. In this paper, we identify costs relevant to economic evaluation in implementation science and discuss practical approaches to costing—paying particular attention to such aspects of evaluation as a decision-maker’s objectives, perspectives, and time horizon.

Costs are incurred when employing an implementation strategy [7]. If the strategy is effective at changing the use of its targeted evidence-based intervention, then this leads to a change in intervention costs. The strategy can also lead to intended and unintended downstream effects and costs, as Fig. 1 illustrates. These distinctions in costs—implementation, intervention, and downstream—are different from costs by implementation phase, described in the section “Costs by implementation phase,” and may include broad representation across types of costs, such as healthcare, patient, caregiver, productivity, or other sectors. The cost categories described here serve as a guide to determine which costs may be important to capture, relative to the research question under consideration.

The “gold standard” for measuring costs is to estimate opportunity costs, which are defined as the value of something that is given up to acquire the next best option, or what was foregone by purchasing what was purchased. Opportunity costs are specific to the decision-maker and they may change as the time horizon changes. Market prices sometimes provide a good estimate of opportunity costs. When estimating the cost of supplies, such as computers or bandages, it is easy to observe the unit costs by searching market prices on the Internet. In other sectors, such as prisons and healthcare, market prices for estimating the average cost of a 4-day stay are typically not readily available, in which case, production costs are used as proxies for opportunity costs.

Micro-costing is prohibitively time consuming for estimating the cost of an inpatient stay. Fortunately, some organizations track labor and supply costs using activity-based cost (ABC) accounting databases to estimate their production costs. These systems track the variable costs provided to each patient. These databases also track fixed, capital costs, including equipment and space, and hospital overhead (e.g., utilities, human resources, security). ABC databases have become the gold standard for estimating healthcare costs in economic evaluation [35]. These systems estimate the production cost and often include subtotals that can be useful for implementation researchers. While beneficial for the specific system costs, the potential limitation of ABC systems is that the production costs reflect only that specific organization or health system and all its unique attributes.

An increasing number of hospitals and healthcare systems are investing in ABC systems, because they enable the hospital managers to examine their production costs [36, 37], but there are implicit assumptions about opportunity costs built into these systems. If an implementation researcher is studying the costs related to an implementation strategy at a specific hospital, then the ABC data provide opportunity costs for that specific hospital and time frame. However, opportunity costs are specific to the decision-maker’s perspective and they change as the time horizon changes.

Because not all organizations use ABC systems, another estimation approach is to use macro-costing, also called gross costing, which is based on grouped or aggregated resources used [13, 38]. Organizations may have information on their charges for a service and its associated components (e.g., hospital stay, laboratory test), which is simply the amount listed on bills for a group of services and resources, which could be adjusted to estimate costs. Charges include profits and can be used in negotiations with third-party payers, so any charges need to be deflated to represent costs more closely, for example, by applying a hospital or hospital department’s cost-to-charge ratio [23]. Payments for services, especially federal reimbursement rates such as through the US Medicare program, provide another common source of proxy cost information. The payment is the amount paid by the insurer or public agency, and in the case of the US Medicare program, is based on work effort, overhead, and other operating costs associated with providing healthcare services. For US medication cost estimates, the US Federal Supply Schedule is recommended based on Veterans Affairs purchasing prices [22]. A limitation with both charges and payments is the inability to separate variable and fixed costs. The inclusion of fixed costs can confound an analysis with a short time horizon; not only do the fixed costs overstate the costs that can be varied, but variation in the fixed costs can also reduce the power to detect meaningful differences in variable costs [3].

There are national databases in the USA, such as the Healthcare Cost and Utilization Project (HCUP) [39], Medicare administrative claims, or commercially available health insurance claims databases, that can be analyzed to provide details on average charges or payments for services, which can be a complicated process [40‐43]. Finally, any relevant out-of-pocket healthcare costs to patients can be calculated from some claims databases using copayments, coinsurance, or deductible amounts to assess patient costs and the full healthcare sector cost. However, in each case, these estimates may have little relevance to the opportunity cost of a specific hospital(s) or organization being studied, because each entity has its own way of converting inputs into patient care and outcomes.

How to generalize cost estimates from specific healthcare systems or settings is a question that remains. However, national-level estimates of costs and disaggregated resources required may be the most useful so that local organizations can adjust to their locality. At the very least, one can conduct sensitivity analysis to assess a range of costs that might be relevant to various localities (e.g., vary costs ± 5–10% or by a cost-of-living adjustment). Importantly, using data from one hospital or organization for another makes implicit assumptions about the production process. Variation in hospital efficiency may be less consequential in economic evaluations that use a long-run time horizon, but they can be important when assessing variable costs in the short run [44, 45].

Regardless of the potential difficulties, implementation scientists should focus on production costs when estimating costs, which are likely to lead to the best approximation of opportunity costs for implementation studies, with special attention to the contextual issues relevant to estimating production costs. Because there is no database reporting opportunity costs, implementation researchers might consider qualitative work prior to estimating production costs to ensure that the estimates reflect the variable cost within a decision-maker’s time horizon [4].

Different types of costs may be incurred by stage in the implementation process [9] (Table 1). Those stages could include pre-implementation, exploratory, and planning, as well as adaptation, implementation, sustainment, spread, and de-implementation [46]. Summing costs by phase can be useful for stakeholders to determine the feasibility of implementation over time [28]. Costs ascertained from any of these phases should be included if they are required to replicate the implementation strategy and associated intervention elsewhere (or to adapt, sustain, spread, or de-implement the strategy and intervention). Sometimes it can be difficult to disentangle costs by phase, for example, implementation and intervention costs, or implementation and adaptation costs. It is important to note where there may be overlap or lack of clarity on phase distinction. Furthermore, there may be site-specific or centralized costs within each phase or across cost categories; if relevant to analyses, these should be captured [28].

Follow-up within implementation studies is often short, even as one seeks to estimate longer-term cost (and potentially health) outcomes for downstream impacts. For example, decision-making about an intervention that prevents injurious falls, a significant source of morbidity and mortality for older adults, is best informed by estimates of costs saved per fall avoided. However, the impact of such an intervention may not be observed for months or years afterwards [47]. To overcome limited follow-up, simulation modeling can be used to explore the cost or health impacts of the intervention beyond short-term outcomes (and guide resource allocation decisions) [48‐50]. The use of modeling also can highlight the spillover effects of the intervention to other conditions or sectors, including substitution of services, and allow analysts to perform sensitivity analyses. Implementation scientists need to account for any differential timing of costs if they occur beyond one year, by discounting costs (and benefits) using context-relevant discount rates (e.g., 3.5% for studies in UK settings [NICE]; 3% for studies in the USA) [16, 22, 51].

Depending on the stakeholders or decision-makers, there can be many analytic perspectives to consider for the cost and economic analysis of intervention implementation, for example, society (e.g., federal government program), patient, caregiver, healthcare system, healthcare sector, criminal justice system, employer, or public health department. The costs that should be included in an analysis are those that accrue to or are paid by the decision-maker, and in some cases, stakeholders may care about broader costs than ones attributed to them [52]. For example, a public health department may care about the implementation strategy and intervention’s costs associated with providing health services and the costs for participants, because if the cost burden is too high for the participants in the intervention, they may not participate, making the effort wasteful.

A societal perspective considers every potential cost associated with the implementation effort, including healthcare sector resources, non-healthcare sector resources, informal caregiver or volunteer time, patient time and transportation, costs of changes in patient productivity, and future costs related and unrelated to the intervention [16] (Fig. 1). Conducting an economic analysis with a long-run, societal perspective will identify cost-effective implementation strategies that maximize well-being for society, because it accounts for all economic impacts. Such broad evaluations may not provide information on the optimal policy from a specific perspective, however. A pragmatic solution for the implementation researcher would be to report on cost and economic analysis from multiple perspectives (e.g., health system, clinic, societal) and time horizons (e.g., 1 year, 5 years), which allows decision-makers to use the results relevant for them.

A recent study evaluated practice facilitation as an implementation strategy for improving small primary care practices’ adherence to clinical practice guidelines for cardiovascular risk reduction [53, 54]. To capture societal costs, one would include every potential cost, such as costs of practice facilitation, office staff time (e.g., meeting time, potentially office space), physician time and reimbursement for services, patient and caregiver time requirements stemming from primary care practice changes (e.g., smoking cessation counseling, lost productivity), immediate and downstream healthcare service use (e.g., smoking cessation medication, hospitalizations), and more. Estimated rent for required space that is above-and-beyond usual space use should be considered if space is scarce, because the space cannot be used for another purpose if it is used for the implementation strategy. The physician perspective, in deciding whether to participate, would include costs for physician time, supplies, and office staff time. The patient perspective would include costs of patient time, out-of-pocket costs, and potentially paid caregiver time, if patients pay for dependent care to participate in the intervention. If the local public health department were sponsoring the practice facilitator program, the public health department would want to know the costs of the implementation strategy (e.g., training and infrastructure to support practice facilitators, cost of their time in physician offices, travel time). A limited cost analysis was published, focusing on implementation strategy and practice staff time costs [54]. Such a narrow evaluation can lead to more questions, however, and leave other potential stakeholders who are interested in implementing such an approach wanting more information.

Another example of the potential implications of the choice of analytic perspective relates to a study of substance abuse treatment. Societal cost-effectiveness analyses have shown that substance abuse treatment is cost-effective [55]. Yet, the healthcare sector incurs the costs of substance use treatment, while the criminal justice sector reaps many of the benefits [55, 56], highlighting the possible conflict between decision-makers when costs and benefits accrue differentially to these sectors. A focused economic evaluation from the healthcare system perspective would make substance use treatment less appealing. This explains the paradoxical contraction in substance use treatment programs that has occurred over the past two decades [56].

Furthermore, broad economic evaluations that maximize social welfare may conflict with narrower, focused evaluations that are frequent in implementation science. Decision-makers often want a stylized analysis, such as a budget impact analysis [35, 57], or only care about the specific impact per patient. Focused evaluations that consider only specific categories of costs can lead to decisions that seem rational to the decision-maker, but are suboptimal for society and may not be social-welfare maximizing. Because an implementation strategy may affect evidence-based intervention utilization, efficiency, or downstream healthcare service use differentially, considering multiple cost categories is important; a narrowly defined implementation effort may in fact change how healthcare services are used in the near and longer term, which create a broader impact than is captured by only estimating implementation costs.

As a rule, implementation scientists should sum the total costs of an implementation strategy and its impacts (and perhaps by stakeholder perspective) within an explicit time period (Table 1). This may be insufficient if such total costs vary across sites, however, because most decision-makers want to know what it would cost to adopt a strategy or program in their setting. Most costs will vary by some parameter (e.g., number of clinical staff, sites, participants) and over time. Implementation scientists will need to explore and report on such variation, so that decision-makers can anticipate or model the cost of the implementation strategy in new settings (or time periods).

In a study comparing group-based versus individual-based multidimensional-treatment foster care, the authors measured costs of feasibility, planning meetings, selection of service providers, staff training, fidelity monitoring and review, site visits, and feedback by stages of progress toward implementation [9]. By capturing subtotaled resources used (e.g., staff hours by job type, office space) and unit prices (e.g., wage rates, rent) and subtotaled implementation costs by week or month and by activity type (e.g., meetings, training), the analyst can conduct a more detailed analysis of costs. In particular, it would allow learning about such cost aspects as site-level variation in multi-site studies and whether costs change over time, potentially due to learning or efficiencies.

Ideally, implementation scientists should report both total intervention implementation costs and more granular cost measures. Because implementation strategies often target behavior change for organizations or professionals, the cost per practice or organization, per professional, or per end-user may be suitable metrics for the decision-maker. However, the major limitation of such denominators is the inability to attribute costs to a patient or participant and therefore to a subsequent health outcome; ultimately, the goal of implementing evidence-based interventions is to improve health outcomes and population health.

Sometimes a decision-maker wants to know how comparative implementation strategies with their evidence-based intervention affect health outcomes (Fig. 1). Such comparisons are facilitated by economic evaluations reporting the comparative costs by strategy with a measure of a patient’s health or well-being, such as life-years or quality-adjusted life-years gained. A denominator that is program and disease agnostic, such as quality-adjusted life year, allows for fair comparisons across strategies when resources are restricted.

Collecting information on patient outcomes may be burdensome and not central to an implementation study. Where there is robust evidence supporting an intervention being implemented, implementation studies may restrict themselves to measuring organizational and professional behavior change and report relevant outcomes, such as the additional cost for a percentage change in clinical guideline use. Other outcomes to consider for economic evaluation in implementation science could vary as widely as cost-per-patient or participant, cost-per-practice or per-health center, or cost-per-inappropriate MRI avoided. Implementation cost-effectiveness outcomes that incorporate patient-centered health outcomes can be added to decision analytic models by adapting the structure and evidence from intervention models [58].

To address the potential bias in estimating costs in implementation studies, sensitivity analyses are essential. Substituting job classifications, varying program capacity (i.e., estimate cost if the program is at full capacity versus initial rollout), reflecting regional or national wage rates, and estimating best/worst case are approaches to testing robustness of results.

Implementation scientists should consider a broad range of cost categories when assessing the economic implications of strategies for implementing evidence-based care—and measure the costs relevant to the choice of analytic perspective(s), time horizon(s), and outcomes. Implementation scientists should explicitly consider costs from all sectors, such as healthcare, justice, and education, with strong consideration of the magnitude of costs for each sector and the cost or time burden of collecting the data. Although many implementation studies in healthcare focus exclusively on healthcare costs that are relevant for a particular decision-maker, a broader analysis may be useful or applicable to more than that decision-maker alone, leading one to consider additional, disaggregated, or divisible cost categories.

If researchers are using data that originate from accounting systems, where they can identify the variable and fixed costs, then they should determine how that system defines fixed costs and decide whether to treat those costs as fixed or variable given that the study time horizon may be different from the accounting system. Estimating costs by sector, phase, or even participant or patient subgroup can be useful to ensure a wide range of stakeholders understand the costs or savings that would accrue for their particular concern or perspective. Specific types of costs may affect adoption, implementation, fidelity, and sustainability.

Studies need to be explicit about each included cost category, type of cost data and its source, and how costs were calculated, to enhance transparency and improve adoption decisions by others. Engaging an expert in economic evaluation and key stakeholders prior to implementation can help inform which costs to capture and how. Making all analytic assumptions clear, justifiable, and unambiguous, e.g., durability of the intervention’s effect, time horizon, analytic perspective(s), and reasons for including/excluding specific costs, allows readers and decision-makers to replicate or compare results and apply them to their own settings. As outlined here, conducting analyses by societal, stakeholder, and sectoral perspectives will allow for broader uptake and relevance of value assessments by various decision-makers. Such value assessments can also help decision-makers reallocate resources to address health equity, such as redirecting resources to areas of greatest need [59].