Extending analytic methods for economic evaluation in implementation science

The Exploration phase provides an opportunity to review stakeholder perceptions of intervention components and implementation strategies documented in prior literature and to engage context-specific stakeholders in discussions about EBI development. During the Preparation phase, interviews and surveys with diverse stakeholders can elicit their expectations and capacity for EBI implementation. These methods may also provide insight into what usual care or other implementation strategies entail in their local contexts to provide a comparator(s) for how the EBI is implemented. In the Implementation phase, these methods along with periodic reflections [37] can be used to solicit input on EBI implementation successes and challenges and to clarify resource use and unexpected or unintended expenses. Periodic reflections, in which agents are asked about their experiences with EBI implementation at multiple time points, can identify potential challenges and adaptation opportunities [37]. As examples, routine discussions with implementation agents may reveal time-consuming or otherwise resource-heavy steps threatening EBI sustainment, or provide information about a policy or contextual change directly impacting EBI implementation. Capturing this information allows for more accurately measuring resources expended and developing solutions, which can help to obtain stakeholder buy-in for EBI sustainment.

In the case of SCORE, during the Exploration phase, we met with state-level stakeholders (e.g., Colorectal Cancer Roundtable) to solicit input on the feasibility of candidate EBIs and fit with current workflows using local consensus discussions [28]. During Preparation, we engaged CHC and endoscopy providers in workgroups to identify resource barriers and facilitators to implementation [28]. The workgroups created process flow diagrams for each of SCORE’s central components (described later). During Implementation, we employed survey and interview methods to estimate resource use and to evaluate implementation agents’ perceptions of intervention implementation and its impact on usual care. These methods included (1) questionnaires about CHC screening processes in the absence of SCORE (e.g., who performs each activity, how frequently, and time spent per patient), (2) brief, electronic surveys assessing intervention acceptability mid-implementation (e.g., are intervention objectives clear?), and (3) semi-structured interviews about how intervention implementation has affected clinic work processes (e.g., how, if at all, has your work changed because of SCORE?). Insights from periodic reflections [37] are being used to proactively determine how to address any possible threats to sustainment and improve outcomes.

The expected gains of implementing the SCORE intervention depend on patients being receptive to mailed FIT outreach and, if their results are abnormal, completing a follow-up colonoscopy. The implementation strategy of adding centralized staff to navigate FIT-positive patients to their diagnostic colonoscopy will only be successful if patients are willing to respond to the navigator and utilize the services offered. Therefore, we designed an interview guide for FIT-positive patients about their SCORE experiences that inquired about navigation and colonoscopy completion steps. We included a quantitative checklist for patients to report how long each step took (e.g., time spent driving to the pharmacy) and any out-of-pocket costs (e.g., bowel prep kit cost). We then qualitatively assessed which activities are most burdensome to patients and caregivers and how navigation may have alleviated these burdens. This mixed-method approach to estimating patient and caregiver resources will provide detailed cost data specific to the lower-resource CHC population targeted by SCORE. Additionally, it may help to identify ways to adapt, sustain, and/or scale existing implementation strategies to best meet patients’ needs and minimize their burdens.

Time-and-motion (TAM) analysis involves estimating labor-related inputs associated with EBI implementation [42]. Key processes involved in EBI implementation are assessed (for example by using process maps), toolkits are designed to track those activities, and identified processes are observed and recorded using the toolkits. This method allows for estimating the time required per activity, which can be used to estimate per-person labor costs. Conducting these observations at multiple time points allows for estimating differences in time (and thus costs) associated with specific activities across implementation agents and evaluating efficiencies over time. TAM data are an integral component of microcosting (i.e., bottom-up cost analysis) and can inform how to assign common resources that do not fit neatly into a single activity or category [43, 44]. Prior studies have demonstrated how related time-driven costing methods allowed for more accurate cost estimation of health interventions, including variation in delivery and associated costs across sites and personnel [45, 46]. Analysis of TAM data captures the total investment of personnel time and resources in EBI implementation and provides insight into how to optimize processes to support sustainment and scale-up.

In the SCORE study, we used process mapping during the Preparation phase to develop the multicomponent intervention, plan for its implementation, and design our TAM analysis. Through consensus discussions with stakeholders [28], we developed “swimlane” process maps, which use lanes (i.e., rows) to delineate which agents perform specific steps and in which settings (e.g., CHCs, laboratories, mailed FIT outreach center). We used these diagrams to identify groups of activities requiring personnel time that could be observed in batches, such as mailing introductory letters. As with our other methods, we considered which steps are research-specific and which would need to be performed outside of the research context for programmatic success, including only the latter in our economic evaluation. For each activity, we developed a TAM costing tool to document the labor steps involved. For example, mailing introductory letters entailed identifying eligible patients, conducting mail-merges, labeling envelopes, and performing quality control. We piloted these tools during the Preparation phase and scheduled periodic observations during the Implementation phase. We used our swimlane diagrams during the Preparation phase to identify other non-labor costs associated with each step and to develop fidelity measures for tracking potential variations or adaptations in EBI implementation, which may have cost implications [12, 47]. For each process step, we documented how the associated costs and fidelity steps were to be measured, the frequency of data collection, and where to report the collected data (see Fig. 1a). In Fig. 1b, we demonstrate how we used these maps in combination with other methods to further develop our measurement tools. Figure 1c provides a hypothetical example of how these maps might be used to integrate quantitative and qualitative results to document gaps in fidelity along with implementation agents’ perception of burden incurred at each process step.

For the SCORE intervention, we invited CHC clinicians and administrative staff and quality improvement monitors involved in implementation process steps from the swimlane diagrams to participate in SSM sessions. The information gathered will help identify which agents are employing which implementation strategies and, thus, ensure all responsibilities and resources used are appropriately costed in the economic evaluation. Whereas our process maps detailed the specific steps being carried out by implementation agents, SSM allows for identifying possible redundancies, inefficiencies, or misunderstandings about EBI-related responsibilities individuals undertake, unexpectedly resource-heavy or under-supported activities, and further delineation of roles (e.g., which individual conducts each mailed FIT process step among the larger mailed FIT team). Agents’ recommendations for improvement can also be estimated in terms of their expected costs and benefits during the Implementation or Sustainment phases to inform decision-making about EBI adaptation and/or sustainment. Similarly, SSM may identify opportunities to streamline responsibilities or better support staff needs, which can be evaluated in the economic evaluation.

Our Cancer Control PopSim work shows how simulation can be used to project the downstream impact of EBI implementation in higher-risk populations. Using our model, we estimated the cost-effectiveness of multiple EBIs for improving CRC screening and long-term CRC outcomes (e.g., cancers averted) in priority populations, such as African Americans [25], the uninsured [23, 24], and Medicaid enrollees [22‐24]. Costs of EBI implementation, CRC screening and diagnostic procedures, and CRC treatment were included. Among Oregon Medicaid enrollees, for example, we found three of five EBIs simulated to be cost-effective compared to usual care if Medicaid decision-makers are willing to spend up to $230 per additional year up-to-date on CRC screening [22]. In North Carolina, we identified mailed reminders for Medicaid enrollees and mass media campaigns for African Americans as cost-effective EBIs, costing approximately $15 and $30, respectively, per additional life-year up-to-date on CRC screening [24]. We also showed that expansion of North Carolina’s Medicaid program would more substantially reduce CRC diagnoses among African Americans, compared to non-Hispanic Whites, and result in greater cost-savings over the long-term due to averted treatment costs [25]. These analyses allow for targeted EBI implementation planning by detailing the funds needed to efficiently address health inequities.

Related to SCORE specifically, our plans are to have a cost-effectiveness model focused on our target CHC population and with more detail on how the different intervention components and implementation strategies affect success at micro-level steps. The model can then help to project the downstream impacts associated with our outcomes (fidelity, reach, etc.) at each process step.

Many types of sensitivity/uncertainty analyses can be conducted using the methods for estimating costs and benefits previously described. Examples include scenario analysis in which variation in conclusions is assessed using specific values for uncertain parameters; threshold analysis to identify the particular value(s) at which EBI implementation becomes or is no longer cost-effective; and probabilistic sensitivity analysis where multiple uncertain parameters are varied simultaneously using distributions of possible estimates [54]. Regardless of which analyses are conducted, areas of uncertainty related to further EBI implementation in the current setting or in other settings should be proactively identified (potentially through systems mapping and stakeholder engagement) as EBIs are planned and implemented. For example, capturing variations in TAM estimates by agent type and over time can provide ranges of estimates for conducting a best-case/worst-case analysis. Questions of most importance to decision-makers about sustaining cost-effective EBIs and implementation strategies should be prioritized; for example, depending on context and perspective, the outcomes of focus in an economic evaluation may vary and may not include patient-level utility estimates. Analytic prioritization could also include comparing different scenarios of how personnel, start-up funds, and other resources are allocated across EPIS phases and their relative impact on cost-effectiveness over time. Varying the analytic time horizon could also reveal important insights, such as how long EBIs may need to be implemented to achieve objectives. The impact of uncertainty on outcomes of interest and sensitivity of the results to changes in EBI implementation should be evaluated for the local context in the Implementation phase and to support long-term planning during the Sustainment phase.

Our Cancer Control PopSim model outputs demonstrate how sensitivity/uncertainty analysis can be used to consider the impact of tradeoffs related to EBI implementation. We used this model to evaluate the effectiveness of multicomponent CRC screening interventions in reaching national screening targets (Hicklin et al: "Assessing the impact of multicomponent interventions on colorectal cancer screening through simulation: what would it take to reach national screening targets?", in progress). We varied the percentage of the target population reached by each intervention, initially considering the differences in impact on selected outcomes assuming 25%, 50%, 75%, and 100% intervention reach levels. We then conducted a threshold analysis to determine which specific level of reach is needed to achieve screening targets under different circumstances. Our analysis demonstrated that the expected downstream effects of implementing EBIs are driven by multiple factors, including intervention effectiveness, intervention reach, implementation costs, and equity considerations (i.e., which subpopulations are targeted by which interventions). With our SCORE model, we will use data on implementation costs and outcomes associated with individual process steps, together with input from stakeholders, to explore how intervention operations and implementation strategies might be adapted to support sustainment. Using mixed methods to collect comprehensive and context-specific costs across implementation outcomes will aid greatly in having meaningful estimates to weigh tradeoffs over the short- and long-terms.