A national evaluation of a dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: the ESCALATES study protocol

The ESCALATES team will collect Cooperative documents that are important to understanding the interventions they propose, the modifications they make to their plans, and the partnerships they develop to carry out their work. This includes, but is not limited to, grant applications, documents that establish changes to study and intervention design, training materials for Cooperative staff, meeting agendas and minutes, and educational materials for practices.

Each Cooperative will identify at least five diary-keepers who will participate in an online diary to document experiences during project development and evolution, including practice engagement, recruitment, and providing external support for implementation. The use of online diaries as a qualitative data collection method has been described in detail elsewhere [43]. Briefly, each team will have a private online diary space to post entries in a blog type format. Cooperatives will be encouraged to make diary entries at least twice a month. Only diary-keepers at a particular Cooperative and evaluation team members will be able to view entries from that Cooperative, and we shall maintain strict external confidentiality about the information posted within each Cooperative. The ESCALATES team will read entries weekly and interact with Cooperative diary-keepers via the online diary to encourage frequent posting.

The ESCALATES team will visit each Cooperative annually to facilitate collaborative work and to fully understand the D&I approaches they are implementing and their lessons learned. The site visits will last approximately 2 days each year, and ESCALATES team members will meet with each Cooperative’s project personnel including the research team, practice facilitators, coaches, consultants, and other partners. The ESCALATES team will observe Cooperative intervention strategies, foster discussions on implementation experiences, and observe interactions among partnering institutions and with individual practices. In year 4 of the study, after funding for the Cooperatives has ended, the ESCALATES team will conduct an additional site visit that will focus on assessing the sustainability of the network infrastructure.

During site visits, the ESCALATES team will conduct five–ten semi-structured interviews with key stakeholders from the Cooperative. Interviews will focus on filling knowledge gaps about the intervention strategies being tested, understanding the experiences of partnering stakeholders, the mechanisms by which interventions are believed to change outcomes, and the factors that affect or explain implementation experiences. Interviews will follow a guide to be refined for use with each Cooperative and each key stakeholder. Interviews will last 40–60 min and will be audio recorded and professionally transcribed.

Cooperatives will provide practice-level ABCS data to the ESCALATES team, as specified in Table 3. Cooperatives will collect data from participating practices’ EHRs using a range of data extraction methods (e.g., programming, manual chart review). ABCS measures will be collected at baseline—before interventions begin—and quarterly through the end of each Cooperative’s study. Cooperatives will also contribute practice-level ABCS data stratified by gender, race, ethnicity, age, and insurance type to allow for the examination of disparities.

In addition to ABCS clinical quality measures, information about practice characteristics is needed as well as practice capacity for change, a key outcome measure. Each Cooperative will administer surveys to collect practice level data on characteristics and capacity using descriptive questions. The “practice survey” (PS), will collect data on practice organization and infrastructure (e.g., size, ownership, staffing, EHR capacity). This survey will include the change process capacity questionnaire (CPCQ) [44, 45], which is a measure of both a practice’s capacity for QI and the strategies used to improve. One person in the practice (preferably a practice leader) will complete the PS. The second survey, the “practice member questionnaire” (PMQ), will be administered to practice members and include a measure of organizational culture and capacity using the adaptive reserve [10, 46‐49] scale. In addition, the ESCALATES team will work with Cooperatives to identify high-priority measures to understand implementation, as informed by the CFIR and PCM frameworks. The PS and PMQ surveys will be administered by Cooperatives at baseline, immediately post-intervention, and 6-month post-intervention. The ESCALATES team will work with Cooperatives to help ensure a high response rate.

It is important for this evaluation to track characteristics of the interventions delivered to practices, including the type and intensity of external support interventions received, along with changes that practices implement to increase ABCS delivery. The ESCALATES team will work with Cooperatives to establish methods (e.g., practice facilitator contact logs, collaborative (onsite) or webinar attendance records) to track the type, frequency, duration and mode of external support provided to practices and their engagement with it. Data collection related to external support will continue throughout each Cooperative’s active intervention period, which will vary by Cooperative. The ESCALATES team will also work with Cooperatives to develop mutually agreeable methods to track and share the change strategies practices implement to improve ABCS (e.g., registries, standing orders) and the extent to which these changes are implemented within these practices.

The ESCALATES team will also conduct practice site visits. We will purposively select a sample of 40–60 practices using both qualitative and quantitative data to do so. The ESCALATES team will develop a matrix that ranks practices by ABCS outcomes at baseline (from high to low), and will also include percent change in ABCS over study period, Cooperative, ownership, and characteristics relevant to the region within the Cooperative (e.g., rural, underserved) where relevant. The ESCALATES team will use this matrix of quantitative information along with qualitative implementation data to select a maximum variation sample of high and low performing practices, approximately six–eight per Cooperative. We will conduct preliminary analyses based on the first two–three visits and use this information to revise sample selection criteria as needed. This iterative process of selecting practices, collecting data, and using this information to refine sampling decisions will continue until variations in outcomes are adequately explained as indicated by theme saturation.

The ESCALATES team will work with Cooperative teams to invite practices to participate. Practices will receive a monetary fee for their time and participation. A multi-person team will spend 1–2 days in a practice intensively observing how ABCS care is delivered and conduct interviews (scheduled to minimize disruption) with practice members to understand their experiences with external support and their efforts to implement interventions aimed at increasing delivery of ABCS. Interviews will also identify practice, organization and contextual factors influencing implementation. Interviews guides and observation templates will be refined and tailored to each practice and to each member based on information from online diaries, Cooperatives, member role, and site visit preparation calls with practice members.

At each site visit, study staff will conduct five–ten patient pathways [50]. Patient pathways are designed to observe patients’ exposures to and experiences with how the practice has implemented changes. Front desk staff will alert a field researcher when a patient meeting sampling criteria checks-in. The field researcher will ask the doctor and patient for permission to observe the visit. If both agree, the field researcher will follow the patient through the visit (e.g., intake, visit with clinician, check-out), being unobtrusive and making brief notes to be written up later as fieldnotes.

A central aspect of Cooperatives’ interventions involves developing the infrastructure needed to rapidly deliver external support to approximately 250 practices. The ESCALATES team will analyze qualitative data to characterize the specifics of each Cooperative’s external support strategy and to create typologies that distinguish interventions and characterize the infrastructure Cooperatives develop to support practices. Typologies are a system for categorizing Cooperatives and their participating practices into groups based on a set of salient characteristics. Groupings will be quantified and become variables in subsequent quantitative analyses. This mixed methods procedure is to converge the data using data transformation so that a quantitative analysis can be used based on quantifying the qualitative findings. To do this, the ESCALATES team will analyze qualitative data using a grounded theory approach [54‐56] to characterize the factors that emerge as barriers/facilitators to implementation (e.g., practice, context/environmental, and intervention characteristics), how and by whom external support is delivered and the factors that affect this, and what changes practices implement and why. We will make comparisons across Cooperatives, paying particular attention to factors leading to a Cooperative excelling or struggling in response to the intervention. We will create matrices to display data for comparative analyses as suggested by Miles and Huberman [57]. As the ESCALATES team develops initial classifications, these will be shared with Cooperatives as a form of “member checking,” a qualitative research verification step accomplished by asking key informants to validate study findings [58‐61]. This analytic strategy will result in rich interpretive summaries of each Cooperative’s intervention, particularly with respect to core features of intervention strategies and other characteristics the ESCALATES team identify as important to D&I. Summaries and matrices will lead to the development of a series of typologies that categorize and transform important aspects of the Cooperatives and their interventions into quantitative variables. For example, external support (e.g., practice facilitation, expert consultation), mode of intervention (e.g., face-to-face, email, phone), frequency (number) and duration (length) of contacts might be made into a composite variable to represent intensity of external support for use in subsequent quantitative analyses.

Examining the effectiveness of external support interventions will involve estimating differences in ABCS and practice capacity outcomes for different intervention strategies received by practices engaged in each Cooperative, as well as differences in practice implementation. Intervention strategies will be compared to each other and/or to a control group of practices using a differences-in-differences (DID) approach [62]. The DID model is a flexible approach which can accommodate different study designs Cooperatives may propose to evaluate their own interventions. Conceptually, cluster-randomized designs, interrupted time studies, and step-wedge designs can be nested within the DID framework. The primary requirements are the inclusion of baseline (pre-intervention) data and data from a control group. In addition to examining the effectiveness of intervention typologies as assigned, we will also conduct sub-analyses that include differing levels of fidelity to quality improving strategies practices use. The DID approach has been frequently used by health services researchers and economists to account for potential secular effects and changing policies that would affect both intervention and control groups over time, while adjusting for potential confounders [7, 63‐68].

The overall analysis to assess intervention strategies on ABCS performance can be considered as shown in the equation

where: Y _it represents clinic i’s performance on an outcome measure (i.e., one of the ABCS measures, mean adaptive reserve score) at time t,; I _j represents a dummy variable (intervention strategy received variable described above) taking a value of 1 if the clinic was involved in intervention j and zero otherwise; and τ _j is a dummy variable taking a value of 1 if the observation took place after the intervention and 0 otherwise, and f represents a general function that captures our intent to model performance through a beta regression for ABCS measures and linear regression for practice capacity outcomes. The coefficient α is an overall intercept representing average baseline performance for all clinics; β _j represents the incremental baseline performance differential between clinics in typology j, and δ represents the change in performance pre- and post-intervention, for all clinics. The θ _j coefficients are the coefficients of interest, as they capture the relative performance improvement for each group using intervention typology r. The qualitative analyses described above will produce r variables (e.g., high, medium, low intensity support strategy) (r lying somewhere between 3 and 6). In this model, control groups serve as the reference. This equation can be adapted to include additional dummy variables for each time period or a continuous variable to capture linear time trends as well as fixed effects (equivalent to dummy variables) for practices.

The ESCALATES team will use two-way fixed effects models (i.e., a fixed effect for each practice and for each time period) so the results do not rest on some of the stricter assumptions required for random effects models. However, we will also conduct analyses by introducing a practice random intercept and slope to allow intervention effects to vary across practices in the different study arms. This approach explicitly models different clustering levels (e.g. Cooperative, region, practice) [63]. In this case, rather than treating the practice as a nuisance factor, this approach allows each practice to respond differently over time to the intervention and allows the ESCALATES team to estimate those changes and study the heterogeneity of effects due to variation in context across Cooperatives and differences across practices within each Cooperative.

The analyses described above will be conducted for each ABCS measure. In addition, ESCALATES will produce a model that pools all measures and uses measure-specific fixed effects, which will allow for an assessment of an “overall” effect of the interventions.

To assess heterogeneity of intervention effects, ESCALATES will identify factors associated with substantial improvement for some practices and lack of change or worsening performance for others. For instance, the ESCALATES team may observe that practices receiving the external support strategy of data feedback/benchmarking resulted in significantly improved ABCS performance measures over time as compared to a group of practices receiving expert consultation or a control group of practices. On further analysis of the effect of type of practice (moderator) on the observed association, however, the ESCALATES team may find that practices affiliated with large health systems showed greater improvements in ABCS measures than independent practices. Thus, this effect modification analysis is designed to capitalize on the large sample size and diversity of practices and several different intervention strategies that will be tested by Cooperatives to identify heterogeneity of ABCS measures and practice capacity measures of association across levels of modifying variables. Another key variable of this analysis involves the fidelity with which practices implement QI tools to address ABCS. Data on implementation fidelity will be collected using the external support/practice implementation tracking data. ESCALATES will create meaningful categories of this variable to include in the modification analysis, such as practices that implement QI tools consistently and to a high degree of fidelity will likely perform better at improving ABCS and practice capacity than those that do not, even when both groups of practices receive the same external support.

This analysis will extend the DID models proposed above by testing individual moderators with a three-way interaction of intervention group, time point and potential moderator. Our choice of variables to include will be based on theory and on triangulation with qualitative findings. Potential moderators include practice characteristics, as significantly associated with ABCS outcomes at baseline, and contextual factors that emerge during implementation. Moderators may be identified through surveys or qualitative data. Moderators identified through qualitative data will be quantified whenever possible. In addition to testing individual moderators, we will consider a composite moderator, which may more strongly moderate the effect of the intervention strategies on ABCS outcomes than any single moderator. Specifically, ESCALATES will create a combined moderator that is developed as a weighted combination of individual selected moderators to identify for which practices different intervention strategies may be preferred. Combining moderators has been shown to exhibit a larger moderator effect size than any individual moderator [67]. We will develop and evaluate a composite moderator using the parametric approach developed by Kraemer [69], which constructs a composite moderator from a weighted average of moderators that are weighted to maximize the correlation between the outcome difference between randomly paired practices and the average of the two values of the moderator variable.

In addition to these quantitative analyses of heterogeneity, we will also conduct qualitative analyses to evaluate the multilevel contextual factors that could affect either the interpretation of the results of each analysis, or that might be useful for others attempting to replicate or reinvent interventions in different times or situations [70, 71]. These analyses will use multiple data sources across multiple levels (from the practice and health care system, the intervention/evaluation team, and the local and national policy context), and will attempt to include diverse stakeholder perspectives of the most important contextual factors at critical points in the project/intervention, to help understand what happened here and why, and what someone else would need to know to transport/re-invent what was learned here in a different time and place. These analyses will enhance the external validity of ESCALATES findings.

To analyze why some intervention strategies are more effective than others, the ESCALATES team will analyze data among a group of high and low performing practices, using the same steps for qualitative data analysis described above. Data for this analysis will include fieldnotes, interviews, and external support/practice implementation tracking data which will allow us to establish what the practices implemented to improve ABCS delivery. Importantly, the qualitative team will be blinded to a practice’s status as a high or low performer during the data collection and analysis process. The qualitative team will identify mechanisms of practice change with regard to ABCS performance, the factors that influence a practice’s ability to implement or not implement change, and practice members’ experiences with their respective Cooperative’s intervention. At this stage, the qualitative team will bring in other data sources to make sense of the characteristics that distinguish practices. The end product will be a table of 40–60 practices and the factors or elements the qualitative team identifies as potentially important to practice change and improvement, and whether this element is present or absent for a practice. Next, the quantitative team will add practice performance to this table. A joint display will be created for a mixed methods analysis comparing the high and low performance practices in terms of qualitative findings and quantitative results. In order to distill a clear understanding of the factors associated with successful implementation, the qualitative team will use this information to conduct a deviant case analysis [72], examining in greater detail practices not conforming to our initial interpretation of what distinguishes low and high performing practices. As a final step, the full ESCALATES team will reflect on findings to fully identify the range of factors and characteristics vital to successful implementation of ABCS among which types of smaller practices and the key elements of external support required for practice improvement and for developing ongoing QI capacity.