Formative evaluation and adaptation of pre-and early implementation of diabetes shared medical appointments to maximize sustainability and adoption

Understanding the many factors that influence implementation of new programs, in addition to their success or failure, is extraordinarily complex. Formative evaluation, defined as a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts, is an essential means to systematically approach this complexity [1].

Many implementation studies rely exclusively on summative data-- or data outputs, products, and outcomes—to determine program success or failure. While summative data is useful, it is not adequate to understand critically important implementation processes [1, 2]. When used in isolation, summative data often leads to the term “implementation black box” [3]: there is no way to understand the specific reasons the intervention succeeded or failed, how it was actually implemented, and how local contextual factors affected implementation. This is where formative evaluation comes in, to fill in these gaps and to systematically examine key features of the local implementation setting, detect and monitor unanticipated events and adjust if necessary in real-time, optimize implementation to improve potential for success, and avoid type III errors—the failure to detect differences between the original intervention plan and the ultimate manner of implementation that lead to failure to achieve outcomes [1, 4]. This understanding is essential for efforts to sustain, scale up, and disseminate any new program–otherwise there is potential for failure to account for specific contextual issues in program implementation.

Adaptations have been found to be necessary for sustainable implementation [5] and are considered part of the traditional translation pipeline (adaptations as required) [6]. Progress has been made in advancing the science of implementation, but too often the complexity of translating research into practice is overlooked or follows an overly simplistic model [7]. In the words of implementation experts Dr. Chambers and Dr. Norton, “Rather than assuming that adaptation of a manualized intervention is at odds with good implementation, the field can systematically collect information on the impact of adaptation to individuals, organizations, and communities and use this information to extend the knowledge base of implementation of evidence-based practices as well as ongoing improvement of the evidence-based practices themselves” [8]. Indeed other efforts in implementation have been described where adaptations are catalogued on an ongoing basis. [9] While the static view of adaptation has been that it is bad or to be avoided and/or eliminated, the dynamic sustainability model believes adaptation to be “inevitable and encouraged.” [7] Likewise, simplified intervention implementation overlooks the complexities of translating research to practice and relies on a set of assumptions that limits enhancement of fit between evidence-based interventions and delivery setting. [10]

Additionally, while the evidence-to-practice gap for interventions is receiving attention, it tends to be understudied in primary care. [11] A recent study highlighted the importance of paying attention to context, which was noted as frequently failing to be acknowledged, described, or taken into account during implementation in primary care. [11]

Accordingly, to fill this gap in the literature, we sought to illustrate the role and importance of pre-implementation (early) interviews for guiding ongoing adaptations to improve implementation of a clinical program, achieve optimal change, and avoid type III errors. We gathered detailed pre-implementation data across five health system sites, within primary care settings, that had each committed to institute Shared Medical Appointments (SMAs) and in some SMA cohorts an additional offered mutual peer support program (P2P) for adult patients with poorly controlled diabetes. In particular, we examined modifiable barriers and facilitators to implementation. We then documented the adaptations that were made in real-time to attempt to improve implementation. This comprehensive data will also facilitate and enhance understanding of the future success or lack of success of implementation of new innovative clinical programs such as SMAs and inform potential for translation and public health impact of these [2].

This qualitative study is part of a larger implementation study, “The Shared Health Appointments and Reciprocal Enhanced Support (SHARES) study” [12]. The SHARES study is a multi-site cluster randomized trial of five geographically diverse Veterans Affairs (VA) health systems evaluating the effectiveness and implementation of diabetes Shared Medical Appointments (SMAs) with and without an additional reciprocal Peer-to-Peer (P2P) support program, when compared to usual care.

SMAs bring patients with the same chronic condition together with an interdisciplinary team of providers to provide shared education and support. The diabetes SMAs consist of a series of 1–2 h sessions of 8–10 patients led by a team of health professionals. At each site, participants have a total of approximately 8 h of sessions, and the sessions are intended to be interactive and focus on key diabetes self-management topics (e.g., diet, medications, physical activity, self-monitoring). The P2P program comprises periodic peer support, [13, 14] group sessions, and telephone contact between SMA participant pairs to promote more effective diabetes self-management and sustain gains achieved through the SMAs after completion of these sessions. Outcomes will be examined across three different treatment groups: (1) SMAs; (2) SMAs plus P2P; and 3) usual care.

We undertook a type of formative evaluation (FE), labeled implementation-focused evaluation, of the SMAs and of P2P [1]. This type of FE occurs throughout implementation of the project plan: before, during, and after implementation. This manuscript focuses on the real-time FE that took place during the early stages of implementation and pre-implementation.

The overarching framework for the SHARES qualitative discovery-- including the interviews, formative evaluation, implementation, and analysis-- was the Consolidated Framework for Implementation Research (CFIR) [15]. The CFIR provides a framework of 39 constructs from across published implementation frameworks that describe the organizational and contextual setting and are believed to influence implementation. We are tracking the CFIR constructs across the sites throughout the project and will ascertain how they influence implementation success during future program evaluation.

Institutional Review Board (IRB) approval was obtained from the Central IRB.

Key constructs or areas of focus varied across sites. Table 2 is organized by site and CFIR construct and within that row, highlights how the findings (barrier column) informed our actions to improve or facilitate implementation (facilitator column) and what actions were taken (adaptations).

Based on our work with the participating sites to date, we identified a subset of CFIR constructs that are most likely to play a role in the implementation and possibly effectiveness of the diabetes SMAs and the P2P program, including evidence strength and quality, relative advantage, adaptability, complexity, patient needs and resources, compatibility, leadership engagement, available resources, knowledge and beliefs, and champions (Table 4). Although data from any organizational aspect mentioned by interviewees, and all CFIR constructs, were coded (see Table 2), these specific constructs formed the basis of primary qualitative analyses due to the depth and frequency of the construct throughout the interviews and qualitative analysis. A definition of each CFIR construct is included in Table 2.

In this article, we described our method for identification of contextual factors that influenced implementation of complex diabetes clinical programs -SMAs and P2P. The qualitative phone interviews aided implementation through the identification of modifiable barriers or conversely, actionable findings. Implementation projects, and certainly clinical programs, do not have unlimited resources and these interviews allowed us to determine which facets to target and act on for each site.

Likewise, our facilitation team used formative evaluation to understand the context and organizational issues at each of these VA health systems. Our approach used the CFIR to guide us to actionable findings and help us better understand barriers and facilitators, and variations of those, across constructs and sites. Using the CFIR to do this allowed us to improve the generalizability and efficiency of our findings by highlighting factors that prior research have identified as influencing implementation (each of the published constructs). Additionally, our project team and the local sites benefited from the use of formative evaluation throughout the early implementation process; we identified, in an ongoing manner, problems that we had not anticipated but that needed to be addressed to optimize implementation.

The implementation of a new clinical program is very complex and the field has recognized the need to utilize theoretical bases of implementation to facilitate implementation itself and there have been more calls for researchers to utilize existing frameworks to gain insights into the mechanisms by which implementation is more likely to succeed and to achieve common terminology. [21] We believe our approach of using the CFIR to accomplish those goals has broad applicability and can be used by other projects to guide, adapt, and improve implementation of research into practice.

We have illustrated how pre- and early-implementation FE is critically important in preparing for and gaining early understanding of key factors that influence implementation processes, and future success or failure. This early formative research shaped our implementation to minimize type III failures. Our rich examples highlight areas that were challenging as well as those that facilitated implementation of both shared medical appointments, and peer-to-peer programming. It is important to note that there was no site that was universally positive or negative across constructs, as often is assumed of “laggard” or “early adopting” sites. Evidence strength and quality was a negative issue at only one site (1002), but it was very impactful there (see Table 3, − 2 ranking) and important for the facilitators to be aware of and to work to overcome. As a result, our team presented evidence during the site visit and during a local primary care team meeting to help educate and influence the physicians. In contrast patient needs and resources was a negatively rated construct at 4 of the 5 sites, but at the 5th (1005) was a positively ranked construct—illustrating that implementation scientists need to be very cautious of labeling any construct as universally problematic.

Because of our intentional broad range of interviewees, CFIR constructs could be mixed within a site. When this was the case, the findings were discussed, weighed and used to come up with one overall score as per CFIR guidelines. The process is similar to consensus-based coding; “Analysts apply a summary rating, taking all the individual ratings and supporting qualitative summary and rationale into consideration, and then discuss ratings to achieve consensus” [22].

There are several limitations to this study. Constructs were assigned only one rating per site using weighted data from all respondents. Additionally, it is challenging for implementation researchers to identify when modifications create an additional intervention; however, in this case we classified these as local adaptations because the core underlying conceptual nature of the interventions was maintained in each case. Program drift is sometimes thought of as resulting in lower intervention success due to lack of fidelity [7]. However, experts in the field recognize that view is overly simplistic and encourages an unnecessarily rigid view of fidelity; “this designation decreased opportunities to learn from evidence-based intervention adaptations that result in improvements beyond what is expected” [10].

As the SHARES study progresses, these findings will be compared and correlated to outcome measures. This comprehensive adaptation data collection will also facilitate and enhance understanding of the future success or lack of success of implementation and inform potential for translation and public health impact. Crossing the bridge from research to practice in primary care and family practice settings is crucially important because in many ways we are not reaping the full public health benefits of our investment in research. [23] While the evidence-to-practice gap for interventions in primary care is receiving attention, it tends to be understudied. [11] We believe our approach of using the CFIR to guide us to actionable findings and help us better understand barriers and facilitators, has broad applicability and can be used by other projects to guide, adapt, and improve implementation of research into practice in primary care and other clinical settings.