The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change

While the traditional linear process of intervention development, derived from pharmaceutical medication development models [13], has often resulted in the creation of initially successful interventions, it may be less helpful in enabling these innovations to maximally benefit health. A linear approach may be particularly challenging to apply to complex, multi-component interventions, psychosocial treatments, treatment of the growing number of people with multimorbid conditions [14, 15], and systemic approaches to care [15]. Linear approaches place a premium on creating and 'freezing’ an intervention, developing manuals to ensure its consistent delivery with fidelity [16], and then minimizing deviations from the intervention. For example, the COMMIT stop smoking national project had expert researchers design the complex, lengthy intervention protocol based on research evidence, and then gave a several hundred page manual of operations to local staff to implement in their communities [17].

The importance of internal validity to the scientific process should not be ignored, but its overemphasis relative to generalizability and adaptation runs the risk of creating interventions that will not fit within different, complex or changing settings and of failing to benefit settings, clinicians, and patient populations who are underrepresented in the intervention testing process [7, 16]. Two key implicit assumptions within the traditional intervention development approach may limit ultimate progress toward intervention sustainability and population impact:

First, interventions are often developed with the idea that they can be optimally constructed, manualized, and then tested in a single form applicable across settings and over time. Efficacy trials are designed to screen out noise in the system (patient comorbidities, competing demands and skill variance of clinicians, resource limitations, varying motivations of patients) [7], and thus maximize outcomes. As interventions move to effectiveness and into implementation, one expects that the individual benefit of the intervention will likely drop, due to the added complexity of heterogeneous patients, providers and settings. This is referred to as 'voltage drop’ (Figure 1). The assumption of 'voltage drop’ results in missed opportunities to refine and improve the intervention, instead concluding that the declining benefit is expected and acceptable, and the best possible outcome is that which is achieved at the efficacy stage.

Second, the assumption that interventions can be optimally constructed in the early stages of the development and testing process, independent of context, suggests that, even at the stages of implementation and sustainability, change to the intervention is expected to have negative consequences, and that the further a practitioner deviates from the manual, the lower the benefit. This is the concept of 'program drift’ (Figure 1). Delivering the intervention within an efficacy trial may require adherence to protocols that are challenging to deliver within real-world practice. Fidelity ratings then assume that 100% fidelity to original protocols will yield optimal outcomes, and effort is expended to ensure that practitioners do not deviate from the manual. Where clinicians do deviate from the protocol, the field expects that the resulting 'program drift’ will compromise outcomes [12]. We see that this over-reliance on quality assurance to prevent 'program drift’ leads to extensive pressure on real-world practices to adhere to the intervention protocols without evidence that this adherence will lead to optimal outcomes. Quality assurance may inadvertently hamper sustainability and ongoing improvement, customization and optimization of interventions to the detriment of population health.

In contrast, we reject the notion that an intervention can be optimized prior to implementation, and explicitly reject the validity of 'program drift’ and 'voltage drop.’ Rather, we suggest that the most compelling evidence on the maximal benefit of any intervention can only be realized through ongoing development, evaluation and refinement in diverse populations and systems [18]. Instead of viewing contextual factors as interfering with the delivery of an effective intervention and needing to be controlled, we see the opportunity to learn about the optimal fit of an intervention to different care settings [2]. For example, strategies have been developed to adjust organizational characteristics (e.g., culture, climate, structure) to enable improved fit between the intervention and the setting [19]; harnessing the understanding of context can enable beneficial adaptation of the intervention and improve sustainability. Without rejecting these assumptions, we reify early phase interventions tested in the most artificial settings, set quality assurance of interventions as an optimal outcome, and miss opportunities for continued learning and development.

As the field of implementation science has matured [20, 21], more emphasis has been placed on understanding sustainability. Researchers have recognized that implementation of interventions, which can often require substantial resources, is meaningless without successful long-term use. Following Rabin et al.’s glossary for Dissemination and Implementation Research in Health [9], we draw distinctions between 'implementation,’ which relates to the initial process of embedding interventions within settings and 'sustainability,’ which relates to the extent that these interventions can continue to be delivered over time, institutionalized within settings, and have necessary capacity built to support their delivery.

Recent articles [22‐24] have advanced the idea of an adaptation phase that bridges from the initial implementation effort to a longer-term sustainability phase. They argue the need to examine the fit between the practice setting and the intervention and make changes necessary to improve the integration of the intervention into ongoing care processes. This is consistent with the institutional theory of organizations, which argues that the final stage of innovation requires the 'institutionalizing’ of the new practice so that it becomes a working part of the organization [25].

As a consequence, assessment of organizational characteristics (e.g., structure, climate, culture, resources) is seen as an essential component of sustainability, and indeed, the fit between context and the intervention is at the center of a sustainability phase [24]. There has also been an emphasis on planning for sustainability much earlier in the intervention process [26]. Recent approaches to sustainability locate key efforts squarely in the implementation phase, arguing that once a practice has been implemented within a care system, those who manage the delivery of that practice should turn their attention to ensuring that the practice can be maintained over time [6, 24]. Authors typically suggest that this entails attention to issues of long-term financing, training of the workforce, supervision, and organizational support for the practice [26, 27]. A characteristic of this approach is to postpone emphasis on sustainability until after implementation is well underway, assuming that implementation and sustainability are sufficiently independent.

Authors have also highlighted the utility of assessing outcomes of those who have received the practice, something infrequently collected in routine practice [24, 28]. Seldom is it demonstrated that continued delivery of an intervention confers benefit on the patient population that receives the intervention or the system that delivers it (e.g., cost containment, efficiency of care, quality metrics). Measurement of outcomes over time to determine continued benefit has been shown to support sustainability of the practice [29, 30].

Recent implementation projects have created new tools and scales to study sustainability, including needs assessments, long-term action plans, tracking of program adaptation, financial planning, mapping of community networks, and measurement of the degree to which practices are integrated and institutionalized into service systems [31‐33]. While this emerging focus on sustainability is an advance, many studies still assume a largely static service delivery system that needs to be assessed only at key time points, but not in an ongoing manner. To better reflect complexity and change within the system and in context, a more dynamic approach to sustainability is needed.

Interventions benefit from ongoing optimization as they are applied in different contexts [37]. The evidence that supports the benefits of health interventions arises from trials that represent a very small slice of the diversity of demographics, preferences, and health status of the population at large [7, 40], but we should not expect evidence collected in one set of narrow, relatively optimal circumstances to apply perfectly in other, vastly different contexts [41]. A 'corollary’ of this recommendation is that, other things being equal, quality improvement approaches that involve adjusting and refining program should be more effective than 'quality assurance’ procedures that emphasize fidelity to an initial protocol.

There is tremendous opportunity to aggregate evidence on the real-world impact of interventions when used in practice. We can apply models of continual refinement that have been the cornerstone of software development (e.g., Firefox, Reaper, iTunes), as well as Web 2.0 sites (e.g., Wikipedia, Facebook, etc.) [42]. By augmenting trial data with practice-based evidence, we can understand much more about what works for whom, the question underlying personalized medicine [43, 44]. This articulation of the DSF suggests the need for a long-term plan to commit resources for training and ongoing improvement. One implication of the DSF is that intervention impact can also be enhanced through increases in efficiency. The field has developed a plethora of multi-component interventions, often without studies that determine what the minimal set of components are needed to ensure benefit [45]. The DSF, congruent with the Consolidated Framework for Implementation Research [46], emphasizes the importance of streamlining interventions to peel away components that may not be central to improving outcomes or to adapt intervention components to a particular context.

Too often, intervention trials focus on markers that are psychometrically valid but of less relevance to patients and clinicians [3, 47]. For example, very specific, intervention-related symptomatic scales may be most sensitive to change, but there is little guarantee that the measured change translates into a tangible, functional benefit for the patient [48]. The DSF thus suggests the use of measures such as checklists that are relevant to desired outcomes of patients, as well as sensitive to the 'fit’ between interventions and context, and can be feasibly implemented [49]. Across each of the changes in Figure 2, we see available streams of data that can offer leverage points for improving interventions. Environmental changes, for example, can be tracked via population surveys, and market and claims data. Practice changes can be captured through electronic health records, claims data and practice surveys. Evidence reviews can provide key information on knowledge changes, and policy changes can be tracked via available Federal and Non-Profit sources (e.g., CMS, Kaiser Family Foundation, etc.). This is consistent with the CQI model, but with specific emphasis on the ongoing refinement of the intervention to counteract the assumptions of 'Program Drift.’

We reject the assumption that the more diverse and complex a patient population is, the smaller the benefit of intervention, referred to above as 'voltage drop’ [11]. This stems from an expectation that intervention studies require control of the environment to isolate a treatment effect [7]. If we embrace CQI of the specific health intervention, we expect that with more experience, we will better be able to adapt interventions to contexts and patients. As we learn more about what works and what doesn’t and adjust protocols accordingly, the 'voltage’ could maintain or possibly increase over time. This echoes the computer industry, where each new release of a hardware or software line is expected to be better than the prior version. It also finds consonance with the evolution of the flu vaccine, which is constantly refined in response to the changing nature of the influenza virus each season. A culture of improvement is central to ongoing intervention use and treats improvement of the intervention as central to the sustainability process [33].

The concept of 'fit’ has been discussed by other authors (e.g., Estabrooks, Glasgow, Dzewaltowski), [50] and goes back at least as far as Rogers’ Diffusion of Innovations[51], where the concept of reinvention evoked the notion of departing from the original intervention concept to 'create’ a new version suited to the preferences and constraints of the local context. Fit is a multi-level construct and involves alignment along multiple dimensions [50]. The DSF posits that fit will likely change over time, due to changes in the way in which an intervention is delivered, the characteristics of patients, providers and settings, and the broader ecological system within which healthcare settings reside. Attention to this fit, through ongoing assessment and quality improvement efforts, should improve sustainment and ultimately identify opportunities for intervention improvement.

While training and ongoing analyses exist in many organizations, the demands imposed by multiple levels of change require learning to be central to organizational activity for interventions to become sustainable. The context both within an organization and in the broader ecological system is constantly changing, and requires a 'learning organization’ [52] to engage in problem-solving capacity at multiple levels [53]. Organizational learning should also target appropriate adaptation of evidence-based interventions, possibly in rapid learning cycles [44, 54], followed by ongoing assessment and feedback loops. The DSF is congruent with the concepts of the learning healthcare system, again with the emphasis on learning how to better develop, deliver and sustain interventions.

Continuously engaging stakeholders throughout the planning, implementation and adaption processes should help increase the fit between the intervention and the local context, and help address evolving issues that might interfere with sustainability. Just as researchers have proposed intervention development processes [12] that focus on the ultimate site where interventions will be delivered, we argue that partnership among all relevant stakeholders is essential to maintaining and improving interventions within care settings.

As Figure 3 depicts, we view sustainability as akin to the challenge of fitting a puzzle piece within an evolving large tableau. Without sensitivity to the characteristics of the intervention, practice setting and the larger system, there is little expectation that the intervention will fit well within the setting, and as the context changes (noted by the changing shape in the figure), sustainment will be harder and harder to achieve. However, by examining and adapting the intervention to a changing context, we believe that sustainment is not only possible, but that the utility of the intervention can be optimized. The experience of delivering the intervention in vivo over time serves to inform the ongoing evolution of the intervention (noted by the change in shape of the puzzle piece). By concentrating on the dynamic 'fit’ between an intervention and its delivery context as the core ingredient underlying sustainability, we embrace opportunities to refine and improve the intervention.

Table 2 compares static views of sustainability with the DSF, offering sharp contrasts in data collection and analysis, opportunities for knowledge development and incorporation of the 'noise’ within healthcare contexts. Rather than seeking to simplify the phenomenon of study, either by avoiding adaptation of interventions, or assuming the context to be unchanging, the DSF embraces change as a central influence on sustainability. Adaptation is expected, and even encouraged. Assessment of care settings and outcomes is ongoing and incorporated within practice, and staffing and policy changes are incorporated in sustainability planning. Perhaps the biggest contrast of the static and dynamic views is that the static view limits lessons that can in turn provide feedback to other areas of science; the DSF views an abundance of ongoing evidence that can be cycled to continuously improve intervention design, testing, and ongoing system change.

This initial formulation of the DSF has implications for future practice, research and policy. For practice, the DSF highlights the need for continuous assessment of the local context, not just prior to implementation. This enables care settings to better manage the fit between their resources, needs and the interventions, including generating consistent feedback on how interventions are delivered to diverse patients and how patients do as a result. The collection and analysis of this information allows practitioners to make informed decisions about how best to utilize existing interventions, allows for potential enhancements to the interventions to be made and shared, and offers better information on which to make decisions to cease delivering interventions that do not have benefit. The intention is to recognize and support rapid learning, real-time problem-solving organizations [54] that are full partners in the generation of knowledge, not just its application. Thus, the DSF promotes the use of multiple methods of planning for sustainability, including simulation modeling of the impact of different decisions, pilot testing of adaptations within local contexts, and continued experimentation. Perhaps an even greater benefit to practice would come through pooling of data across a larger set of sites, practitioners and patients, something done with success for chronic disease [55, 56].

For research, the DSF dispels the notion that intervention development, refinement and improvement are completed prior to real world implementation. In contrast, we suggest that development and refinement is never complete. Rather, sustainability is the process of managing and supporting the evolution of an intervention within a changing context. We recommend (and welcome testing of the idea) that programs that monitor context and adjust accordingly do better long-term. In addition, we see research studies testing whether settings and programs using ongoing CQI or other means of feedback improvement perform better over time. More broadly, we see the DSF as changing the notion of a linear transition from research to practice into a shared process of continual experimentation and analysis through the use of both practice settings and ecological systems to track changes and assess evolving fit between interventions and practice settings. Principles of 'crowd sourcing’ made popular within the IT industry and resulting in open source products like Firefox, Wikipedia and Reaper, blur the lines between research and practice. We see this in the evolution of the electronic health record, and the rapidly evolving patient health record, which is dramatically changing the scenario from one where the medical expert makes all the decisions to one of collaborative care and shared decision-making [57, 58]. Instead of a small team of researchers developing a priori an 'optimal,’ static product, a large and often virtual community including users and consumers continuously upgrades dynamic products.

This initial formulation of the DSF also has implications for policy. Incentives are needed to support ongoing adaptation of interventions, particularly where evidence is limited, specifically including monitoring of progress and documentation of adaptations, using quality measures relevant to stakeholders and patients. In addition, research funders must determine how to support longer-term projects related to sustainability with flexible research designs, since the ultimate benefit of integrating and modifying interventions may not be evident for many years. In addition, infrastructure to support pooling of 'practice-based evidence’ will be needed, in order to ensure sufficient information is available about long-term use and adaptation of interventions. The DSF aligns directly with a number of existing policy initiatives, at national, state and local levels, including the advance of Patient-Centered Medical Homes, Accountable Care Organizations, Pay for Performance initiatives, and support for local demonstration projects.

We recognize that this conceptualization of the DSF model should, consistent with its internal logic, be refined and improved over time. Whether this happens through testing of the tenets laid out in the previous section or contributions of others’ theoretical or empirical studies, we offer the DSF as the beginning of a longer debate. For example, while Figure 2 shows three levels (intervention, practice setting, ecological system), we appreciate that many more levels of the system exist than what we have depicted in the figure. We see further specification of the interrelationships of those levels as a useful area for further research and development. In addition, we see the utility of aligning the DSF with alternative methods of developing and testing dissemination and implementation interventions from the prevailing linear model. In an effort to present examples of how the DSF might be used to consider the sustainability of various types of interventions, we have included Table 3.