Background
Understanding how to address issues of sustainability is “one of the most significant translational research problems of our time” [
1]. To achieve lasting effects on health, it is essential to sustain the implementation of evidence and the ensuing outcomes. In their 2004 systematic review on implementation, Greenhalgh and colleagues noted the “near absence of studies focusing primarily on the sustainability of complex service innovations” [
2]. Since then, the literature continues to grow, providing evidence-based guidelines [
3‐
5], registries of evidence-based programs [
6,
7], systematic reviews on the effectiveness of implementation strategies, and organized lists of implementation strategies [
8‐
10]. However, much of the literature on sustainability remains theoretical, with little guidance on how to sustain program (or the clinical intervention) delivery, implementation strategies, and outcomes [
11,
12].
Two of the foundational challenges of sustainability are the lack of a standard definition for the term and the variety of synonyms that are used in the literature [
1]. Without a standard, widely accepted definition, it is unclear how researchers operationalize and measure sustainability. Indeed, in their development of a research agenda in sustainability, Proctor and colleagues prioritized improving clarity in sustainability terminology and concepts [
1]. The use of different terms leads to challenges in finding the literature on sustainability, which in turn hamper the ability of researchers to grow this science and avoid duplication of effort. Commonly used alternative terms for sustainability include maintenance, continuation, institutionalization, routinization, and durability [
11,
13]. These definitional challenges have likely arisen because multiple disciplines (e.g., medicine, health systems, child welfare, prevention science, education, justice, and juvenile justice) [
14] are addressing similar problems.
Most implementation studies do not include an explicit definition of sustainability [
11]. For example, in an early systematic review of sustainability, published in 2005, Scheirer noted that few studies provided a definition [
13]. In 2016, Tricco and colleagues found that 8 (13%) of the 62 studies in their scoping review of interventions in healthcare decision-making related to chronic disease management included a definition of sustainability [
12]. In their systematic review on the sustainability of clinical practice guidelines, Ament and colleagues noted that only 2 (11%) of the 18 studies provided a reference for their respective definitions of sustainability [
15]. One explanation for the lack of referenced definitions could be that selecting a definition can be challenging, given that sustainability can refer to the sustained delivery of a clinical intervention or individual changes in behavior and can occur at multiple levels (e.g., patient, provider, organization, community, system) and given that commonly used definitions describe different constructs (see Table
1).
Table 1
Sustainability definitions
“Sustainability of organizational innovations can be thought of as the point at which new ways of working become the norm and the underlying systems and ways of working become transformed in support” [ 2]. |
“The simplest definition of sustainability is the ‘capability of being maintained at a certain rate or level’” [ 20]. |
“We use the term sustainment to denote the continued use of an innovation in practice” (Aarons et al. [ 33]). |
“(1) [W]hether, and to what extent, the core elements (the elements mostly closely associated with desired health benefits) are maintained; (2) the extent to which desired health benefits are maintained and improved upon over time after initial funding or supports have been withdrawn; (3) the extent, nature, and impact of modifications to the core and adaptable/peripheral elements of the program or innovation; (4) continued capacity to function at the required level to maintain the desired benefits” [ 11]. |
Building on existing sustainability research and the recommendations from Proctor and colleagues, our goal for the current study was to systematically develop a comprehensive definition of sustainability based on definitions already used in the literature [
1].
Results
We identified four published knowledge syntheses of sustainability [
11,
12,
15,
20]. Inclusion criteria varied across the four reviews. Gruen and colleagues included both conceptual frameworks and empirical studies about program sustainability within healthcare organizations or in community settings [
20]. Stirman and colleagues searched for peer-reviewed studies that addressed the sustainability of specific interventions or programs, were written in English, and were published or in press by July 2011; they included all studies in which the authors used one of a predetermined set of terms to describe sustainability or made an effort to determine the extent to which a program or intervention continued after an initial period of training, implementation, or study [
11]. Ament and colleagues included studies of sustainability that had at least two measurements (other than self-reporting), obtained before and immediately after implementation, of professionals’ adherence to a clinical practice guideline [
15]. Tricco and colleagues included studies with an experimental, quasi-experimental, or observational (with one or more comparator groups) design, conducted in any clinical setting and involving adults with a chronic disease (excluding mental illness) who received a knowledge translation intervention (targeting the patient, a healthcare provider, or the health system) that lasted more than 1 year after implementation or termination of study funding [
12]. The four reviews encompassed a total of 240 original research publications (39 from Gruen and colleagues [
20], 125 from Stirman and colleagues [
11], 14 from Ament and colleagues [
15], and 62 from Tricco and colleagues [
12]); because of overlap among the reviews, there were 209 unique citations.
Phase 1: abstracting sustainability definitions from individual articles
In phase 1, data abstraction, 24 (11.5%) of the 209 eligible articles provided a definition of sustainability.
Phase 2: familiarizing with definitions and consolidating key constructs
In phase 2, the three reviewers identified a total of 94 constructs (32 by AM, 32 by JB, 30 by JEM), which were consolidated into nine constructs not included in the Scheirer definition. All three coders identified the following three constructs: institutionalization (i.e., establishment of a new practice, program, or clinical intervention that becomes the norm within an organization or other setting, such as a particular community; 29/94 [30.9%]), routinization (i.e., establishment of a new practice at an individual level; 20/94 [21.3%]), and adaptation or evolution (i.e., change in the nature of programs, implementation strategies, and individual behavior, in response to changes in the broader ecological context; 21/94 [22.3%]). The remaining six constructs were each identified by a single coder: time (9/94 [9.6%]), improvement trajectories (i.e., outcomes are not only maintained but steadily improve over time; 4/94 [4.3%]), benefits and partnerships with stakeholders (4/94 [4.3%]), maintenance of core elements of evidence-based programs (3/94 [3.2%]), implementation as sustainability (i.e., the implementers are both the means of achieving sustainability and part of the outcome; 2/94 [2.1%]), and interactions with the environment (i.e., achievement of sustainability not in isolation from the environment, but as a result of interactions between the program and the environment; 2/94 [2.1%]).
We reviewed all of these nine constructs and expanded our list of constructs for analysis. A key theme from the review of constructs was that some definitions considered the implementation of evidence-based programs (e.g., multidimensional treatment foster care [
21]), whereas others focused on the implementation of a specific evidence-based practice (e.g., recommendations in a clinical practice guideline) that was not part of a program. Definitions referring to evidence-based programs primarily focused on continuing program delivery at the organizational level, using descriptions such as “the continued use of an intervention” [
22], “program or policy becomes institutionalized or part of the routine organizational practices and policies” [
23], “program components…are maintained” [
13], or “core elements are maintained” [
11]. Definitions that focused on evidence-based practices described maintenance of a behavior by individuals: “new ways of working…become the norm” [
24], “new working methods…are maintained” [
25], or “enduring part of the behavioral repertoire of an individual” [
23]. Some of the discrepancies across definitions were related to different considerations of what was being implemented: individual behavior change or organizational programs. To address these two perspectives, our comprehensive definition includes constructs related to both maintenance of behavior change by individuals (e.g., program recipients, patients, caregivers, or clinicians) and continued delivery of a program at the organizational level (e.g., organizational or community-level implementation of programs).
Several definitions suggested that sustainability involves, in part, the evolution or adaptation of programs, implementation strategies, or practices over time. Examples included “a suggested ‘adaptation phase’ that integrates and institutionalizes interventions within local organizational and cultural contexts” [
22], “the dynamism of continuing change” [
26], and “creating an environment for innovations to adapt to the system, if necessary” [
27]. Some of the definitions referred to “time,” either providing specific criteria (e.g., 2 years after end of funding) or using vague allusions (e.g., after initial funding ends). We believe that the timeline for sustainability depends on the individual practice or program and the outcomes of interest, so our definition does not specify a particular timeline; nonetheless, we believe it is essential for operational definitions of sustainability to specify a timeline. Therefore, our comprehensive definition of sutainability includes the following five constructs: (1)
after a defined period of time, (2)
the program, clinical intervention, and/or implementation strategies continue to be delivered and/or (3)
individual behavior change (i.e., clinician, patient) is maintained; (4)
the program and individual behavior change may evolve or adapt while (5)
continuing to produce benefits for individuals/systems.
Phase 3: mapping definitions to the consolidated list of constructs
In phase 3, we re-mapped the new sustainability constructs from the definitions (Table
3). Inter-rater reliability for the mapped constructs was 94%, indicating substantial agreement among the three reviewers [
28]. Of the 24 definitions, 17 (71%) described the continued delivery of a program, 17 (71%) mentioned continued benefits or outcomes, 13 (54%) mentioned time, 8 (33%) addressed maintenance of behavior change by individuals, and 6 (25%) described evolution or adaptation of a behavior or practice. Additional file
2 provides quotes from the source definitions describing each of the five constructs to allow future researchers to use alternative wording for individual constructs in adapting our definition to their own projects.
Table 3
Summary of definition references abstracted to sustainability constructs
Aarons et al. (2011) [ 33] | | | X | | |
| X | X | | | |
| X | X | | | X |
Buchanan et al. (2005) [ 25] | X | | X | | X |
Chambers et al. (2013) [ 22] | X | X | | X | X |
| | | X | | X |
Evashwick and Ory (2003) [ 37] | X | X | | | |
Fleiszer et al. (2015) [ 26] | X | X | | X | X |
Glasgow et al. (1999) [ 23] | X | X | X | | X |
Goodman et al. (1993) [ 38] | | X | | | |
Greenhalgh et al. (2004) [ 2] | | | X | | |
| | X | | | X |
Johnson et al. (2004) [ 27] | | X | | X | X |
Mancini and Marek (2004) [ 39] | | X | | | X |
National Health Service (2007) [ 24] | X | | X | X | X |
| | | | | X |
Paine-Andrews et al. (2000) [ 30] | X | X | X | | |
| | X | | X | |
| X | X | | | X |
Schell et al. (2013) [ 42] | X | X | | | X |
Shediac-Rizkallah and Bone (1998) [ 19] | | X | | | X |
Stetler et al. (2007) [ 43] | X | | X | | X |
Stirman et al. (2012) [ 11] | X | X | | X | X |
Swerissen and Crisp (2004) [ 44] | | X | | | X |
Discussion
We identified five key constructs that describe individual and organizational sustainability. Our definition of sustainability was crafted to encompass these five key constructs: (1) after a defined period of time, (2) the program, clinical intervention, and/or implementation strategies continue to be delivered and/or (3) individual behavior change (i.e., clinician, patient) is maintained; (4) the program and individual behavior change may evolve or adapt while (5) continuing to produce benefits for individuals/systems.
The five constructs are distinct yet inter-related. For example, continued delivery of a clinical intervention or program (i.e., institutionalization) refers to whether an organization or community is continuing to provide a program (e.g., delivering multidimensional treatment foster care) or continuing to use the strategies necessary to support behavior change (e.g., education, audit, and feedback). Maintenance of behavior change at the individual level pertains to whether the implementer is following the recommendations of the evidence-based program, guideline, or practice (e.g., how the implementer is interacting with patients, clients, or community members). Both institutionalization and maintenance may evolve over time. The advantage of implementing evidence-based programs, rather than newly developed and untested programs, is that if such programs are delivered with high implementation quality, they can be expected to produce anticipated outcomes [
29]. However, initial program outcomes will not necessarily continue following the implementation phase; therefore, maintaining outcomes is considered a separate construct, one that should be measured separately from program delivery and maintenance of behavior change. Institutionalization, maintenance, and benefits can each be defined and measured over a period of time.
Continued delivery of a program or a set of implementation strategies (i.e., institutionalization) and maintenance of benefits were the most commonly included constructs. The prominence of these two constructs (cited together in 12 [50%] of the 24 definitions) is not surprising, given that the majority of individual studies focused on the implementation and sustainability of evidence-based programs, rather than individual-level practice changes. Only 2 (8%) of the definitions incorporated both maintenance of a behavior by individuals and continued delivery of the program [
21,
30].
Just over half of the definitions (13/24 [54%]) mentioned the word “time.” However, all but three of these time-related references were undefined (e.g., “over time” or “after funding ended”); continued program delivery, maintenance of behavior change, and outcomes can all be measured over time. Depending on the nature of the intervention and its outcomes, the appropriate time to start measuring sustainability will vary.
Evolution and/or adaptation constituted the least commonly described construct, mentioned in 25% of the definitions. Adaptation can refer to either changes in the program or implementation strategies or changes in an individual’s maintenance of a behavior. Given that sustainability frameworks, such as the dynamic sustainability framework [
22], strongly emphasize the evolutionary nature of evidence, programs, and practices, it seemed imperative to capture this concept in the definition.
Some definitions included the related assumptions that researchers were measuring the continuation of a program and that initial implementation would be affected by context but the program itself (or maintenance of the behavior) would not change because of context [
23]. In these cases, contextual factors are conditions that may increase or decrease the likelihood of sustainability [
11]. Some definitions described the program’s level of stability [
23]. Other researchers approached sustainability from more of an ecological perspective, whereby the program and the environment were perceived as being interconnected [
13,
22]. These various approaches may yield different research study questions: (1) factors or determinants affecting sustainability or (2) ways in which the environment and program adapt and evolve together. Additionally, each of these approaches has significant implications for how people plan for sustainability.
We flagged several constructs that are associated with sustainability, for example, factors that affect the evolution of a program or practice over time. These were similar to the influences on sustainability identified by Stirman and colleagues in their review, which included the following broad categories: innovation characteristics, context, capacity, processes, and interactions [
11]. We did not include these predictive constructs in our definition of sustainability, although they are integral to an understanding of evolution and dynamism and should be considered when planning for and measuring sustainability.
Fleiszer and colleagues completed a concept analysis on the sustainability of healthcare innovations where they described three characteristics of sustainability: continued benefits, routinization/institutionalization, and development [
26]. Authors described their conceptualization of sustainability as needing further development prior to measurement or testing. Our comprehensive definition builds on this work by distinguishing between routinization (i.e., individual level change) and institutionalization (organization/system level change) and by including the construct of time.
The lack of a comprehensive definition of sustainability has been a foundational challenge to moving the field forward because a definition may help implementers think about
what it is they hope to sustain [
12]. In a recent systematic review on the sustainability of health interventions in sub-Saharan Africa [
31], only half of the studies (
n = 21, 51.2%) clarified what it is they are sustaining using different definitions from the literature. The majority of all studies (
n = 19, 46.3%) reported outcomes on the continued delivery of a program (46.3%). Similar results were found in a scoping review on the sustainability of chronic disease health programs [
32] the majority of studies (
n = 37, 88.1%) included sustainability indicators related to the maintenance of program activities. Furthermore, the system or organization was the unit of analysis for the majority of studies, with only 23.8% (
n = 10) of studies measuring individual-level sustainability outcomes [
32]. These results suggest a more narrow understanding of sustainability than what we outlined in our definition in the current study. Our comprehensive definition of sustainability prompts implementers to think about what it is they hope to sustain at the individual level, the organization/system level, and the level of intervention outcomes.
A few limitations of this study should be noted. First, we did not conduct a systematic search for articles about sustainability; rather, we identified primary articles only from the existing knowledge syntheses. Inclusion criteria varied within each knowledge syntheses, and therefore, we might have missed some articles that included definitions of sustainability. In particular, newer articles (published in 2015 and 2016) are not likely to have been identified by our method, and the search was limited to articles published in English. Second, this search of knowledge syntheses was only performed in one healthcare database. There may be knowledge syntheses on sustainability in literature outside of healthcare (e.g., social services research) that could have been excluded from our study, and there may be knowledge syntheses we missed from other healthcare databases. Third, given differing perspectives on sustainability (e.g., the ecological approach emphasizing evolution and the determinant approach focusing on stability), other researchers analyzing the same literature might have drawn different conclusions about the constructs to be included in the definition. Future research could build on our work by conducting a more comprehensive search for sustainability definitions and then testing the constructs from our work in that literature.