Background
Project aims and rationale
Methods
Phase one: identifying and recruiting sustainability experts
Phase two: conducting CM research on sustainability
Phase three: action planning through an in-person working conference
Phase four: generating and consolidating recommendations
Results
Cluster 1: RQ: organizational issues | ||||||||
83 | Paying special attention to issues related to the organizational learning curve and other issues related to evolution or drift of the practices | .41 | ||||||
77 | Understanding when innovation or the natural life cycle of a program takes precedence over sustainability | .43 | ||||||
34 | Identifying organizational adopter/sustainer archetypes | .45 | ||||||
30 | Considering the organization as a learning organization rather than sustainability as an end-point | .52 | ||||||
76 | Developing guidance for how organizations should decide to end a program and instead adopt a newer/better/more effective one | .69 | ||||||
5 | Understanding the “rapid learning” or problem solving skills needed by key individuals/leaders and organizations in order to respond to changing environmental challenges | .78 | ||||||
Count: | 6 | Std. Dev.: | 0.14 | Minimum | .041 | Average: | .55 | |
Variance: | 0.02 | Maximum | .078 | Median: | 0.49 | |||
Cluster 2: research and program funding | ||||||||
70 | Funding | .09 | ||||||
67 | Finding funding sources for conducting sustainability research | .22 | ||||||
86 | Understanding and commitment on the part of funders for such work | .46 | ||||||
82 | The availability of RFAs that explicitly call for sustainability research | .48 | ||||||
6 | Determining why implementing agencies do/do not seek continued funding | .51 | ||||||
19 | Fully exploring the role of adequate funding in sustainability | .62 | ||||||
41 | Funds available to sustain a program | .70 | ||||||
36 | Making sure that the research addresses required infrastructure and a viable business model to provide longstanding revenue support for the program | .73 | ||||||
Count: | 8 | Std. Dev.: | 0.21 | Minimum | 0.09 | Average: | .47 | |
Variance: | 0.04 | Maximum | 0.73 | Median: | 0.49 | |||
Cluster 3: research practice and training | ||||||||
22 | Establishing training in sustainability research | .20 | ||||||
15 | Developing and implementing graduate curriculum relevant for sustainability researchers | .29 | ||||||
56 | Training and capacity building in a public health agency | .42 | ||||||
91 | Training and technical support to providers/deliverers of the program/intervention | .49 | ||||||
45 | Identifying a broad set of journals and professional conferences that are good places for dissemination of sustainability research | .68 | ||||||
Count: | 5 | Std. Dev.: | 0.16 | Minimum | 0.20 | Average: | .42 | |
Variance: | 0.03 | Maximum | 0.68 | Median: | 0.42 | |||
Cluster 4: research question: ROI | ||||||||
78 | Determining the return-on-investment of sustainability given a variety of likely parameters (e.g., number of individuals reached; number of infections averted) | .43 | ||||||
50 | Conducting ROI (return on investment) studies to make it clear to stakeholders and funders how much is actually gained when effective programs are sustained | .43 | ||||||
33 | Determining the return-on-investment of sustainability during different time periods (e.g., 6-months, 12-months, 24-months) | .46 | ||||||
9 | Investigating to what extent benefits (e.g. cost-savings, improved clinical outcomes) are sustained along with sustained use/behavior | .47 | ||||||
84 | Conducting longitudinal cost-benefit analysis comparing implementation vs. sustainability | .50 | ||||||
46 | Considering cost and economic issues from multiple perspectives | .66 | ||||||
23 | Taking advantage of research already conducted in other fields and industries regarding sustained implementation of technologies and practices | .75 | ||||||
74 | Creating survey of disciplines most involved in sustainability research | 1.00 | ||||||
Count: | 8 | Std. Dev.: | 0.19 | Minimum | 0.43 | Average: | .59 | |
Variance: | 0.04 | Maximum | 1.00 | Median: | 0.48 | |||
Cluster 5: research question: factors affecting sustainability | ||||||||
13 | Figuring out how to predict sustainability based on the experience or knowledge of context gained through implementation | .23 | ||||||
39 | Identifying key features of Evidenced Based Practice associated with variations in sustainability (e.g., strength of evidence; consistency with established practice; size of difference between prior and the new/evidence-based practices; staffing/other) | .25 | ||||||
53 | Understanding which variables and factors are more important for sustainability than others | .25 | ||||||
7 | Identifying key or core program sustainability components | .25 | ||||||
75 | Investigating the relationship between sustained use, routinization and resistance to change | .29 | ||||||
12 | Exploring whether the factors influencing sustainability differ from those influencing implementation | .30 | ||||||
60 | Understanding if certain types of programs are less likely to be sustained | .30 | ||||||
69 | Identifying common and independent factors that drive adoption vs. initial implementation vs. long-term use | .35 | ||||||
42 | Exploring the supporting interventions (e.g., feedback) that are needed to sustain behaviors/use, for how long, and what intensity | .35 | ||||||
88 | Understanding the reasons why strategies are/are not sustained | .39 | ||||||
Count: | 10 | Std. Dev.: | 0.05 | Minimum | 0.23 | Average: | .30 | |
Variance: | 0.00 | Maximum | 0.39 | Median: | 0.30 | |||
Cluster 6: research question: adaptation | ||||||||
64 | Understanding the tension between fidelity and adaptation as it pertains to sustained use or feasibility of continued use | .13 | ||||||
21 | Documenting adaptations and their impact on the effectiveness of evidence-based practices | .14 | ||||||
47 | Determining the core vs. peripheral or adaptable components of interventions | .14 | ||||||
55 | Figuring out how to characterize adaptations | .16 | ||||||
54 | Determining the point at which the intervention or program can no longer be considered sustained because of extensive adaptations | .18 | ||||||
66 | Considering how the evidence-based program may change over time | .19 | ||||||
38 | Understanding the implications of “partial” sustainability and adaptations | .24 | ||||||
28 | Identifying interventions that are effective and cost-effective in fostering sustainability | .33 | ||||||
4 | Framing sustainability as a partnership in which participants continue to adapt an intervention in response to changing conditions, while trying to remain true to its core principles | .34 | ||||||
Count: | 9 | Std. Dev.: | 0.08 | Minimum | 0.13 | Average: | .21 | |
Variance: | 0.01 | Maximum | 0.34 | Median: | 0.18 | |||
Cluster 7: research question: environment | ||||||||
17 | Defining the key attributes of organizations and systems that successfully sustain effective practice (e.g., ongoing leadership attention, ongoing measurement, systematic hardwiring of effective innovation, etc.) | .20 | ||||||
49 | Identifying the key contextual factors (e.g., organizational characteristics) associated with variations in sustainability | .21 | ||||||
58 | Defining and assessing the multiplicity of environmental variables that are likely to affect sustainability | .26 | ||||||
51 | Discerning situations in which sustained use may be at odds with adopters’ (e.g., organizations) best interests | .31 | ||||||
8 | Characterizing the context or environment of the intervention to be sustained | .33 | ||||||
65 | Understanding cultural barriers to adoption | .33 | ||||||
62 | Identifying which factors that advance or inhibit sustainability are amenable to management intervention | .53 | ||||||
63 | Understanding how to sustain programs/policies in low resource settings | .71 | ||||||
Count: | 8 | Std. Dev.: | 0.16 | Minimum | 0.20 | Average: | .36 | |
Variance: | 0.03 | Maximum | 0.71 | Median: | 0.32 | |||
Cluster 8: research stage: measurement | ||||||||
2 | Developing methods for studying sustainability across the complexity dimension (e.g., sustainability of a specific clinical treatment vs. sustainability of a complex state-level chronic disease program) | .00 | ||||||
3 | Using multilevel measurement | .00 | ||||||
37 | Developing measures of sustainability (overall and sub-dimensions) | .01 | ||||||
18 | Determining which analytic methods are most appropriate for sustainability research | .02 | ||||||
48 | Identifying appropriate study designs for measuring sustainability | .04 | ||||||
85 | Considering the role of self-reported data in assessing sustainability outcomes | .09 | ||||||
68 | Identifying valid data sources for assessing sustainability | .10 | ||||||
81 | Deciding how long to follow-up on a newly implemented program to determine whether it has been sustained | .10 | ||||||
27 | Developing and validating fidelity measures for assessing adherence of a program to an evidence-based model | .11 | ||||||
89 | Constructing reliable and validated tools to measure core sustainability constructs | .13 | ||||||
61 | Having an operational definition of sustainability, with measurable criteria | .14 | ||||||
44 | Determining “what” should measured as an outcome (patient level outcomes? fidelity? program activities? capacity?) | .26 | ||||||
14 | The ability to move rapidly and use innovative research methods to learn from emerging opportunities and examples | .29 | ||||||
Count: | 13 | Std. Dev.: | 0.09 | Minimum | 0.00 | Average: | .10 | |
Variance: | 0.01 | Maximum | 0.29 | Median: | 0.10 | |||
Cluster 9: research stage: design and analysis | ||||||||
72 | Discussing how systems science methods such as modeling and network analysis can be used to study important sustainability questions | .03 | ||||||
59 | Increasing use of non-experimental study designs in sustainability research | .08 | ||||||
43 | Using a participatory approach to research that values the perspective of developers of original idea as well as the target group | .14 | ||||||
35 | Clearly defining research hypotheses and goals linked to real-world application outcomes | .19 | ||||||
87 | Identifying the indicators of sustained use so that we will know it when we see it | .25 | ||||||
29 | Developing ways researchers can better integrate their documentation needs into the agency, so that it creates a smaller burden on those who deliver care (e.g., integrated data collection with electronic records) | .39 | ||||||
90 | Conducting observational research of implemented programs to identify barriers/facilitators to sustainability | .42 | ||||||
Count: | 7 | Std. Dev.: | 0.14 | Minimum | 0.03 | Average: | .21 | |
Variance: | 0.02 | Maximum | 0.42 | Median: | 0.19 | |||
Cluster 10: research stage: frameworks | ||||||||
11 | Discussing whether to study sustainability separately from implementation (e.g., How are the two related? not related?) | .28 | ||||||
10 | Testing of theories/frameworks for sustainability | .40 | ||||||
31 | Creating greater distinction between predictors of sustainability (e.g., organizational capacity) and sustainability outcomes (sustained programming) | .42 | ||||||
26 | Studying a greater variety of sustained activities, including interventions, programs, and policies | .43 | ||||||
16 | Assessing the dynamic processes underlying sustained use | .50 | ||||||
57 | Developing case studies to identify key characteristics of those that do sustain vs. do not | .63 | ||||||
Count: | 6 | Std. Dev.: | 0.11 | Minimum | 0.28 | Average: | .44 | |
Variance: | 0.01 | Maximum | 0.63 | Median: | 0.42 | |||
Cluster 11: research stage: definitions | ||||||||
24 | Conceptualizing and defining “sustainability”, its sub-dimensions, and related concepts (e.g., fidelity, routinization, institutionalization) | .08 | ||||||
79 | Defining sustainability outcomes | .10 | ||||||
73 | Creating a visual depiction of a sustainability model | .16 | ||||||
32 | Identifying dimensions and degrees of sustained use | .16 | ||||||
80 | Clarifying terminology (e.g., assimilation, institutionalization, continued use) | .17 | ||||||
71 | Developing a formal conceptual model that links dissemination, implementation, and sustainability | .17 | ||||||
40 | Balancing the use of pertinent theory, evidence, and experience | .29 | ||||||
20 | Determining whether there is a standard level of initial “success” necessary before a project or organization or community is “eligible” to be considered for a “sustainability” evaluation | .32 | ||||||
Count: | 8 | Std. Dev.: | 0.08 | Minimum | 0.08 | Average: | .18 | |
Variance: | 0.01 | Maximum | 0.32 | Median: | 0.17 |
Domain one: research agenda |
Defining the key attributes of orgs and systems that successfully sustain effective practice (e.g., ongoing leadership attention, ongoing measurement, systematic hardwiring of effective innovation, etc.)a |
Assessing the dynamic processes underlying sustained useb |
Testing of theories/frameworks for sustainability |
Identifying common and independent factors that drive adoption vs. initial implementation vs. long-term use |
Conducting ROI (return on investment) studies to make it clear to stakeholders and funders how much is actually gained when effective programs are sustained |
Understanding how to sustain programs/policies in low resource settings |
Identifying key or core program sustainability components |
Determining the return-on-investment of sustainability given a variety of likely parameters (e.g., number of individuals reached; number of infections averted) |
Identifying the key contextual factors (e.g., organizational characteristics) associated with variations in sustainability |
Domain two: advancing methodology for sustainability research |
Constructing reliable and validated tools to measure core sustainability constructs |
Domain 1: a substantive research agenda on sustainability
Improve clarity on the concepts and terminology used in sustainability research
Identify the value of sustainability
Test the effect of interventions properties and adaptation on sustainability
Test theories, frameworks, and models for their ability to explain and predict sustainability
Identify contextual factors that affect sustainability and understand explanatory mechanisms
Domain 2: advancing methods for sustainability research
Advance measurement
-
Determinants of sustainability, in reference to correlates and predictors of sustainability (organizational, contextual, and strategies); and
-
Outcomes of sustainability or sustainability outcomes in reference to subsequent impact (healthcare improvement or public health outcomes) of sustained intervention use.