Contributions to the literature
-
Research gaps exist in how to optimize sustainability of implementation of evidence-based interventions.
-
We found that funded implementation researchers vary in their definitions of sustainability and their use of sustainability frameworks.
-
We identified strategies for funders to consider to advance the sustainability research agenda in implementation science.
Background
Methods
Phase 1: Document review
Data source
Phase 2: Qualitative content analysis
Integration of phase 1 and phase 2 results
Results
Phase 1: Document review
Description of funded implementation projects
Implementation | Implementation projects | |
---|---|---|
(n) | (%) | |
Chronic care hospital | 3 | 3.9 |
Clinic | 24 | 31.6 |
Community | 24 | 31.6 |
Government | 2 | 2.6 |
Home | 2 | 2.6 |
Acute care hospital | 12 | 15.8 |
Workplace | 1 | 1.3 |
School | 1 | 1.3 |
Online social network | 1 | 1.3 |
Not reported | 6 | 7.9 |
Total | 76 | 100 |
Conceptualization of sustainability in implementation project profiles
Reason for referring to sustainability | Frequency (n) | Percentage (%) |
---|---|---|
Intent to examine factors associated with sustainability during or after implementation | 13 | 21.6 |
Intent to evaluate the impact of a strategy on sustainability of programs or practices | 11 | 18.3 |
Intent to measure sustainability (e.g., sustained delivery of programs or implementation strategies, sustained health outcomes, or sustained practices) | 10 | 16.6 |
Public health relevance statement or hypothesis of how work will impact sustainability of EBPs | 7 | 11.6 |
Describing need for better implementation supports/strategy development | 5 | 8.3 |
Reference to evidence that intervention has produced sustained health or behavioral outcomes | 4 | 6.6 |
Intent to disseminate of R01 results to promote sustainability of EBPs in practice | 3 | 5.0 |
Intent to evaluate comparative sustainability of each strategy | 2 | 3.3 |
Reference to stakeholder engagement and impact on sustainability | 2 | 3.3 |
Intent to sustain implementation efforts with capacity building and leadership development | 1 | 1.6 |
Intent to estimate sustainability costs and benefits for an intervention | 1 | 1.6 |
Intent to determine elements of programs that will enhance sustainability | 1 | 1.6 |
Total | 60 | 100 |
Phase 2: Qualitative content analysis
Interviewee demographics
Interviewee ID | Unit of analysis | Researcher setting |
---|---|---|
62 | Organization | Academic research institute |
22 | Individual | Academic research institute |
11 | Organization | Academic research institute |
10 | Organization | Academic research institute |
12 | Organization | Non-profit research institute |
24 | Individual | Academic research institute |
026 | Organization | Academic research institute |
044 | Community | Academic research institute |
050 | Community | Academic research institute |
09 | Organization | Academic research institute |
75 | System | Non-profit research institute |
How do R01 implementation researchers define sustainability?
Interviewee ID | After a defined period of time | A program, clinical intervention, and/or implementation strategies continue to be delivered | Individual behavior change (i.e., clinician, patient) is maintained | The program and individual behavior change may evolve or adapt | Continuing to produce benefits for individuals/systems |
---|---|---|---|---|---|
62 | X | X | |||
22 | X | X | X | X | |
11 | X | ||||
10 | X | ||||
12 | X | X | X | ||
24 | X | X | |||
026 | X | ||||
044 | X | X | |||
050 | X | ||||
09 | X | X | |||
75 | X | X | X | X |
“ … continued and ongoing use of implementation strategies that work to improve adoption of guidelines”—22 | |
“ … ability of the organization in which the evidence-based program that has been implemented to maintain its offering at a high or desired level of fidelity”—10 | |
“Having the intervention be able to sort of stand-alone if you will, either online with some sort of a company or some sort of a system that would allow it to remain online or publication of sort of the manual”—11 |
How do R01 implementation researchers use sustainability frameworks?
-
Consolidated Framework for Implementation Research (CFIR) (n = 3)
-
Scheirer and Dearing framework (n = 2)
-
Framework for dissemination by Peter Mendel and colleagues (n = 1)
-
EPIS framework (n = 2)
-
Dynamic Sustainability Framework (DSF) (n = 2)
-
RE-AIM (n = 2)
-
Plan, Do, Study, Act (PDSA) (n = 1)
-
REP Framework (n = 1)
How do R01 implementation researchers plan for sustainability?
Category | Description |
---|---|
System-level stakeholder buy-in | Advocate for the development of policies that encourage the use of the intervention/program/practice both before and during implementation. Network with professional organizations that could either promote the intervention/program/practice or incorporate the intervention/program/practice as part of their larger professional curriculum. Organize ongoing stakeholder consultations to monitor changes in the context that could require adaptations or to advocate for more funding |
Organizational/community-level stakeholder buy-in | Encourage leaders to buy-in and talk about the intervention/program/practice regularly in staff meetings. Consult with individuals implementing the intervention to find out if the proposed intervention/program/practice will be sustainable after research funding is removed (e.g., site visits, formative evaluations) Co-develop implementation strategies with stakeholders |
Organizational incentives | Build in program indicators into performance reviews, organization leaders also build other incentives for employees to use the intervention/program/practice and they document their progress |
Staff-turn over packages | Generate new staff orientation and training packets so that new hires learn about the intervention/program/practice as soon as they are on boarded |
Capacity building at all levels (organization, community, system) | Link implementation teams to a resource package or other organizations that provide ongoing training. Provide implementers with a free social media tool/learning collaborative where they can have fast access to resources and connect with other implementers |
Organization-level continuous quality improvement | Host organization staff that are trained to use PDSA cycles to monitor the fit of the intervention/program/practice, anticipate challenges, and adapt where needed over time |
Intervention monetization | Publish a training handbook that can be purchased at a popular book store. Sell online resources to prospective implementers (e.g., one-time fee for unique log-in) |
Guidance from intervention developers | Provide guidance on what the core and what the kind of adaptable periphery of the intervention/program/practice, so that when changes need to be made implementers have a sense of what key elements need to be sustained |
Programmatic approach to research | Ensure each implementation study has elements of sustainability (e.g., one arm gets early sustainability planning) that can be followed up on in subsequent studies and the funding for the intervention/program/practice continues |
What influences R01 implementation researchers’ decision to plan for sustainability?
sustainability is something that we researchers rarely think much about, I mean we write grants, we say at the end, and at the end of this study we will publish in leading journals. The assumption there is that somebody will do something then. And it may be a fallacious assumption, it frequently is —050
Include the requirement to plan for sustainability in program announcements: This was described as similar to the announcements from the Centre of Disease Control and Prevention. Current NIH announcements include language about studying sustainability but not about planning. | |
Request for proposals (RFP) could include sustainability requirements: Applicants should be required to list sustainability outcome measures (e.g., sustainability 6 months post implementation). | |
Offer a supplement OR non-competitive renewal specifically for sustainability: This could include parameters such as the requirement to use funds only for evaluation or testing of lean, internally resourced strategies for sustainability. | |
Deliver funding in phases so that there is funding specifically allocated for sustainability efforts: This ensures that there is money for sustainability and full funding is not received until it is clear that the program has the potential capacity to be sustained. | |
Include the requirement to engage stakeholders and demonstrate the need for this program/intervention in the relevant setting: Applicants must describe to what extent the work is a priority for the context and to what extent the relevant stakeholders have been involved in the planning and evaluation process. |