Contributions to the literature
-
Reviews report relatively few healthcare interventions that are sustained beyond the initial implementation phase or scaled to different populations or settings.
-
Acceptability, fidelity and feasibility may influence scalability and sustainability of a healthcare intervention.
-
We have developed a testable conceptual framework that can be used to prospectively and iteratively guide the implementability of healthcare interventions.
-
Prospective identification of factors that influence scalability and sustainability of a healthcare intervention is critical to avoid or reduce research waste.
Background
Methods
Step 1: Overview of reviews
Search strategy
Screening citations
Inclusion criteria | Exclusion criteria |
---|---|
Systematic review | Not available in English |
Healthcare intervention | Focus on cost-effectiveness acceptability curves |
Inclusion of acceptability, fidelity, feasibility, scalability and/or sustainability in the title | Reviews on drug development where the focus was only on safety |
Conference proceedings, commentaries, study protocols and editorials |
Full-text review
Assessment of quality
Step 2: Development of the preliminary framework
Step 3: Modified Nominal Group Technique
Results
Step 1: Overview of reviews
Characteristics of the included studies
Definition and measurement (question a) and frameworks (question b) for the five concepts
Acceptability
Frameworks (used in N = 6 reviews) | N (%) |
---|---|
• Asiimwe et al. (2012). Conceptual framework for exploring acceptance and use of mRDT Acceptance and use maybe be influenced by user attributes, the diagnostic tool and the health system. Attributes include learnability, willingness, suitability, satisfaction, efficacy and effectiveness. | 1 (17) |
• Sekhon et al. (2017) [67]. Theoretical Framework of Acceptability Acceptability is a multi-faceted construct that reflects the extent to which people delivering or receiving a healthcare intervention consider it to be appropriate. The Theoretical Framework of Acceptability includes affective attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs, and self-efficacy | 4 (66) |
• Rosenstock et al. (1966). Health beliefs model An individual’s course of action depends on their perceptions of benefits and barriers including perceived susceptibility, perceived severity, perceived benefits, perceived barriers, cue to action and self-efficacy. | 1 (17) |
Commonly measured components | |
• Attrition/dropout rates | 44 (33) |
• Perception of users including satisfaction, experience, views (receivers of interventions and those delivering) | 40 (30) |
• Adherence/compliance | 17 (13) |
• Adverse events/side-effects | 4 (3) |
• Recruitment | 5 (4) |
• Other (effectiveness, cost-effectiveness, efficacy, future intentions, likelihood to recommend to others or repeat intervention) | 4 (3) |
Fidelity
Frameworks (used in N = 34 reviews) | N (%) |
---|---|
• Dane & Schneider (1998) Fidelity of intervention should include a measure of adherence to the program, dose, quality of program delivery, participant responsiveness and program differentiation. | 7 (21) |
• Borrelli et al., (2005). National Institute of Health Behavioral Change Consortium Treatment fidelity should be assessed using 5 categories including design, training, delivery, receipt and enactment. | 19 (56) |
• Carroll et al., (2007) Implementation fidelity should include the measurement of adherence (content, frequency, duration and coverage) and moderators (intervention complexity, facilitation strategies, quality of delivery and participant responsiveness). | 2 (6) |
• Steckler & Linnan (2002) Public health interventions should be measured and evaluated against seven different components including context, reach, dose delivered, dose received, fidelity, implementation and recruitment. | 2 (6) |
• Moncher & Prinz (1991) Fidelity requires a clear definition of the treatment, training in delivery of the protocol, treatment manuals, supervision and adherence to the treatment protocol through treatment verification. | 2 (6) |
• Sidani & Sechrest (1999) Fidelity of implementation should include conceptualisation of the problem, operationalisation of the theory and specification of mediating processes and outcome variables. | 1 (3) |
• Perepletchikova, Treat & Kazdin (2007). Implementation of Treatment Integrity Procedures Scale (ITIPS) Evaluation of treatment integrity in psychotherapy research should include four domains: establishing, assessing, evaluating and reporting fidelity along with therapist treatment adherence and competence | 1 (3) |
Commonly measured components | |
• Dosage | 31 (76) |
• Adherence/compliance | 31 (76) |
• Quality | 9 (22) |
• Responsiveness | 31 (76) |
• Training | 20 (49) |
• Other (program differentiation, supervision, treatment manual, environmental design, therapist qualifications, theory) | 11 (27) |
Feasibility
Frameworks (used in N = 5 reviews) | N (%) |
---|---|
• Bowen et al., (2009) Feasibility studies should address eight general areas including acceptability, demand, implementation, practicality, adaptation, integration, expansion and limited efficacy testing. | 3 (60) |
• Bird et al., (2014). The Structured Assessment of Feasibility (SAFE) Feasibility of complex interventions within mental health services are influenced by 16 factors such as staff training, intervention complexity, time, supervision and adverse events | 1 (20) |
• Joanna Briggs Institute Measure of feasibility, appropriateness, meaningfulness and effectiveness (FAME) | 1 (20) |
Commonly measured components | |
• Dropouts/attrition | 9 (14) |
• Adherence/compliance | 22 (34) |
• Completion | 11 (17) |
• Recruitment | 13 (20) |
• Cost-benefit/economic feasibility | 7 (11) |
• Adverse events/side-effects | 8 (12) |
• Acceptability | 10 (15) |
• Perceptions of users (satisfaction, ease of use, perceived enjoyment) | 15 (23) |
• Other (including but not limited to context-specific and operational issues, intervention practicality/acceptability/integrity, training, equipment, time, knowledge, contraindications) | 25 (38) |
Scalability
Frameworks (used in N = 4 reviews) | N (%) |
---|---|
• WHO/ExpandNet. Scaling-up framework Scaling up consists of five elements: the innovation, resource team, user organisation, broader environment and the scaling strategy | 1 (25) |
• Milat et al., (2020) [82]. Intervention scalability assessment tool (ISAT) Assessment of scalability includes five domains: the problem, the intervention, strategic/political context, evidence of effectiveness, intervention costs and benefits, fidelity and adaptation, reach and acceptability, delivery setting and workforce, implementation infrastructure and sustainability. | 1 (25) |
• Bradley et al., (2012) [83]. The AIDED model Scalability consists of five interrelated components: the landscape, innovation to fit user receptivity, support, engagement of user groups and effort for spreading innovation. | 1 (25) |
• Greenhalgh et al., (2017) [84]. NASSS Framework | 1 (25) |
Commonly measured components | |
• Adaptation | 2 (18) |
• Resources | 4 (36) |
• Partnerships/collaborations | 1 (9) |
• Organisation/community/sociocultural factors | 5 (45) |
• Cost-benefit/economic feasibility | 2 (18) |
Sustainability
Frameworks (used in N = 9 reviews) | N (%) |
---|---|
• Scheirer (2005) Sustainability should be measured across three levels for sustainability: individual, as continuing to deliver services that are beneficial; organisation, as maintaining the programme and community, as maintaining capacity. | 1 (11) |
• Cekan and Zivetz (2016) Sustainability should measure if the program has incorporated a theory of change, presence of explicit sustainability goals in a monitoring and evaluation plan, methods for identifying unexpected outcomes; funding, capacity development and collaboration | 1 (11) |
• Moore et al., (2017) [95] Five key sustainability constructs describe individual and organisational capacity based on time, behaviour, adaptation of the program/intervention, ongoing benefits and ongoing delivery of the program/intervention | 2 (22) |
• Shediac-Rizkallah & Bone (1998) [96] Sustainability can be measured through maintenance of health benefits, integration of the program within an organisation and community capacity building | 2 (22) |
• Greenhalgh et al., (2017) [84] NASSS Framework The framework consists of 7 domains (condition, technology, value proposition, adopter system, organisation, wider system and adaptation over time) and numerous subdomains. | 1 (11) |
• McLeroy et al., (1998) Health related behaviour is influenced by individual factors, interpersonal factors, organisation factors, community factors and public policy factors. | 1 (11) |
• Lennox, Maher & Reed (2018). Consolidated framework for sustainability constructs in healthcare. Includes 40 constructs across six domains: the organisational setting, negotiating initiative processes, resources, the external environment, the initiative design and delivery and the people involved. | 1 (11) |
Commonly measured components | |
• Time (endurance of intervention/program beyond a period of time) | 6 (24) |
• Training | 3 (12) |
• Resources | 2 (8) |
• Partnerships/collaborations | 3 (12) |
• Organisational/community factors | 9 (36) |
Step 2: Modified Nominal Group Technique
Study title Does prospective acceptability of an intervention influence refusal to participate in a randomised controlled trial? Background Blepharospasm and hemifacial spasm are currently managed by regular Botox injections, given approximately every 2 months (appointments scheduled by the physician). But the timeline to return of symptoms following treatments is variable. It is possible that patient-initiated appointments (i.e. when symptoms flare up) could result in a more efficient and effective service. This possibility was tested in a randomised trial. A qualitative investigation of acceptability was conducted as a sub-study. Aims To apply the Theoretical Framework of Acceptability (TFA; consisting of 7 component constructs) to explore: (1) patient-reported reasons for declining to participate in the trial; and (2) associations between decliners’ perceptions of acceptability and their non-participation. Method Eligible patients (n = 242) were approached to participate in the trial. Phase 1: decliners provided a brief reason for refusal. We analysed the reasons descriptively and reviewed them against the TFA constructs. Phase 2: We invited consecutive decliners to participate in short semi-structured interviews, to explore their reasons for refusal in more depth. Interviews were transcribed and analysed, with the TFA as a coding framework. Results Eighty-seven (36%) eligible patients refused trial participation; all provided a reason. From interviews with 15 decliners, four key beliefs about acceptability were identified: happy with standard care, anticipated burden of the patient-initiated service, lack of confidence in ability to engage with new service and uncertainties about the effectiveness of new service. Two themes reflected non-TFA factors: trial participation was a low priority and the burden of completing trial documentation. Conclusion Reasons for refusing trial participation were often, but not always, associated with intervention acceptability. Relationship to implementability 1. Three factors could be improved to enhance acceptability of the new service: reducing burden, enhancing patient support to increase confidence (making sure they are able to make contact by phone), and ensuring that the new service is perceived to be workable (increasing available appointment spaces). 2. Unless these factors can be satisfactorily addressed it appears that the new service would not be implementable. Reference Sekhon, M., Cartwright, M., Lawes-Wickwar, S., et al. (2021). Does prospective acceptability of an intervention influence refusal to participate in a randomised controlled trial? An interview study. Contemporary Clinical Trials Communications, 21, 100698. |
Study title Fidelity of an allied health prehabilitation service for haematologic patients receiving high dose chemotherapy in a large cancer centre. Background Cancer prehabilitation can reduce post-treatment complications, enhance functional capacity, and empower patients to withstand treatment stressors. As part of a larger study, we evaluated the fidelity of a multidisciplinary allied health (exercise, nutrition, and psychology) prehabilitation clinical service as part of routine care in haematologic cancer patients receiving intensive conditioning chemotherapy prior to an autologous stem cell transplant (AuSCT). Method We retrospectively analysed data routinely collected from patients referred between March 2019 and March 2020. All patients considered for AuSCT at a tertiary specialist cancer centre were eligible to participate. The prehabilitation intervention included individualised exercise prescription and input from other allied health teams. Fidelity of the prescribed exercise program was assessed along the pathway from referral to the AuSCT service through to receipt by patients. Results 183 patients were referred to the AuSCT service, 133 (73%) were referred into the prehabilitation service, 128 (96%) were eligible and 116 (91%) participated. Fidelity of exercise prescription was moderate with 72% of patients receiving the intended aerobic and resistance exercise intervention. Hence, 83 (65%) of the original 128 eligible patients actually received the exercise component of the intervention. Conclusion Although the prehabilitation service was well adopted by clinicians, there was some room for improvement in terms of the objective of providing all eligible patients with exercise prehabilitation support. Relationship to implementability 1. Only two-thirds of eligible patients received the intervention as intended. 2. Although the intervention appears to be implementable, further support is needed to increase consistency and equity of delivery. Reference Crowe, J., Francis, J. J., Edbrooke, L., et al. Impact of an allied health prehabilitation service for haematologic patients receiving high dose chemotherapy in a large cancer centre. Under review. |
Study title Feasibility of conducting family meetings for hospitalised palliative care patients. Background A family meeting is a clinical tool for healthcare providers to facilitate communication with patients with advanced disease and their family caregivers. Despite family meetings being advocated as standard practice, minimal evidence existed regarding the balance between costs and benefits. The economic feasibility (healthcare utilisation) of providing a structured family meeting for hospitalised palliative care patients was evaluated as part of a larger cluster randomised trial. Method A pragmatic cluster randomised control trial was conducted across three major Australian hospitals. Patients admitted or referred to specialise palliative care units, and their primary family caregiver, were invited to participate. The intervention consisted of a single structured family meeting tailored according to the individual needs of the participant, family caregiver and treating team. The control group received usual care. Caregiver psychological distress, patient outcomes and healthcare utilisation data were compared between the two groups. Results A total of 297 dyads were recruited and randomised: control group (n = 153); intervention group (n = 144). The intervention group demonstrated significantly lower psychological distress (Diff: − 1.68, p < 0.01) and higher preparedness (Diff: 3.48, p = 0.001) at Time 2. No differences were identified for quality of end-of-life care or health resource utilisation. Conclusion Family meetings may assist in reducing family caregiver distress and preparing individuals for their caregiving role. The results also suggest that family meetings do not increase health service utilisation costs; however, this aspect of feasibility requires further examination. Relationship to implementability 1. Routinely conducting family meetings may not incur additional demands on health care utilisation. 2. Family meetings appear to be implementable but further investigation of other feasibility factors is required. Reference Hudson, P., Girgis, A., Thomas, K., et al. (2021). Do family meetings for hospitalised palliative care patients improve outcomes and reduce health care costs? A cluster randomised trial. Palliative Medicine, 35(1), 188–199. |
Study title Factors affecting sustainability of a quality improvement policy on medications while fasting for surgery. Background Several adverse events associated with patients missing medications while fasting for surgery led to a quality improvement project that aimed to simplify and standardise oral restriction terminology and medication administration instructions to reduce confusion and unwanted practice variations when patients had oral intake restrictions such as fasting for surgery. Method A companion qualitative study to this quality improvement program was conducted after the roll out of the intervention: a new policy about medications and restrictions in oral intake. Results Before the quality improvement intervention, there was confusion, lack of clarity and guidance, and lack of experience and confidence in managing medications when patients had oral restrictions. After the rollout, there was improved clarity and decision support; but problems included lack of awareness about the policy, particularly due to staff movement and turnover; and individual interpretation and acceptance of the policy. Sustainability of the project appears dependent on continuing the role of a project officer combined with educators. These roles also appear important for scaling up the program within one hospital and essential for implementability elements of scaling up, acceptability, and fidelity in other hospitals. Conclusion Elements needed for greater sustainability included strategies and resources to 1) educate staff; 2) minimise variation, and optimise fidelity, in interpreting information; and 3) deal with continuous staff changes. Relationship to implementability 1. Routinely conducting family meetings may not incur additional demands on health care utilisation. 2. The elements of implementability appeared to interact and cannot be viewed insolation. 3. Sustainability was heavily affected by staff changes and requires ongoing investment. Reference To, T‐P, Dunnachie, G, Brien, J‐a, Story, DA. Surgical nurses' perceptions and experiences of a medications and oral restrictions policy change: A focus group study. J Clin Nurs. 2019; 28: 3242– 3251. |
Scenario title Scalability of a digital health intervention Background A suite of new care models was designed and deployed at a hospital to manage the COVID-19 pandemic. One model supported patients in monitoring symptoms at home and advised patients about when and if they needed medical care and could present to the hospital. This virtual care model leverages technology to connect the patient with best evidence and provide targeted advice from their provider when needed. Outcomes included avoidance of emergency room and hospital overcrowding, lower cost to the patient, increase patient control and peace of mind. Many other conditions could be managed with a virtual care model and, after success of the COVID-19 model, the hospital would like to scale the model to better provide care for older adults with complex comorbidities. They decide to adapt the model to patients with chronic respiratory disease, who are one of the main sources of unnecessary Emergency Department admissions. | |
Barriers to scalability | Scenario |
Acceptability | Initial deployment used a homegrown database with a website link • Older adults often have lower digital literacy • Many older adults do not have daily access to a computer |
Fidelity | The intervention was deployed with urban patients, highly educated, and good internet connection • The deployment will be different with rural patients with poor internet |
Feasibility | Will require primary care integration. • Dispersed, independent GP clinics make it difficult to disseminate • There is no reimbursement model for GPs to look at panels of patients • Cannot currently exchange information between hospital EMR and many different primary care electronic medical records Captures oxygen saturation with a digital device • Emergency funding during pandemic not available for continuing program • Device does not work as well on patients with darker skin shades Enrolling patients relied on their coming to ED with COVID-19 symptoms • Identifying eligible patients will require new technology • All data required for eligibility requirements may not be digital or may not be sensitive/specific enough |
Conclusion Adapting a digital health intervention to a new population or setting can be like starting over again due to differences in IT infrastructure, digital literacy, and funding models Relationship to implementability Scalability of a digital intervention like this scenario encompasses acceptability, fidelity, and feasibility |
Step 3: Revising the framework
Discussion
Concept | During development and early evaluation | After initial evidence of effectiveness | Enabling factors to consider |
---|---|---|---|
Acceptability | ✓ | ✓ | Intervention information/knowledge, experience of delivering or receiving the intervention |
Fidelity | ✓ | ✓ | Acceptability, training, supervision, treatment manual |
Feasibility | ✓ | ✓ | Fidelity, training, resources (equipment, physical space, time), social/organisational/political support |
The factors above to be investigated iteratively with stakeholders in the inner setting The factors below to be investigated iteratively with a broader range of stakeholders (inner and outer settings) | |||
Scalability | ✕ | ✓ | Acceptability, fidelity, feasibility, training, resources (equipment, physical space, time), social/organisational/political support, partnerships and collaborations |
Sustainability | ✕ | ✓ | Acceptability, fidelity, feasibility, training, resources (equipment, physical space, time), social/organisational/political support, partnerships and collaborations |