Sustained effects of the INFORM cluster randomized trial: an observational post-intervention study

The design and evaluation of health services interventions is key to improving quality of healthcare and the patient experience. While numerous studies have examined the efficacy and effectiveness of health services interventions, much less attention has been paid to the sustainability of outcomes [1‐3]. Failure to sustain intervention outcomes of effective interventions significantly limits the potential benefits of intervention investment. When researchers withdraw intervention supports, intervention activities and improved outcomes of successful interventions regularly decrease [4, 5], highlighting the need for post-intervention studies on intervention sustainability. Sustainability becomes increasingly challenging with increasing intervention complexity [6‐8]—i.e., with an increasing number of intervention components interacting in complex ways, requiring multiple staff, often affecting multiple outcomes [9, 10]. According to a 2020 systematic review [11], little research has been published to date on the sustainability of complex interventions. This study responds to calls for research that examines the sustainability stage of successful interventions [5, 12] and contributes important knowledge on modifiable factors associated with sustainability of evidence-based interventions in health care settings [2].

Table 2 includes facility, care unit, and care aide characteristics by study arm and time of data collection. Based on our descriptive statistics, total staffing hours per resident day, the proportion of care aide hours among total care staffing hours, and the proportion of care aides whose first language was not English seemed to differ between study arms (at all three points in time) and changed over time within each study arm. Care aide age and sex did not differ substantially between study arms and were stable over time. The Manitoba sample did not differ substantially from our Alberta and British Columbia samples.

Our adjusted analyses (Fig. 2) demonstrate that at T3 (2.5 years after the intervention delivery had ended), care units in both assisted feedback groups had sustained the T2 (post-intervention) gains in care aide involvement in formal communications about resident care, indicating sustainability. While there seems to be a T2-T3 downward trend in the enhanced assisted feedback arm and a T2-T3 upward trend in the basic assisted feedback arm, neither of these group-specific trends are statistically significant. Notably, involvement of care aides in formal communications about resident care did not increase during the intervention period in the simple feedback group. However, it increased significantly at T3, rising to the levels seen in the two assisted feedback arms.

Our sensitivity analyses (Fig. 3) illustrate that in the Manitoba sample formal interactions scores followed a consistent downward trend that differs substantially from the trends seen in the three INFORM study arms.

As per Table 3, evaluation (feedback of performance data) at T3 was associated statistically significantly with higher involvement of care aides in formal communications about resident care at T3 in both models. Higher leadership scores (i.e., care aide ratings of the leadership of the persons they report to—primarily nurses) were not associated with care aide involvement in formal communications about resident care at T3. A more supportive work culture was associated with higher involvement of care aides in formal communications about resident care at T3 in model 1 (that included the simple feedback sample) but not in model 2. However, in model 2, better fidelity enactment at T2 predicted higher T3 involvement of care aides in formal communications about resident care. As noted, none of our interaction terms (fidelity by study arm and each of leadership, culture, evaluation by study arm) were statistically significant (p values consistently > 0.2) nor did they improve model fit and we therefore did not include any interaction terms in our final model. None of the other covariates were associated with care aide involvement in formal team communications about resident care at T3.

This study examined sustained effects of the INFORM intervention as well as the association of fidelity enactment and key variables of organizational context with sustained involvement of care aides in formal communications about resident care (formal interactions). Our findings suggest sustained benefits of the INFORM intervention 2.5 years after the research team had finalized the delivery of the intervention—signified by the lack of a statistically significant difference in formal interactions scores between T2 and T3 in both assisted feedback groups. Like at T2, formal interactions scores in the two assisted feedback groups did not differ at T3. Notably, formal interaction scores did not increase in the simple feedback arm during the intervention (signified by the lack of a statistically significant difference in formal interactions scores between T1 and T2), but rose to the levels of the assisted feedback arms at T3 after the end of the INFORM study. Higher fidelity enactment during the intervention period, more supportive work culture, and more feedback activities (evaluation) with a care unit were associated with higher formal interaction scores at T3, regardless of the intervention intensity (study arm allocation).

Our study is one of the few that has examined the degree to which the effects of a complex intervention can be sustained in a complex care setting, such as nursing homes. TREC’s comprehensive database of longitudinal data is one of the main reasons why we could conduct this work and provide these insights—highlighting the critical need for more longitudinal studies of intervention effectiveness and sustainability that cover time periods of 5 years or more.

Our finding that higher fidelity enactment during the intervention period was associated with sustained intervention benefits is noteworthy. To the best of our knowledge, this is the first study to examine this association. As noted, fidelity enactment refers to the extent to which participants adhere to and carry out core intervention components as intended during the intervention period [43]. Core components of the INFORM intervention include (1) managers and care teams setting learning and performance goals, (2) managers engaging with care teams to work towards goal achievement, and (3) measuring success towards these goals. While fidelity enactment is critical for intervention success, its role for intervention sustainability is poorly understood as reported in contemporary empirical research. The dynamic sustainability framework (DSF) suggests that intervention approaches that emphasize adaptation (adjusting and refining the intervention to fit the local context) sustain an intervention more successfully than approaches that emphasize fidelity to an original protocol [1]. Future research is needed to better understand how and why care teams that adhere more closely to intervention protocols are more successful in sustaining intervention benefits and what role adaptation plays in these processes. Additionally, related research is needed on the extent to which successful care teams are able to adapt interventions to their needs, under what circumstances these adaptations violate intervention fidelity, and attain outcomes that are superior to those originally reported in clinical trials, and how different ways of adapting the intervention affect sustainability [21, 60].

Our finding that evaluation (feedback of performance data to care teams) was associated with sustained intervention success is in line with previous studies [3, 5, 32]. Evaluation is part of what the DSF labels as information systems [1]. Evaluation activities—i.e., consistent feedback on a team’s performance and discussion of possible solutions—facilitate rapid learning and real-time problem-solving among care teams and integrate team members in the generation, rather than just the application of knowledge. The DSF assumes that an organization that develops this kind of a culture will be more successful in sustaining an intervention by improving the fit of an intervention over time.

Our finding that leadership was not associated with sustainability is not consistent with available reviews [3, 5, 32], nor with the DSF (where this construct is labeled supervision) [1]. For example, Hailemariam et al. [3] found 12 studies suggesting an association of leadership with intervention sustainability. However, from these studies, it is largely unclear what type of leadership style was associated with sustainability. For example, in their systematic review, Hailemariam et al. [3] found that “organizational leadership” was consistently associated with sustainment of evidence-based practices, but the authors provide no further details on the leadership styles measured. The ACT leadership scale [45, 46] that we used in our survey measures transformational leadership, which is only one of many leadership styles that may have been measured by other studies. More research is needed to determine whether this leadership style is associated with sustainability of study outcomes. Furthermore, the ACT leadership scale asks care aides to rate leadership characteristics of the person (or group of persons) to whom they report most of the time. In nursing home settings, these individuals are generally registered nurses or licensed practical nurses [53]. Therefore, the leadership scale we used measures care aides’ rating of nurses’ leadership, not their rating of facility-level or unit-level formal leaders/supervisors (organizational leadership). Care aides rate, for example, whether nurses regularly ask them for feedback even if it is difficult to hear, actively mentor or coach them, or focus on success rather than failure. This rating of nurses’ leadership may not be reflective of the managers’ leadership—and managers, not nurses, are the ones enabling, encouraging, and/or requiring care aides to participate in formal team meetings about resident care. Furthermore, the leadership skills listed above are not necessarily specific to enabling and encouraging a care aide to participate in formal meetings about resident care. A specific leadership skill would be, for example, a manager’s ability to recognize and address the care aide’s specific needs (such as lack of confidence to participate in such meetings, inability to envision what to say and how to best contribute, concern to neglect residents while participating, etc.).

Our finding that culture was associated with sustainability is in line with the DSF [1], but is not consistent with available reviews [3, 5, 32]. Culture has been identified as a factor associated with sustainability in only a small number of studies [3, 5, 32]. In their review, Stirman et al. [5] highlight that the small number of studies empirically supporting culture to be a key factor for intervention sustainability stands in contrast with the extent to which culture is discussed as an important factor in the implementation and sustainability literature. This most likely has to do with how culture is operationalized and measured. For instance, qualitative studies suggest that local unit-level processes and interactions that may reflect a particular culture or leadership style are more likely to support sustainability than the overall culture of an organization (which is commonly measured in quantitative sustainability studies) [5]. The ACT culture scale [45, 46] we used in this study asks care aides about particular interactions and processes (e.g., whether they are members of a supportive work group or whether their immediate team works to provide what residents need).

We found that the factors influencing sustainability were independent of whether the intervention participants were exposed to the more versus the less intense study arm. We even saw a slightly lower success and sustainability in our most intense study arm, compared to the less intense study arm (although not statistically significant). It is possible that our enhanced feedback was too onerous for participants (meeting face-to-face for half-day workshops, versus 1.5-h video conferences in the basic assisted arm), limiting short- and long-term benefits. An alternative explanation is that the minimum “dose” of feedback required to facilitate sustainability was achieved with the less intense study arm. This finding contrasts with clinical studies reporting that higher intervention intensity supports better intervention success and sustainability [54‐56]. Shelton et al. [32] highlight the important research gap related to whether sustainability depends on the type and intensity of the intervention. Little work is available on how to determine the optimal intensity of an intervention, and a particular gap exists regarding the influence of intervention intensity on the sustainability of highly complex health services interventions. Our study contributes in important ways to addressing this knowledge gap. However, more research is needed on whether the factors we identified applying a complex intervention in nursing homes are consistent or different in other healthcare settings or when applying different types of interventions.

We found that homes in the simple feedback arm saw a substantial increase in their formal interaction scores in the post-intervention period. In contrast to nursing homes in the assisted feedback arms, simple feedback homes did not receive major intervention components, such as goal setting and support workshops, support by the study team during intervention workshops to set goals and complete an action plan, peer-to-peer support by teams from other nursing homes, etc. However, we systematically fed back INFORM findings to facilities and decision makers as part of our routine feedback activities after the T2 data collections (i.e., after the INFORM intervention period). It is also possible that TREC’s relationships with key stakeholders over 10–15 years (regional and provincial decision makers, nursing home owner-operators and care teams), their systematic involvement with INFORM, and our longitudinal work with nursing homes in our cohort may have played an important role. In fact, many of the key stakeholders in our study know each other and actively collaborate on a regular basis. For example, health region decision makes hold regular meetings with nursing home administrators in their region, giving decision makers the opportunity to interact with facilities and allowing administrators to interact and exchange information across nursing homes. Often, nursing home administrators or directors of care oversee more than one facility. Administrators of facilities operated by the same owner regularly meet and collaborate on improvement activities. While this is not a definite explanation of the increased informal interaction scores we found in non-intervention facilities after the end of our intervention, it is a plausible explanation. However, the exact reasons of this surprising finding are an interesting and hard to understand question. Pending resources, we are hoping to conduct focus groups with decision makers and key stakeholders in our facilities to further explore this question.

Our study findings suggest that the benefits in INFORM’s primary outcome—care aide involvement in formal interactions about resident care—was sustained in the two more intense study arms, 2.5 years after intervention delivery had ended. Care teams in our least intense study arm started to increase care aide involvement in formal interactions about resident care after our trial had ended, raising the possibility of informal spread. Higher fidelity enactment during the intervention period, more supportive work culture, and more feedback activities (evaluation) within a care unit were associated with sustained care aide involvement in formal communications about resident care, regardless of the intervention intensity (study arm allocation). This study supports some of the assumptions posed by the DSF, but also raises questions—especially related to the role of intervention fidelity versus adaptation and their complex interplay.