TIME to reduce agitation in persons with dementia in nursing homes. A process evaluation of a complex intervention

The published protocol paper for the trial and the manual at the TIME website (English version) give a complete description of TIME, the model’s development, and its theoretical underpinnings [18, 21]. TIME is an evidence based, multicomponent, biopsychosocial intervention to be used by the nursing home staff and physicians. Principles from cognitive behavioural therapy (CBT) and person-centred care form the theoretical base for TIME [22, 23]. TIME is divided in three overlapping phases: a registration and assessment phase (Table 1); a guided reflection phase, including case conferences (Table 2); and an action and evaluation phase. The actions to take in this last phase are to be described as SMART. SMART stands for Specific, Measurable, Actual, Realistic and Time framed [30]. The evaluation of these actions is performed using the same clinical scales and observation forms as described in the first phase. Because the actions and treatment measures are supposed to be tailored to each resident, they will display great variations between patients. In this way, TIME serves as a guide for the staff to create actions and treatment measures that are customised to the resident. These three phases are in line with reviews describing the “state of the art” for management of NPS [9, 31]. A qualitative study exploring the staffs’ experiences with TIME showed that the model shifted the way of learning for the staff from a traditional to a shared, innovative and reflection-based learning at work using the case conferences. The staff experienced increased coping when approaching complex problems [32].

The published study protocol gives a detailed description of the control and intervention phases in the trial [18]. The implementation process started with an information meeting (1.5 h) with leaders (i.e. leading ward nurses, managers, and physicians) from the nursing home and the health and care authorities in the municipality to introduce TIME and the necessary steps for its implementation. Before randomisation, three nurses from each ward in both the intervention nursing homes (INH) and the control nursing homes (CNH) were offered a three-hour lecture about the trial and the assessments instruments to be used for the residents. After randomisation, a two-hour lecture covering dementia and NPS was given to the staff in both the INH and the CNH. For the staff in the CNH, this lecture represented the education-only intervention. For the remainder of the trial, the staff of the CNH continued with practice as usual. An additional training program with a three-hour lecture and role play about TIME was offered to the staff in the INH. The responsibility for the implementation of TIME was given to three nurses in each ward in the INH together with the leading ward nurse. These three nurses were given the title TIME administrators. The leading ward nurse selected the TIME-administrators for each ward, based on their interest in professional development, their legitimacy with the rest of the staff, and not having a part-time job. The TIME-administrators received three additional hours of training. A specialist nurse from the education and training team attended the TIME administrators’ first case conference on their first resident in their nursing home in a supervisory capacity. The education and training team in the TIME trial consisted of five specialist registered nurses in geriatrics or old age psychiatry and one physician with special competence in nursing home medicine. All members of the team were familiar with TIME and had used the model for some years in real-world clinical settings. In addition to the training sessions, all the staff members in the INH were given the TIME manual and they had access to a website with assessment sheets, an educational film and other educational material to support the intervention [21, 33].

The intervention with TIME can be regarded as a pragmatic trial due to relevant clinical inclusion criteria and the primary outcome measure, the low involvement of the researchers in the implementation process, the low use of other external resources, the rather modest time allocated for educational purposes and the moderate degree of flexibility for adaption of the intervention to the context of the nursing homes [34].

TIME can be regarded as a complex intervention because it is composed of multiple components with the potential for interactions between them and with a number of possible outcomes [35]. Few studies of complex interventions in nursing homes report from the implementation process and to what degree the interventions were adopted and sustained over time [36‐40]. A common conclusion in these reports is that management support and organisational factors are the main issues in the implementation process. The comprehensive review of the effect of staff training on staff outcomes by Spector et al. in 2016 concluded that many staff factors are difficult to change [40]. One important question for health leaders and policy makers is to what extent the intervention is flexible, easy to implement and can be adapted to the local context. Furthermore, a process evaluation is important to guide projects which aim to replicate the trial. And finally, it is necessary to clarify possible causal mechanisms to better inform and prepare further dissemination and implementation of the intervention on a larger scale [20, 41].

The Medical Research Council (MRC) defines an overarching framework for the development and evaluation of complex interventions emphasizing the value of combining both quantitative and qualitative data in the process [42]. In the MRC guidance the RE-AIM framework is described as a useful framework for assessing the overall implementation impact of interventions. RE-AIM is an acronym for five evaluation dimensions: Reach (the number, proportion and characteristics of the target population that participated), Effectiveness (the ability to change desired outcomes), Adoption (the number or proportion of wards who adopted the intervention), Implementation (how well the intervention was delivered as designed), and Maintenance (how well program effects are maintained and the continued use of the program) [43‐45]. In this report, we have followed the recommendations from the MRC guidance and performed a process evaluation of the intervention with TIME using the RE-AIM evaluation framework.

The primary aim of the study was to perform a process evaluation of the intervention of TIME to ease replication and future implementation. As a part of this evaluation, we wanted to explore what factors inhibited or promoted the implementation. Secondly, we aimed to clarify possible causal mechanisms for the effectiveness of TIME at resident level.

We used a quasi-experimental (a pre-test post-test, control group) and a qualitative exploratory design. Mixed methods of the convergent parallel type according to Creswell and Clark were used [46]. In this type of mixed-method study, qualitative data and quantitative data are collected in the same study and with equal priority. The data are analysed separately but integrated in the overall interpretation of the results.

A total of 89% of nursing homes in Norway are run by the municipalities and 11% are run by non-profit organisations or private enterprises. According to national specific regulations, each nursing home must have an administrative manager, a registered nurse responsible for the nursing care, adequate staffing to ensure necessary care for the residents, and a nursing home physician [47]. Most physicians working in nursing homes are general practitioners in part-time positions. Sixty-three municipalities with a total of 130 nursing homes located in the north, middle, and south-eastern parts of Norway were invited to take part in the cluster-randomised controlled trial (TIME). The choice of this convenience sample was made because of resource reasons to minimise travel time and costs for the research and educational group and, at the same time, strive for a representative sample of NH. The approach was effectuated by an e-mail invitation to the manager of the elderly care department of each municipality. In the e-mail intervention, we clarified that nursing homes already using TIME, nursing homes engaged in other research projects and nursing homes that primarily offered short-term care could not participate in the trial. The research team arranged information meetings for managers and physicians from nursing homes in 32 municipalities who responded positively to the invitation. In these municipalities there were 63 nursing homes. Participation in the study for the municipalities and their NH, as is the rule for most other RCTs, was based on interest and was voluntary [48]. After the information meetings 12 of the 32 municipalities declined to participate in the trial. The reasons they gave for their refusal were, ‘being engaged in other research projects’ (five), ‘lack of resources and being engaged in internal educational projects ‘(five) and ‘already using TIME’ (one). One municipality did not give any reason for the refusal to participate. Finally, a total of 33 nursing homes from 20 municipalities accepted our invitation to take part in the trial. After baseline assessments, 17 nursing homes with 104 included residents from 22 wards were randomly assigned to the intervention group (INH), and 16 nursing homes with 125 included residents from 24 wards were randomly assigned to the control group (CNH) [49].

In this process evaluation of the trial, 797 staff members and 22 leading ward nurses from the INH and 889 staff members and 24 leading ward nurses from the CNH were invited to participate in the survey by receiving questionnaires at baseline. The questionnaires were distributed by e-mail, using a work address list of all employees including both regular and temporary staff members, in each ward who had residents included in the trial. The five focus groups consisted of 32 staff members, leaders, and physicians from 11 of the 17 INH. The nursing homes that were represented in the groups were selected randomly from the pool of INH to minimize selection bias [50]. We did not conduct focus groups in all the 17 INH to minimise travel time and costs for the research group. To achieve information-rich cases and a purposeful sample, the leading ward registered nurse in each nursing home selected who of the staff would attend the focus groups [51]. The criteria for the selection of participants were that they should be familiar with TIME and be able to promote views in a group discussion. A detailed description of the composition of the focus groups can be found in the Additional file 2 (Methods used for the collection of qualitative data and their analyses).

To explore adoption and maintenance, five focus groups interviews with 32 of the caregivers, leaders, and physicians from 11 of the 17 INH were conducted 3 to 6 months after the end of the 3 month intervention [57]. To further assess implementation, we collected the minutes from 84 of the 85 performed case conferences in the INH. A detailed description of the selection and the composition of the focus groups and the collection of the minutes can be found in the Additional file 2 (Methods used for the collection of qualitative data and their analyses).

The mean reach fraction for the attendance to the training sessions in the INH were 61% (SD 22) and 51% (SD 24) for the brief educational session in the CNH. In the INH, seven out of 17 nursing home physicians (41%) and 14 out 22 (64%) of the leading ward nurses attended the training sessions. Of the 78 TIME administrators, 43 (55%) were registered nurses or had an education equivalent to at least 3 years of college or university education in the field of elderly or health care and 35 (45%) were auxiliary nurses. On average there were 8 (SD 2.9) staff members participating in the case conferences ranging from 3 to 19. Nursing home physicians and leading ward nurses attended 27 (32%) and 42 (49%) of the case conferences, respectively.

The results from the questionnaires on the effectiveness at staff level are presented in Table 5. There were no statistically significant group differences throughout the study period for the sum scores for attitudes towards dementia, perceived competence or perception of mastery and social interaction for attitudes towards dementia, perceived competence or perception of mastery and social interaction. The sum score for attitudes towards dementia increased from baseline to twelve months in the INH (p = 0.029), and in the CNH (p = 0.006). The sub-score for hope attitudes increased from baseline to twelve months in the intervention group (p = 0.012) and from baseline to 6 months (p = 0.003) and to twelve months (p = 0.001) in the control group. The sum score for perceived competence for individual staff members regarding NPS increased from baseline to twelve months in the INH (p = 0.002). There was a reduction in the social interaction score for the CNH from baseline to 6 months (p = 0.009).

Sixteen of the 17 INH completed the intervention. One INH, with one ward and seven residents included, withdrew from the trial approximately 2 weeks after the start of the intervention (i.e. after both the randomisation process and the training sessions). The reason given for this by the leading ward nurse, was an extremely high and prolonged level of sickness leave in the staff. It was not possible for her to assign someone from the staff to be interviewed for the assessments of the residents at eight and 12 weeks. Of the 22 wards in the INH that completed the study, all but two wards performed more than 75% of the components in TIME for their included residents as measured by the Fidelity Questionnaire. These two wards completed all or part of the first phase of TIME but performed no case conferences or evaluations of treatment actions. For both these wards the reasons given from the leaders for this reduced adoption of TIME, were lack of time and resources.

The INH and the CNH were not statistically different throughout the study period regarding adherence at ward level to recommended clinical practice for the assessment and treatment of NPS, as measured by the Current Practice Questionnaire (See Table 5). Both groups did however increase their scores from baseline to six and twelve weeks indicating better adherence to - and maintenance of - recommended clinical practice.

Overall, the staff performed 75% or more of the components in TIME for 91% of the included residents as measured by the Fidelity Questionnaire (i.e. a score ranging from 15 to 20). The INH performed a case conference in 91% of the included residents. The number of included residents with completed main components in TIME, is presented in Table 6. No nursing home performed more than one case conference for each resident during the 3 month trial. The average number of weeks that elapsed from intervention initiation (i.e. the randomisation date) to the performance of the case conferences was 5.0 (SD 1.8).

The results from the documentary content analyses of the 84 min of the case conferences are illustrated in Fig. 1. The description of the problems analysed during the case conferences (i.e. description in detail of symptoms or behaviour) was performed adequately in 22 (26%) of the case conferences. For the description of the evaluation procedures this was done adequately in 49 (48%) of the case conferences. For the other structuring components of the case conferences, adequately understanding and use of them varied among the components from 69 (82%) to 84 (100%) of the minutes.

All but one of the 11 nursing homes represented in the five focus groups were still using the model 3 to 6 months after the end of the intervention. The frequency of the case conferences varied from once a week to once a month. The nursing home that had stopped to use the model, did so because they had moved all their residents to a new building. They considered re-implementing TIME as soon as they settled in. Several factors were considered when the focus groups discussed what promoted and what inhibited the maintenance of the model. These findings are summarised in Table 7.

One important condition for using the model was that it was considered simple to learn, to understand, and to start using. Another condition was that they could see an immediate result of their efforts for the residents.

All the groups believed that the intervention had to be properly supported by the leading registered nurse. Time allocated and active support by the leader for assessment tasks was important. In addition, the staff thought that the leaders ought to actively give support in the case conferences and support for follow-up and evaluation procedures. It was deemed vital that as many staff members as possible could participate in the initial educational program. This was important for creating acceptance among all the staff for this way of working. The TIME administrator’s role was highlighted by both the staff and the leaders, and by themselves as essential for the implementation process. At the same time as they were an integrated part of the staff, they also felt a commitment to engage each other as a group, and the rest of the staff in the work to implement the different components of the model.

A theme that was recurring was the difficulty encountered in arranging and finding time for the case conferences. These difficulties were regarded as the main barrier to a proper implementation of the model. Here too, wards with an engaged leader who saw the benefits of the intervention, did better. The leaders in these wards were able to incorporate the case conferences as a part of the routines in the wards. The participants also reported better attendance at the case conferences in these wards. In other nursing homes, attending the case conference was voluntary for the staff who were not on duty. Some nursing homes did not provide compensation for the attendance on free time. Several participants attributed varying attendance at meetings and varying frequency of the case conferences to these factors and to the leader’s engagement.

The importance of aiming to train most staff (i.e. high degree of reach) was highlighted by Rapaport et al. (2017) in a systematic review on the effective components of psychosocial interventions [64]. In this review, staff perceived difficulties sharing new approaches with those who had not attended training sessions.

A high degree of adoption and implementation in our study can to some extent be explained by the finding that TIME was perceived by the staff as an easy to grasp model. To achieve acceptable adoption and implementation, several studies support the importance of interventions to be simple, appreciated by the staff, and not too overly complex [37‐39].

Though implementation and adoption were high in our study, TIME was used with some variation between the study sites. The variation in the time the staff needed to perform the assessment of each resident, how long it took them to organise a case conference, and how many of the staff from each ward that attended case conferences, may in part be explained by organizational conditions. Allowing for some adaption of complex interventions to the organisational context of the settings is in line with viewing organisations (e.g. nursing homes) as complex systems. This flexibility of a complex intervention has been emphasized as an important factor to improve the effectiveness of the interventions by increasing their applicability [16, 35, 42].

The INH and the CNH did not differ throughout the study period regarding adherence to recommended clinical practice towards NPS as measured by the Current Practice Questionnaire (see Table 5). This can be due to several reasons. Firstly, participants in a research project could be affected by the so-called Hawthorne effect meaning that the participants change their behaviours only because they are monitored in a study [65]. Secondly, the questionnaire is based on self-reporting of practices by the leading ward nurses. Self-reporting of behaviour change in these settings is prone to bias [17]. It is possible that leaders from both groups gave more favourable answers than what are reflected in their practices. This could have offset any differences between the two groups. Thirdly, both groups had similar high scores at baseline, indicating that they already followed most of these recommendations providing less possibilities for improvement. Finally, both groups received lectures on NPS and recommended clinical practice for the approach towards NPS. Since both groups significantly increased their scores on this questionnaire throughout the study, this could indicate that both groups followed, to some degree, the recommendations from these lectures. As will be further discussed in the next paragraph, the effectiveness of TIME at resident level might to a lesser degree depend on the clinical practice routines.

Though there was an increased level of positive attitudes towards dementia in both groups, and an increased level of knowledge in the intervention group, there were no between-group differences in any measurements of effectiveness at individual staff level throughout the study period. These results are in line with the results in two recent systematic reviews on the impact of staff training on staff outcomes [40, 43]. However, in the same reviews some studies found increased staff knowledge and self-efficacy, but these findings were not consistently maintained over time. On the other hand, our recently published report exploring in depth the staff’s learning and coping experiences with TIME, showed that the staff perceived an increase in specific knowledge and coping related to each individual resident [32] . These changes were interpreted as a consequence of a new innovative learning process that took place during the interdisciplinary case conferences. This learning process created a new and shared situated knowledge about the resident and made it easier to customise the approach towards the individual resident. Most questionnaires assess mainly general attitudes and general knowledge about dementia [66]. These general attitudes and knowledge do not necessarily reflect the staff’s capability to translate this general competence into their real-world approach to an individual resident with, for example, severe agitation by which they are confronted on an everyday basis. The creation of a new, shared and situated knowledge about each resident might be one of the most important contributions to the effectiveness of TIME at staff level.

Our data suggest that most of the nursing homes were still using TIME 3 to 6 months after the end of the intervention. Adherence to recommended clinical practice was sustained after 12 months. In accordance with our results from the focus groups, process evaluations of complex interventions in nursing homes highlights the importance of management support to achieve both adoption and sustainability [37‐40, 43, 64, 67]. In the systematic review from Rapaport et al. (2017), a leadership attending training sessions and meetings, and allocating space and time for the staff to engage in the intervention, was conceived as one of the most important facilitators for a sustainable implementation [64]. Most of the leading registered nurses in the TIME trial attended the same educational training sessions as the rest of the staff and they also attended nearly half of the case conferences. The finding that the TIME administrators were important for the implementation process and acted as implementation leaders, is in line with other studies [68‐71]. Scalzi et al. (2006) used the label “change champions” and regarded them one of the most important enablers for culture change [69].

As discussed for adoption and implementation, the staff expressed that TIME was an easy to grasp model. They emphasised that this feature had an impact on the possibility for continuing using the model. In the trial, after a short education and training programme, the TIME administrators in cooperation with the leading ward nurse carried out the intervention independently of the research team. This will increase the probability for maintenance of the intervention, reduce the cost for implementation and ease the proliferation of the model.

Turnover of care staff is often reported as a main barrier to sustainable interventions in nursing homes [64, 72]. However, turnover did not emerge as a subtheme in the focus group discussions when raising questions about barriers to the implementation process. The high degree of attendance to training sessions from all staff including those working part-time and with temporary positions, participation of the leading ward nurses in the training sessions, selecting three TIME-administrators among the nursing home staff as implementation champions and an easy-to-grasp model may have outweighed some of the effect of staff turnover.

The main strength of this process evaluation is the rigorous use of a well-established and recommended framework for evaluation of complex interventions [42, 44]. The RE-AIM framework mainly emphasises quantification of the five dimensions reach, effectiveness, adoption, implementation, and maintenance to explore what is called the public health impact of the process of implementation [42, 44]. Less emphasis is put on the process of implementation. We have therefore in our study included issues regarding facilitators and barriers of the implementation process and considerations on causal assumptions of the effectiveness of the intervention [42].

The only interference during the randomised controlled trial was the use of the checklist for assessing performance of the main components of the intervention to explore implementation (fidelity). This checklist could have possibly served as a reminder for the TIME administrators.

During this process evaluation we have used a quasi-experimental design giving us the same quantitative data from the control condition as from the intervention group. Another strength is that we used a mixed-method design which allows us to compare and interpret quantitative results from the survey with findings from the perspectives of the staff, leaders, and physicians derived from the focus groups. This comparison adds to the validity of our results and to the value of our assumptions on possible causal mechanisms of the effect of TIME at resident level [41, 46].

One important limitation is that we do not know to which extent the model became a stable enduring part of the clinical routines of the nursing homes beyond 6 months after the end of the intervention.

One other limitation is that this process evaluation was performed by the same research team as the team responsible for the trial [42]. It was also performed when results from the trial were known, even though all the data collection for the process evaluation were planned for in a published protocol, and collected independently of the trial [18]. Both these phenomena could have created some bias in the analysis of the data. On the other hand, integrating process and outcomes’ evaluation by the same team, may limit the risk of one collection of data disturbing the other.

The relatively low response to some of the questionnaires is also an important limitation. One of the reasons for this is probably that the nursing home staff did not regularly use the e-mail account at work. We do not have data on the part of the staff who did not participate in the survey. This limits the external validity of the results. However, a recently published study comparing staff characteristics in special care units and regular units in Norway did find similar staff characteristics as in our sample [73].

The questionnaires to the staff members at 6 months and at 12 months were only administered to those of the staff wo had answered the questionnaires at baseline and at 6 months, respectively. Given the staff turnover within nursing homes, this procedure of data collection limits our possibility to draw conclusions on the impact of the intervention on the whole group of staff members throughout the trial and on an eventually pass on effect of the intervention among the staff members. Future process evaluations of interventions in care settings, should explore also this aspect of an intervention since this might be crucial for the intervention’s sustainability.

The successive reduction of responders at 6 months and at 12 months may also have influenced the results on the measured outcomes. It is possible that those who continued to respond are more positive to the intervention and in general more interested in professional development. However, the response rates were similar in both INH and CNH throughout the study period. Assuming that the characteristics of the staff that did not participate in the survey and their reasons for not participating are equally distributed in the two groups, the comparison between the two groups should, however, be judged as valid.

Two of the of questionnaires in the study were developed by the research team to be brief and to evaluate the implementation process inherent to TIME. With these questionnaires we wanted to evaluate the self-reported competence of the staff members and the clinical routines in the NH regarding the approach especially towards NPS, not towards dementia in general. Because of these requirements we did not find other equivalent questionnaires. They were not validated, and their psychometric properties are not known. This is an important limitation in this process evaluation study. However, since the questionnaires are carefully developed to reflect the content of the lectures and training sessions in both the INH and the CNH, we do believe they measure, at least to some extent, what the staff members eventually have learned from these lectures and training sessions in both the INH and the CNH. This is supported by the finding that perceived competence of the staff members increased significantly from baseline to 12 months in the INH, but not in the CNH. The Fidelity Questionnaire is a check-list for the performance of the main components in TIME, conducted by interviewing the TIME administrators by telephone, and is, therefore, less prone to self-reporting bias. There is still a concern about the accuracy of self-reporting of behaviours without checking with objective fidelity measures [17]. However, the documentary analysis of the meetings from the case conferences in our study contributed to a more valid judgement of the implementation (fidelity) dimension of the RE-AIM framework.