Results
After reporting response rates we present our results for each link in the logic model. For brevity we present only overall scores except where necessary to interpret or disambiguate the results.
DLC trainees’ response rate was 100%. For the staff survey (all sites) it was 202/603 (33%). Staff sickness and turnover data for were available for 18/23 sites (78%). Pre- and post-intervention Dementia Care Mapping (DCM) data were obtainable for 15/23 sites (65%), of which 5 were controls and 10 DLC sites. QUALID data were obtained for 246 residents, i.e. 38% of the study homes’ total population. Data on Treatment Escalation Plans were obtained for 201 residents (31%). Ambulance data for ‘See & Treat’ and ‘See & Convey’, but not for ‘Hear & Treat’, call-outs were available for 21/23 sites (91%). 62 (i.e. 19%) of the initial 332 residents in the study had no formal diagnosis of dementia at the start of the evaluation, but 47 of those 62 had a GDS score of 4 or above, indicating ‘deficits … clearly manifest in a detailed clinical interview … subjects … who enter this fourth stage almost invariably manifest subsequent deterioration characteristic of dementia of the AD type’ [
55]. Two DLC sites dropped out, one because a local authority had concerns about quality of life there and ceased referrals to it. The other changed ownership. Testing for the presence or strength of a given link is only possible, however, for the sites which supplied data about both its antecedent and its consequent, so the amount of usable data for each link was less, as reported below.
Link a: Training and impacts on staff
The same training took place in all DLC sites, and they all implemented it as planned. Table
2 shows the main training outcomes that participants reported.
Table 2
Self-reported training outcomes
Better prepared for dealing with aggressive resident behaviour | 13 |
Confidence building | 5 |
Learn about end of life care | 1 |
Communicate better | 12 |
Understanding dementia [named specifically] | 108 |
‘Awareness’, ‘knowledge’, ‘learning’ [topic unspecified] | 102 |
No response | 40 |
Total | 282 |
Dementia champions were recruited by asking the care home managers to identify volunteers for the role, which all except two managers did. In default these two managers took on the dementia champion role themselves. As their intentions for working differently in future, trainees mentioned listening to and observing residents more (staff from sites A,F,J,M,U,W); learning more about the residents as individuals from documented information, other staff and residents themselves (sites A,B,F,I,J,R,U); interacting more with residents and spending more time with them (sites A,B,G,J,I,M,Q,R,S,W); helping fellow-workers work more as a group and communicating more with them (sites B,F,G,J,I,Q,W) e.g. by making better use of the home’s communication book. A few mentioned giving more ‘person centred’ care (sites G,M,S) or being more compassionate and tolerant (sites B,G). One, more modestly, intended to ‘stop annoying the residents’ (site S). Only one trainee asserted that there was ‘nothing’ he/she wanted to do differently.
As the logic model predicted, DAS and ADQ total scores rose in the DLC sites and the SNCW total scores fell. However the training mostly did not appear to make the staff characteristics change further in the predicted direction in the DLC than in the control sites. Mean DAS total score rose by 2.18 in the DLC group but by 4.39 in the control group, over a scale 120 points long. The difference was not statistically significant (Wilcoxon rank sum W = 57, p = 0.86). Neither was the difference for mean ADQ total score, which fell by 0.76 in the intervention group and rose 0.5 in the DLC sites, on a 76-point-long scale (W = 41,p = 0.42). Mean SNCW total score fell 7.53 points in the DLC sites but rose 0.56 points in the control sites, on a scale 128 points long, a difference which was statistically significant (W = 65,p = 0.045), because of decreases (separately not quite statistically significant) in the Quality (W = 62.5, p = 0.07) and Workload (W = 63, p = 0.06) components of SNCW. The Cooperation (W = 45.5, p = 0.47), Development (W = 58,p = 0.14) and Patient Knowledge (W = 60.5,p = 0.09) components of SNCW showed no significant difference. Mean sick days per staff member per year significantly decreased by 1.35 in the DLC sites but increased by 0.42 in the control sites (W = 54, p = 0.05). We found no significant difference in staff turnover.
Link B: Training and PDSA cycles
The initial training did trigger cycles of PDSA activity in all homes that participated. Partly the dementia champions expressed their aims for the PDSA cycles in general, aspirational terms, most often ‘To show us the right path regarding the dementia experience’ (site J; and, differently worded, sites F, I,Q,R,S,U,W). ‘Person centred (‘individualised’, ‘personalised’) care’ was mentioned in sites G,I,J,Q,R,S and W, and care home culture by the dementia champions for sites A, B and M. However, the dementia champions also stated more concrete, practical aims for their PDSA activity. In descending order of frequency, these aims were: to find out more about the home’s residents in order to inform staff interactions with them (sites G,I,M,S,U,W) or, more concretely, by producing one-page resident profiles or ‘This is me’ notes (sites A,B,F,I,Q,R). A second set of planned activities concerned environmental enhancement for residents, including reviewing care routines and making them more person-centred. These activities included the use of memory boxes (site B), residents helping themselves to vegetables at mealtimes (site S); having books, magazines, newspapers, games (cards, dominoes) for all residents to use when they wanted (site U), enabling residents to have personal items to hand (site U) or having their room set up as they liked (site U), and having assisted baths first thing in morning (site G). Staff received further training in dementia awareness at sites J,M,R and W.
Link C staff characteristics and immediate effects of work routines
Initially the study sites had generally low overall Well- and Ill-Being (WIB) scores. Only six sites scored ≥2.5, and only three of them (B,U,V) met the DCM standards across all categories. At the end of the scheme (approximately one year), five DLC homes showed marked changes in their WIB scores but another five did not (Table
3).
Table 3
WIB score changes
D | C | 1.10 | 0.90 | −0.2 |
H | C | 0.73 | 0.7 | −0.03 |
O | C | 0.95 | 0.94 | −0.01 |
K | C | 1.02 | 1.2 | 0.18 |
V | C | 0.66 | 0.85 | 0.19 |
A | DLC | 0.89 | 0.00 | −0.89 |
B | DLC | 1.80 | 1.25 | −0.65 |
I | DLC | 0.96 | 0.87 | −0.09 |
S | DLC | 0.49 | 0.54 | 0.05 |
M | DLC | 0.88 | 1.14 | 0.26 |
R | DLC | 1.15 | 1.95 | 0.8 |
F | DLC | 1.34 | 3.21 | 1.87 |
U | DLC | 1.06 | 3.01 | 1.95 |
Y | DLC | 0.03 | 2.11 | 2.08 |
W | DLC | 0.31 | 2.61 | 2.3 |
In sites A,B,I,M and S the changes in WIB scores were within the same range as those for the control sites, i.e. they either increased by less than 30% or fell. Indeed DLC sites A and B showed larger falls in WIB score than any of the control sites. In aggregate there was no significant difference between the Low-WIB DLC sites (A,B,I,M and S) and the control sites. A second group of sites (F,R,U,W and Y) showed another pattern. Their WIB scores rose by between 70% and 742%, a statistically significant mean increase of 1.8 points (W < 0.1, p < 0.01). For short we label the two groups ‘High-WIB’ and ‘Low-WIB’ sites. It therefore appeared that either or both of links C and D were present in five ‘High-WIB’ sites, and stronger there than in the controls.
To test whether link C was present, we tested for association between the measures of staff characteristics in the DLC sites, and in particular for the High-WIB sites taken alone, and the respective WIB Scores. Taking all the DLC sites together, we found no association between the proportionate changes in DAS total score and WIB score (W = 52, p = 0.86) or ADQ total score and WIB score (W = 66, p = 0.47). The proportionate change in WIB score was however associated with proportionate change in SNCW score, and negatively as predicted (SNCW being reverse-scored) (W = 97, p < 0.01).
Of the presumed antecedents of link C, ADQ and ADS scores (of staff attitudes and knowledge) did not significantly differ between the DLC and the control sites, but SNCW scores did (W = 65, p = 0.05), both for all DLC sites and the High-WIB sites taken separately (W < 0.01, p < 0.01).
Link D: PDSA activity and immediate effects of improved work routines
We found evidence (both interviews and physical artefacts) at the DLC homes that knowledge from the training sessions had been used to initiate post-training PDSA activities directed at improving working practices there. At the High-WIB sites (F,R,U,W,Y) PDSA activities focused on: elaborating residents’ care plans using information gathered at the training session and subsequently (site U); periodic staff meetings about residents (site F); designating key-workers responsible for particular tasks and/or residents (site R); setting up routine team meetings so as to re-iterate and sustain future PDSA cycles (site Y); and training staff by letting them experience what life as a care home resident is like (e.g. being fed by someone else, wearing incontinence pads etc.) (site W). However PDSA cycles in low-WIB homes initiated many similar activities. Sites A and I identified key-workers responsible for particular tasks and/or residents, Sites I and S initiated routine team meetings to sustain future PDSA cycles, since ‘You can learn all the time about dementia’ (Dementia champion). Site B did small-scale initial testing to see if new ideas worked. Homes M and S initiated planning morning activities for residents. Two homes for which we did not have before-and-after WIB data also reported similar post-training PDSA activity. Periodic staff meetings about residents were introduced at sites G and Q. In addition site G identified key-workers responsible for particular tasks and/or residents, established routine team meetings so as to sustain future PDSA cycles, undertook small-scale initial testing to see if ideas worked, and introduced planned morning activities for residents.
So far as we are aware, data from PDSA activities were not usually documented. Neither, therefore, were such data re-used over time. PSDA cycles were predictively-based only in the sense that participants anticipated certain broadly-defined outcomes from the ‘Do’ phase (e.g. that residents would become happier). Thus PDSA cycles were implemented, but compliance with the PDSA model [
22] was patchy. There was no PDSA activity in the non-DLC sites.
What differentiated the high-WIB and low-WIB sites was not whether PDSA cycles followed the initial training nor, mostly, what the contents (foci) of the PDSA cycles were. So perhaps other factors not recognised in the logic model, such as the organisational character of High-WIB homes, were responsible instead. High-WIB homes tended to be larger (a mean of 29 beds versus 25.8) and have a higher staffing ratios (1.16 wte staff per resident versus 0.93) but in our data these differences were not statistically significant. High-WIB and Low-WIB sites did not differ in terms of the aspirations stated in the post-training questionnaires and follow-up sessions, home location or ownership type. The only resistance to DLC activity was in Low-WIB sites. In one, the manager reported that some staff were suspicious and defensive about the DLC and would not complete residents’ ‘This is me’ documents. That and another Low-WIB site had difficulty recruiting dementia champions. The absence of link C leaves only PDSA activity (link D) to explain the increased WIB scores in the five ‘High-WIB’ DLC sites, but apart from eliminating home location, size, staffing and ownership the data available to us did not reveal what other contextual differences between High-WIB and Low-WIB homes enabled PDSA activities in the former to change working practices sufficiently to raise their WIB scores.
Link E: Effects of improved work routines and quality of life
At baseline, not one site scored better than the mid-point of the QUALID scale. Taking DLC and control sites together, change in WIB score was associated with proportionate change in total QUALID score (W = 155, p = 0.03) and in each QUALID component, although not in the High-WIB homes taken alone. Comparing DLC and non-DLC sites, total QUALID scores fell in both intervention and control homes, by a mean of − 1.68 and − 0.57 respectively for a 44-point scale, but the difference was not statistically significant (W = 39, p = 0.23). We found the same pattern for each separate component of QUALID. We also compared only the High-WIB homes with the controls, but again found no significant differences. Considering the high mortality among this population it was striking how few end-of-life care plans there initially were; 42 in the control and DLC homes combined (646 beds). There was no difference between DLC and non-DLC homes in the change in frequency of use of end-of-life care discussions, care plans, or TEP forms.
Link F: Quality of life and external impacts
Contrary to the logic model, improved QUALID scores were, across all sites, associated with increased (not reduced) ambulance call-outs (by an average of 1.6 call-outs per year per home) (W = 343, p = 0.03). Comparing all DLC sites with the controls also showed no significant difference in the change in rate of ambulance call-outs (W = 70, p = 0.06). In respect of ambulance call-outs, link F was not present.
Across all the sites, changes in QUALID total scores were not associated with changes in the rate of all admissions (W = 373, p = 0.08), nor with changes in the rates of emergency admissions specifically (W = 295, p = 0.3). At the start of the study DLC sites already had lower admission rates (both planned and unplanned) than the control sites. Nearly all hospital admissions (374/389, 96%) from all the homes were as emergencies. During the project the number of admissions per bed per year did fall in the DLC sites (by 20% for all admissions and 27% for emergencies) but since admissions also fell in the control sites the difference-in-differences analysis showed that for all admissions the ‘treatment’ (i.e. DLC) effect was negligible (δ approaching zero). For emergency admissions it was in the predicted direction but small (δ = − 0.3 emergency admissions per bed per year), and still not statistically significant (p = 0.29). For hospital admissions too link F was absent.
Discussion
Our methods assumed that no confounding change occurred during the DLC project, and no ‘contamination’ of the non-DLC sites with DLC work routines, even though DLC and control sites were often nearby and staff turnover (hence transfer between workplaces) was frequent. Also our methods assumed a comparable case-mix during the two years of the study, and between homes, despite high mortality among residents. Having volunteered as DLC sites, one might expect the study homes to be if anything more motivated than other care homes to implement and exploit the DLC model. Comparing the above results with those of an initial pilot analysis of the external impacts six months into the study, it was noticeable how rapidly the
p-values fell towards significance as the quantity of data increased, raising the question of whether a longer evaluation might yield results more favourable to DLC. Evaluations of other ‘culture change’ interventions in residential long-term care suggest the importance of allowing interventions enough time to mature or ‘bed down’ before evaluating them [
56]. Our results also suggested that if the observed changes in emergency admissions were indeed due to the DLC, they took over six months to appear.
As reported above, ADQ and ADS scores (of staff attitudes and knowledge) did not significantly differ between the DLC and the control sites, but SNCW scores did. We also found that the location, size, staffing and ownership of High-WIB homes, did not appear to explain why Link D, beween PDSA activity and immediate effects of improved work routines, differed between high-WIB and low-WID sites. These results suggest revising the logic model, and in particular re-interpreting link C.