Introduction
Process evaluation study
RE-AIM dimension | Definition in PACE Process Evaluation |
---|---|
Reach | Proportion of caregivers in care settings that participated in the intervention during the study |
Effectiveness | Primary and secondary outcomes (positive and negative) |
Adoption | Extent to which caregivers actually adopt the intervention in the study (showed compliance with the intervention) |
Implementation | Extent to which the intervention is implemented as intended in the real world, including implementation barriers and facilitators |
(Intention to) Maintenance | Extent to which the intervention is intended to be sustained over time |
Methods
Design
Data collection
Component1 | Source | Timing of collection | Measure | Scoring criteria |
---|---|---|---|---|
Reach | Attendance lists from training steps 1-6 | Month 3–18 | Mean attendance rate (number of care staff attending a training session divided by the total number staff working in the LTCF) on all 6 PACE steps | < 30% = low 30–69% = medium ≥ 70% = high |
Effectiveness1 | - | - | - | - |
Adoption | Report from PACE coordinator at end of consolidation period | Month 12 | Proportion of residents with Looking and Thinking Ahead document (number of residents with document completed divided by the number of PACE beds) | < 40% = low 40–79% = medium ≥ 80% = high |
Implementation | Diaries country trainers | Month 1–12 | 1. Score for fidelity (0-8 points) (a) Number of training steps delivered (0-6 points, 1 point per step) (b) If 6 training steps delivered: delivered in right order from 1 to 6? (yes = 1 point) (c) If 6 training steps delivered: delivered within 8 months? (yes = 1 point) | Fidelity ≤ 4, satisfaction 0–8 = low Fidelity > 4, satisfaction < 4 = Low Fidelity ≥ 5, satisfaction 4–5.9 = medium Fidelity 5–6, satisfaction ≥ 4 = medium Fidelity ≥ 7, satisfaction ≥ 6 = high |
Evaluation questionnaire | Month 8 | 2. Score for satisfaction of care staff members (0-8 points) (a) Mean score for care staff members’ satisfaction with trainer’s teaching competences (not at all = 0 points, a little = 1 point, somewhat = 2 points, quite a lot = 3 points, very much = 4 points) (b) Mean score for care staff members’ overall evaluation of complete PACE Programme (very poor = 0 points, poor = 1 point, fair = 2 points, good = 3 points, very good = 4 points) | ||
(Intention to) Maintenance2 | Interview with manager | Month 13–15 | 1. LTCF manager’s intention to continue with (elements of) PACE in the future (no = 0 points, yes = 1 point) | Manager = 0, care staff < 5 = Low Manager = 1, care staff < 4 = low Manager = 0, care staff ≥ 5 = medium Manager = 1, care staff 4–5.9 = medium Manager = 1, care staff ≥ 6 = high |
Evaluation questionnaire | Month 8 | 2. Care staff members’ intention and recommendation to work with PACE (0-8 points): (a) Mean number of PACE steps/documents they intend to use in the future (0-7 points, 1 point for each PACE step/document) (b) Mean score for recommending PACE to other LTCFs (no = 0 points, yes = 1 point) |
Data analysis
Results
Participants
BE | FI | IT | NL | PL | EN | CH | Total | |
---|---|---|---|---|---|---|---|---|
N intervention LTCFs | 6 | 6 | 6 | 6 | 4 | 5 | 4 | 37 |
N country trainers | 3 | 3 | 2 | 2 | 3 | 1 | 2 | 16 |
N PACE coordinators | 21 | 22 | 11 | 13 | 12 | 10 | 10 | 99 |
N staff members who completed evaluation questionnaire [response rate] | 182 [63%] | 348 [70%] | 164 [77%] | 57 [39%] | 204 [91%] | 74 [50%] | 139 [84%] | 1168 [74%] |
N interviewed staff members (N interviews) | 27 (6) | 33 (6) | 20 (5) | 13 (5) | 22 (4) | 16 (4) | 20 (4) | 151 (34) |
N interviewed PACE coordinators (N interviews) | 12 (3) | 16 (3) | 8 (6) | 9 (4) | 11 (2) | 7 (5) | 10 (2) | 73 (25) |
N interviewed facility managers1 | 6 | 6 | 2 | 4 | 4 | 3 | 4 | 29 |
Ratings on RE-AIM components
Reach
Factors affecting Reach
Reach | |
Organization of PACE Training | (1) The meeting was held in the beginning of the evening shift so only two people were able to attend. Attendance list, Finland |
- Time of the day | (2) “There was the willingness to double the training session, or the country trainer was available for those who were not in nursing home in the evening, to repeat the same step again in the afternoon.” PACE coordinator, Italy |
- Scheduling in advance | |
- Extent to which training is promoted | (3) Our aim is not that as many care staff members as possible attend the training. In this LTCF, only the PACE coordinators, head nurse, unit managers and maybe one other member of the team participates. Attendance list, Belgium |
- Number of sessions of same PACE training step | |
- All staff members or selected group invited to participate | |
Availability of personnel | (4) “I find that that the idea of doing sessions one staff at a time is great. But in reality, come the day, you might have an incident going on and where we are there is one nurse on the shop floor. And if you are that nurse and you are attending that session something happens, it goes out the window. You are either dealing with another professional who is come in doctor or whatever it is or you are dealing with some other matter with relatives or client. That kind of goes to pot a bit.” PACE coordinator, England |
- Staffing problems | |
- Conflict ‘attending training vs. caring for residents’ | |
Motivation and expectations | (5) “Especially after the first session, because before that it was a bit vague what it entailed and what was expected from us and what is meant by this. But especially after the first session I think everyone had an idea of it. I: Was that helpful to increase motivation? R: It only increased after this session.” Care staff, Netherlands |
- Level of interest in palliative care | |
- Concurrence with other projects | |
- Change of motivation/enthusiasm during project | |
- Extent to which expectations are met | (6) “When the training sessions started, it’s true that we felt a weakening of the staff’s motivation. Because there were things that have already been seen or differently applied, because it mobilizes things we couldn’t mobilize, in the way they were set in the functioning.… It’s true, people expected something more technical or more this or more that…And eventually they had something which was not announced as such or defined as such. Consequently, they were disappointed in regards of their expectations.” PACE coordinator, Switzerland |
Stimuli or incentives | (7) “We also said that if you attend 4 of the 6 training sessions, you’ll receive a certificate. That helped, that really was to them…they really wanted that, to receive a certificate, so that was a trigger.” Manager, the Netherlands |
- Financial reimbursement | |
- Certificate | (8) “They told us attending the course was obligatory. Some of the staff took this very badly. It isn’t very stimulating if you have to attend even at your free days or recuperation days. Especially if you don’t know what the training will be about.” Care staff, Belgium |
- Extent to which training was compulsory | |
- Manager freeing up time for staff to attend training | |
Adoption | |
Content of documents | (9) “And then there was that list, also a depression list…I: yes for people without dementia..R: Yes everyone scored high on that one. And I think with those questions ‘do you usually feel happy? And do you feel your life is empty?’. Everyone scored quite high on that, but then again there is a part of loneliness and.. so I found that difficult, is this really added value? No I didn’t find it [fit to use here], at least we are not using it.“ PACE coordinator, Netherlands |
- Language | |
- Applicability to LCTF population | |
- Perceived completeness | |
Organization of daily care practice | (10) “We see each other almost every day, sometimes every two-three days, they treat us almost like close relatives. Those who are for example several years, for a very long time, they do not feel any barriers to answer our questions.” Care staff, Poland |
- Allocation of staff members to residents | |
- Accessibility of documents (electronic or in resident chart) | (11) “But it’s like, you can’t draw a graph for all the people and especially because your documents don’t go hand in hand with the electronic systems and then again we use the electronic ones to follow up and do all the data.” Manager, Finland |
- Being used to working with documents | |
Resistance to use documents | (12) “Unfortunately we found quite a lot of the staff were resistant to attending the training, found the documentation complicated. It was a case of the if It’s not broken why fix it type scenario, a lot of them saying well we don’t need to do that we already do that by doing X Y and Z. Why should we do A B and C?” PACE coordinator, England |
- High amount of paperwork and little time | |
- Unwillingness to change way of working | |
- Anxiety that documents are checkable | (13) “I don’t know if I would be able to assess if someone feels the pain on the pain measuring scale, from 1 to 10 points, if someone feels the pain on 5,7,8 or whatsoever. It would be very difficult for me to assess. I think that this should be done by physicians or nurses. And besides, the talk with the families of the residents, it would be also difficult, because I am the therapy worker and on these subjects the families usually talk with the social workers, because they have contacts. We have got a very casual contact.” Care staff, Poland |
- Preference to discuss pain/depression without scales | |
- Feeling unprepared/not skilled to assess pain/depression | |
- Cultural taboo on discussing death and dying | |
(14) “We still are afraid to talk about the death and we are in great stress when we are talking with the persons who are dying., and even more stressful is to talk with their families...The talks on this topic are very difficult, indeed.” Care staff, Poland | |
Target group | (15) “I think with that pain score, the technique to ask it on a scale from 0 to 10, I thought, well that is useful, because then I can tell the doctor, yesterday it was 4 but now it is 8..so something needs to be done.” Care staff, the Netherlands |
- (Assumptions on) preferences of residents, family and GPs with regard to discussing ACP and assessing pain/depression | |
Stimuli from others | (16) “My role in the implementation of the project was massive. Besides promoting the project itself, my role was to encourage the correct use of the tools and then to control and supervise if the instruments were using over time. I invested a lot of time and energy to do this. I can therefore say that the project was adopted for about 70%-80%.” Manager, Italy |
- Extent to which use of tools was compulsory | |
- Manager supervising use of tools | |
- Tools used as indicator of good quality care | |
Implementation | |
Organizational issues | (17) I had a very big problem with performing this Step (step 3) - in one LTCF the staff deceived me several times in order to avoid the necessity of a staff meeting, as well as discussing all residents. In the other one - the staff were resistant in the accomplishment of earlier arrangements, contested them just before starting, during or after the meetings. Country trainer, online discussion group |
- Difficulties in scheduling training steps | |
- Cancellation of training steps | |
Characteristics of PACE trainer | (18) “In my opinion, the country trainer wasn’t very able. People didn’t like to go to the training because she just read everything. She had no experience in palliative care, and that’s what we really missed.” PACE coordinator, Belgium |
- Professional background | |
- Way of teaching | (19) “I found it [country trainer] a really nice man, very capable, no really alright. He did it very well, he knows a lot too, so that is nice. I liked that he also gave us time and space for each of use to tell our story.” Care staff, the Netherlands |
- Approachability and flexibility of trainer | |
Characteristics of training programme | (20) “They could be improved in terms of dilution: I'll explain: to leave more time between one step and another to metabolize and explore the content; because sometimes it seems almost a run, and because between one step and another is totally changing the subject” PACE coordinator, Italy |
- Length, number, order, frequency of training sessions | |
- Adaptability of programme to specific context | |
Maintenance | |
Usefulness of PACE documents | (21) “I am a big supporter of the multidisciplinary meetings and I absolutely want that these stays implemented. Since PACE these meetings are organized every month. I am always present during these meetings and I really think it is useful for myself, because I have very little contact with the residents, and that way I am able to follow the condition of the resident”. Manager, Belgium |
- Added value/changes to daily practice | |
- Balance benefits and costs | |
(22) “We simply don’t have time to sit with people and even have a chat. So, many people have the feeling they were forced to follow a “useless” course, because they can’t use it anyhow.” Care staff, Belgium | |
Future availability of PACE coordinators | (23) “I think that it would be easier to consolidate into the teams had there been more coordinators. Then, it could have been done more deeply, step by step.” PACE coordinator, Finland |
- Appointing staff to consolidate PACE | |
(24) “An important PACE-coordinator left and hasn’t been replaced yet. Another one was on pregnancy leave and will now also leave. These people are young and I understand their choices but it is sad for the project”. Manager, Belgium | |
Organization and policy | (25) “But what I just said, we got extra budget from the care administration office, so we had a budget to do that [roll out PACE]. So we had the luxury that they could get me off of the ward so I was not missed there, that someone else was working there.” PACE coordinator, Netherlands |
- Involvement management | |
- Size of LTCF | |
- Availability of budget | (26) “Yes we will keep it going, because it is a tool we can evidence to people like when they [Care Quality Commission] come in they will be interested in to seeing it. So yes we will keep it going.” Manager, England |
- Electronic accessibility of PACE documents | |
- PACE Programme usable for obtaining other registrations |
Adoption
Factors affecting Adoption
Implementation
Factors affecting Implementation
Intention to Maintenance
Factors affecting Intention to Maintenance
Country-specific challenges in implementing PACE
Discussion
Process evaluation findings in relation to intervention effectiveness
Factors affecting RE-AIM components and recommendations to improve implementation
Category | Recommendation |
---|---|
PACE Programme and way of delivery | Reduce the amount of paperwork by making the tools electronically available, allow a flexible length of time between training sessions, provide clear materials, and ensure that trainers are well qualified in palliative care and teaching. |
People working with PACE Programme | Involve manager throughout the entire implementation period, free up time for PACE coordinators, stimulate attendance of staff members to training sessions, and choose PACE coordinators carefully. |
Contextual factors | Carefully determine programme start, take current level of palliative care knowledge and practice into account, allow flexibility in content and timing of the PACE Steps, integrate PACE into existing procedures and documentation. |