Background
Understanding how interventions are implemented in trial settings is important for interpreting the main trial results and indicating how the intervention might function beyond the trial [
1,
2]. Process evaluation can assess where, when, and why variations in implementation occur [
3]. Variations can be measured by the degree to which the intervention was conducted as planned (fidelity), including adherence to protocol, quality of delivery, and participant response (acceptability) [
4,
5]. The amount of intervention delivered (dose), and proportion of intended target population receiving it (reach) can also be used to assess implementation [
1,
6]. Such data can shed light on why an intervention may or may not be effective and indicate how sustainable the delivery model might be if rolled out into routine practice.
The ‘Hands up for Max!’ hand hygiene study is a cluster randomised controlled trial (RCT) to determine whether an educational package to promote hand washing is effective in reducing absenteeism among pupils and staff in primary schools. Hand hygiene is important in preventing infection as person-to-person contact, including via hands, is a common mode of transmission for gastrointestinal and respiratory infections [
7]. Previous educational interventions in school settings have been associated with reduced absenteeism [
8,
9] and gastrointestinal infections [
10], and increased compliance with hand washing [
11,
12]. Reviews of interventions to promote hand washing, including several studies in school settings, concluded that the interventions were associated with reduced respiratory infections, but that the studies were generally poor quality [
13,
14]. Interventions to improve water quality, hygiene behaviours and sanitation in schools have also been shown to reduce absence among primary school pupils in developing countries [
15].
If results from the trial of the ‘Hands up for Max!’ intervention indicate that it is effective in reducing absenteeism, successful transfer of the intervention into school practice may depend on many factors not evident in quantitative analyses of the main trial data [
1]. Conversely, negative results may be due to implementation not occurring as intended [
16]. This paper uses process evaluation data embedded within a cluster RCT to determine the fidelity, dose and reach of this intervention, and to consider issues of acceptability and sustainability beyond the trial setting.
Results
Reach
Overall, 61.4% of schools delivered the educational package completely (n=101 schools) or partially (n=8). HPA and university staff were unable to determine if it was delivered in 16.9% of schools (27 control and 3 intervention schools). Reasons stated on the log sheets for non-delivery included the school being too busy (6 control schools) and staff changes (3 control, 1 intervention school). One intervention school refused to deliver the program and one control school closed during the study. Reasons for non-delivery were unknown for 19 control and 8 intervention schools because schools did not provide a reason in their phone or email communication with HPA or university staff. Intervention schools were more likely to deliver the program than control schools (adjusted Relative Risk, RR 2.21, 95% confidence interval, CI, 1.69, 2.90, Table
2). The proportion of schools who delivered the educational package (fully or partially) varied by geographic area, the highest proportion being in Bath and North East Somerset. Schools participating in the sub-study were more likely to deliver the program (Table
2).
Dose
The educational resource reached pupils in 61.4% of schools, but there was variation across and between schools in the dose received by pupils and the fidelity of implementation. A description of the observed delivery of the intervention in the four intervention schools selected to participate in the qualitative data collection is provided in Table
3. Not all schools provided all components to all classes. For example, all classes were shown the DVD, but both lessons were not taught to all classes. The ‘What are germs?’ lesson was taught in three out of four observed KS1 classes, and the ‘Healthy hands, healthy school’ lesson was taught in three out of four observed KS2 classes. Results from observation checklists showed that posters were put up in classrooms and toilets in all four schools observed. Stickers were also well used, mostly distributed to pupils after they watched the DVD animation of how to wash hands, or at the end of one of the lessons. KS1 pupils at one school received a sticker as a reward after the teacher had observed them washing their hands properly. The optional components of the educational package (homework, art competition, game) were not used in the four schools observed. Only two teachers made use of the fun facts as additional information in their lessons (Table
3).
Table 3
Components of the intervention observed in the four intervention schools participating in the direct observation
DVD or CD-ROM animation | All schools used the animation. In two schools it was shown as part of whole school assemblies. In one school it was shown to separate KS1 and KS2 assemblies and combined with a demonstration of how to wash hands according to the DVD instructions, with input from pupils. At another school the DVD was shown to individual classes in combination with other intervention lessons. The instructions in the DVD were used by one KS1 teacher as a vehicle for teaching time connecting words such as “first”, “then” and “next” in a grammar lesson. | 4 | 4 | 10 to 25 |
‘What are germs?’ lesson | At one school this lesson was only used for KS2, not KS1. The colouring germ character worksheets were used in KS1 classes. Time spent on the content of this lesson ranged from 8 minutes in a class where it was combined with the ‘Healthy hands, healthy school’ lesson, to 75 minutes in a KS2 class where the pupils were interested and engaged. Three KS2 classes designed their own germ, as suggested in the lesson plan. One KS2 class also used the germ character colouring worksheets designed for KS1. | 3 | 4 | 8 to 75 |
‘Healthy hands, healthy school’ lesson | All KS1 classes used the glitter activity. One KS1 teacher used the glitter activity as a science experiment. The KS2 teacher who did not use this lesson explained that they used the ‘What are germs?’ lesson, and left the glitter lesson to the younger pupils. In one KS2 class the activity was scaled down due to limited time, so that only 6 pupils demonstrated the glitter activity rather than the whole class. | 4 | 3 | 11 to 60 |
Posters | Posters were used in all schools, displayed near sinks, on bathroom walls and classroom doors. One KS1 class cut out pictures from photocopied posters and pasted them to make their own hand washing instruction pictures. Another KS1 class used the poster to review correct hand washing technique. Posters at one school were laminated to enable them to be displayed for a longer period of time. | 4 | 4 | - |
Stickers | Stickers were used at all schools, sometimes distributed to all pupils after watching the animation, or provided as a reward for correct hand washing technique. | 4 | 4 | - |
Fun facts | Additional background information about hand washing and germs were used in lessons by two KS2 teachers. | 0 | 2 | - |
While data on the specific components of the intervention delivered were not systematically collected for all schools in the study, particularly among control schools, ad hoc communications recorded on log sheets supported the observations from the process evaluation schools that there was variation in the dose of the intervention delivered. For example, five intervention schools stated that they did implement the homework, but two intervention schools stated that they did not because they had no-homework policies. Staff at four intervention schools commented on their use of stickers but a head teacher at another intervention school indicated they did not use stickers for the older pupils.
Fidelity
Communications with schools recorded in the log sheets indicated that most intervention schools delivered the DVD and lessons with fidelity, as described in the resource pack and during the telephone discussion with a HPA or University staff member. Of the 34 intervention schools that provided specific details about the intervention delivery, 33 schools stated that they showed the DVD at whole school assemblies, consistent with the instructions. Fewer details were recorded about the intervention delivery among control schools in the log sheets but feedback received in interviews with coordinating HPA and university staff indicated that fidelity was perceived to be better when the intervention was taught in intervention than in control schools.
I think some of the schools might have used it in a class here or there, but it wasn’t necessarily that they used it as well as we wanted them to in this year’s study (control schools), whereas the previous year’s study (intervention schools) many more schools used it the way we wanted them to. [ID 2, HPA staff interview]
In the four intervention schools included in the process evaluation observations, the DVD was shown to pupils in different environments. In two schools it was shown as part of whole school assemblies. In one school it was shown to separate KS1 and KS2 assemblies and combined with a demonstration of how to wash hands according to the DVD instructions, with input from pupils. At another school the DVD was shown to individual classes in combination with other intervention lessons. In these four schools, time spent on the ‘What are germs?’ lesson ranged from 8 minutes in a class where it was combined with the ‘Healthy hands, healthy school’ lesson, to 75 minutes in a KS2 class where the pupils were interested and engaged (Table
3). The lesson plan recommended all children take part in the glitter activity, but in one class a few pupils demonstrated while the rest watched.
It was for the whole class to get glittered wasn’t it? So I just scaled it down because we didn’t have much time. [ID 303, KS1 teacher interview]
Acceptability
HPA staff reported generally positive feedback from schools that had used the resource.
Usually if they had used the (resource) pack, it was positive… I think they found it useful and also the fact that the pack had the lesson already prepared for them, so it was something else that they didn’t have to worry about. [ID 2, HPA staff interview]
Many pupils and teachers enjoyed the DVD and thought it fun, although one pupil said it was boring.
It wasn’t just like strict ‘you have to wash your hands’ , it had like a kids cartoon so you could actually pay attention and catch like the kids’ side. [ID 303, upper KS2 focus group]
Pupil A: A couple of weeks ago we watched a DVD about hand washing and…
Pupil B: Hands up for Max.
Pupil C: It was about how to wash your hands and when you should wash your hands.
Pupil A: No it was boring.
Pupil C: I thought it was actually quite interesting.
Pupil D: Yeah I liked it.
Pupil C: Most of us didn’t know how to wash our hands really properly. [ID 317, lower KS2 focus group]
While some teachers thought the cartoon was appropriate for all ages, some thought it was too young and not relevant for older pupils.
The video was quite good they liked it, for our class. It was a bit too patronising for the older ones. They (younger pupils) thought it was quite funny, when they were watching it they were like “Oh, that’s the little man from the posters”, they realised it connected so they liked the video and they liked the musical bits. [ID 317, KS1 teacher interview]
I liked the DVD, I thought it was very simple but very effective… I think it worked well with both (age groups). [ID 303, KS2 teacher interview]
The children laughed at the DVD and perhaps it wasn’t quite so appropriate for us at KS2 as it would be for other age groups. [ID 317, KS2 teacher interview]
Some pupils found the worksheets difficult, whereas some teachers adapted the lesson plan when they felt the worksheets were inappropriate. For example some teachers thought the cartoon images of germs were either too juvenile for older pupils, or too unrealistic to provide accurate information to younger pupils.
Pupil A: I always thought we found it really difficult. I think it was the first lesson we had, do you remember?
Pupil B: The quiz.
Pupil C: This germ’s called… it’s spread by…
Pupil A: Yeah it was quite hard. [ID 317, upper KS2 focus group]
I felt that the worksheets that were there to back them up we hadn’t covered the things in the lessons, the things like the different types of viruses… so they couldn’t do them. And quite a lot of them when we had to design our own germ and they looked at some of the germs that were on the sheets and were saying “Germs haven’t got legs, this is silly” and I wondered how appropriate that was for upper KS2… because you know we do work on germs anyway… and we’re teaching them that germs are very microscopic… and it sort of contradicts the way we’ve done it. [ID 317, KS2 teacher interview]
I wouldn’t do the cartoons because the children, they take it literally at that age… then the children will just… presume that all germs have skateboards and skates and have funny glasses on so I think if you’re going to show the picture it would have to be the picture what it is. [ID 317, KS1 teacher interview]
Data from observation notes about the ‘Healthy hands, healthy school’ lesson indicated that children who volunteered to demonstrate hand washing appeared “disappointed when not chosen”, and that “those not demonstrating got a little distracted as time passed”. The majority of comments related to this lesson were positive.
Of course they love glitter so that is great and it is difficult to get off… so you do have to wash your hands to get the glitter off and I think that it was a really visual way for those kids to learn about washing their hands well it’s not just visual because it’s kinaesthetic as well… I think it worked great. It’s probably the best hand washing lesson I taught. [ID 104, KS1 teacher interview]
Data from observations indicated that pupils liked receiving stickers and one teacher thought the posters and stickers made the ‘Hands up for Max!’ program better than a previous hand hygiene program.
I think this has been stronger (than School Council programs) because there has been nice visual posters to look at and it’s that gimmick… stickers to go home, it does hit home a little bit more. [ID 303, KS1 teacher interview]
Sending the resource electronically may make it easier to disseminate and use.
I think it would be useful if it could be electronic because I think then as well for us it would be easier to send out because you’re not relying on sending it out through the post, and they’ve then got an electronic copy that’s not going to get dog-eared that they can then print out easier. [ID 7, HPA staff interview]
Sustainability
The first aspect of sustainability relates to the ongoing delivery of this educational resource in schools. Teachers thought the intervention could be incorporated into several areas of the curriculum and thereby taught on an ongoing basis.
Because we’ve still got the resources… I think we probably would voluntarily run it on a yearly basis and get involved in the PSHE (Personal, Social and Health Education). [ID 104, KS2 teacher interview]
I think it will be part of science because the topic we’re doing in science this term is healthy eating and healthy living so that I think will fit in well. [ID 317, KS1 teacher interview]
The second aspect of sustainability relates to how, and by whom, the educational resource is disseminated to schools. The HPA was seen to have a role in providing education and information related to hand washing, including via their website and their contact with schools, although some HPA staff did not see coordinating delivery of such a resource as part of their specific role.
We have infection control guidance that we send to schools, so we could also put in hand washing resources into that and certainly when we are dealing with schools in an outbreak situation, we can be guiding them to where they can find these sort of resources. [ID 1, HPA staff interview]
In terms of supporting public information and education, that is something that HPA do… I don’t think it has any relation to my day-to-day role at all. [ID 6, HPA staff interview]
The HPA may not have the capacity, alongside their existing competing priorities, to ensure that a large number of schools use the educational resource, and to be available for teachers during their limited non-teaching periods.
There is nobody else in this team, if the phone rings and it’s a school and they ask for me… all they can do is take the phone number of the person… you could have tried eight times and that was the only time they had managed to be able to get back to you because they are in and out of classes and that can be, not stressful, but there was no back-up here. [ID 2, HPA staff interview]
The length of time between schools agreeing to participate in the study and receiving the educational resource was crucial in terms of getting schools to successfully deliver the intervention. This is a likely contributor to the low proportion of control schools using the resource.
I just felt that the timing was quite crucial. So if you had a conversation in November, and you’re not phoning them until February, they are not going to remember, well it’s not going to be so keen and fresh in their mind. [ID 6, HPA staff interview]
To manage this issue it was suggested that the educational package be sent out in batches.
Instead of trying to do all the schools in one go… and get packs out to everybody within three days, and then have 100 phone calls to make within two weeks, I would probably do it in chunks so you would do ten schools one week and then start the next ten schools the week after… I think it would help getting back to the schools. [ID 2, HPA staff interview]
It was suggested that other agencies, for example healthy schools coordinators, health promotion nurses or education departments, who had more direct contact with schools, may be better placed to ensure delivery of the educational resource beyond the research setting.
I think only if it was maybe led more from an education department side, if it was put into the curriculum, I think that would… be a really good way to get it across to all the kids. Because our way we are doing it on a project basis, whereas if it was already part of what they have to do… I think that would be great. [ID 2, HPA staff interview]
I would have said within Healthy Schools type staff, you know those co-ordinators, but I know that they are not always around now, but somebody who is much more in touch with the school on a day-to-day or a month-to-month basis who has built up relationships with school members. [ID 5, University staff interview]
Discussion
The educational resource was acceptable to schools and delivered in most intervention schools (85%) within the trial but few control schools (38%) outside the trial setting. Delivery of the intervention was also more likely to occur in schools who participated in the sub-study, where additional student absence data were collected, and university staff had more opportunities for contact with these schools. Fidelity of delivery was also reportedly better in intervention schools than control schools in the trial. These results suggest that delivery of the intervention beyond the trial is unlikely to be sustainable using this model of a centralised, non-research agency to coordinate intervention delivery. Some variations in dose were also apparent and optional intervention activities that might have helped to reinforce hand washing messages, particularly at home, were rarely used. Similar variations in fidelity and dose have been observed in other studies of health promotion interventions in school settings. A review of drug abuse prevention programs taught by teachers found considerable variability in the number of key curriculum objectives covered or the number of modules taught [
4]. Teachers delivering a nutrition curriculum completed 70% of the lessons with a rating of 76% faithful to the curriculum [
27]. Reach in the Trial of Activity for Adolescent Girls was high, with 91% of girls in seventh-grade and 77% of girls in eighth grade taught all of the health education lessons [
6]. The level of fidelity was found to be acceptable in 76% and 64% of these lessons in the first and second year of the study, respectively [
6]. Whether the dose and fidelity obtained in the current study were sufficient to bring about the level of behaviour change required to reduce infection-related absence will only be able to be inferred from the main trial results. The reach obtained in intervention schools, however, compares favourably with a review of prevention and health promotion programs for children and adolescents where few studies documented implementation levels greater than 80% [
1]. Few programs are also able to achieve complete implementation in real-world settings [
28].
Qualitative data from this process evaluation provide useful insights for updating the educational resource if it were to be distributed more widely. Teachers generally found the intervention useful and straightforward to deliver. The element generating the most positive reaction was the glitter activity in the ‘Healthy hands, healthy school’ lesson, because of the effectiveness of the kinaesthetic aspect in demonstrating how germs spread, and the importance of hand washing. The glitter lesson may possibly be less effective if some children are passive observers rather than being actively involved. Teachers and pupils also enjoyed the DVD animation showing how to wash hands, although there was some suggestion this may be better suited to younger audiences. The stickers and posters were also well used and received. There was some concern that the ‘What are germs?’ lesson worksheets were too difficult, even for older pupils, as insufficient background information was provided for them to be able to answer the questions. In addition, the pictures of germs that could be coloured in by KS1 pupils were considered too childish because they were presented as caricatures rather than realistic, if somewhat magnified, pictures of germs.
While all four main elements of the educational resource (DVD, lesson plans, posters, and stickers) tended to be delivered to all pupils in the four intervention schools observed, the length of time spent on each lesson varied. It is likely that the depth and breadth of information covered, and therefore the understanding reached, may be quite different for a lesson lasting 10 minutes compared to one lasting over an hour. It is also possible that the rates and duration of lesson delivery noted in this part of the process evaluation may be a result, at least in part, of the classes being observed by a researcher. The trial was designed to be pragmatic, aiming for implementation to be as close to how it would be delivered in practice [
29]. Adaptations teachers make so that it is appropriate and relevant to their class may be acceptable [
5] and may not result in reductions of effectiveness [
30], although effectiveness of this educational intervention is yet to be established. Given differing views on resources and different abilities of pupils within year levels across schools, future versions of the resource could include a range of lesson plans and teachers could choose those appropriate for their pupils. This will require communication between program developers, who understand about the essential components of an intervention and its effects, and teachers, who understand pupils, pedagogy and school settings, so that prescription and adaptability can be combined for maximum effectiveness [
4].
In addition to updating the resource itself, consideration should be given to the dissemination process as HPA staff coordinating the intervention delivery had limited capacity to follow up and respond to schools. While the HPA are seen to have a role in providing hand washing information, the intervention may be more sustainable if it were embedded as part of the curriculum by education departments, and maximising effectiveness of staff such as healthy school coordinators.
A strength of this study is the use of both quantitative and qualitative methods to measure and explore intervention reach, dose, fidelity, acceptability, and sustainability. In addition, the process evaluation was conducted and analysed by a researcher not involved in the intervention delivery and before the outcome evaluation, so interpretation of key process factors likely to affect outcomes were not influenced by prior knowledge of the outcomes [
31]. A limitation is that direct observation of intervention delivery was only possible within the four intervention schools. We were also unable to elicit reasons why the intervention was not delivered among some schools, due at least in part to limitations on the capacity of HPA staff to ensure that schools were using the resource.
Hand washing is a relatively simple, inexpensive and important public health measure [
32]. Teaching primary school children to wash their hands properly and encouraging regular hand washing may not only reduce infection related absenteeism but also help to habitualise this behaviour at an early age. The ‘Hands Up for Max!’ intervention was a deliberately modest educational intervention, designed to be readily integrated into the existing school curriculum, and provided to schools by a public health body whose main contact with schools traditionally occurs when there is an infectious disease outbreak. Schools are an important setting for health promotion but, as this study demonstrates, simply providing carefully designed health educational resources may not be sufficient to ensure their use. The World Health Organization Health Promoting Schools framework [
33] also suggests that it is insufficient to merely introduce health into the curriculum but indicates that there also needs to be concomitant changes in the school environment and in the wider community. The ‘Hands up for Max!’ intervention had a homework element with the potential to effect change in the wider community through involving parents, but this element was optional and inconsistently used even within the trial. The intervention did not include any changes to the school environment, but these may be crucial in the case of hand washing [
17]. Changes to the school curriculum and environment, plus the wider community, are only likely to be achieved if health and education authorities work together on a regular and continuing basis. This integrated working can create a shared understanding that education can improve health, and that improving health in schools can also improve educational attainment [
34].
Competing interests
Rona Campbell is a Director of DECIPHer IMPACT a not-for-profit company wholly owned by the Universities of Bristol and Cardiff. The company licences use of evidence-based health promotion programmes and provides materials, training and quality assurance to support their delivery.
Authors’ contributions
CRC was involved in design of the process evaluation, and collected, analysed and interpreted data, and drafted and revised the manuscript. ALN and SB were involved in study design, data collection, interpretation of data and revising the manuscript. EY collected, analysed and interpreted data and revised the manuscript. RC conceptualised and designed the study, interpreted data and revised the manuscript. All authors read and approved the final manuscript.