Background
Methods
Review methods
Data extraction and quality appraisal
Process evaluation data synthesis
Results
Search results
Intervention name | Description of intervention | Location | Targeted grade of participants | Process data collected on | Report |
---|---|---|---|---|---|
Reading Writing, Respect and Resolution (4Rs) | A literacy-based social-emotional learning curriculum for elementary school students. There are two components: (1) a seven-unit, 21–35 lesson literacy-based curriculum in conflict resolution and social-emotional learning for children in primary school (to grade five); and (2) intensive professional development for teachers. | USA | Kindergarten to grade 5 | Fidelity and acceptability | Sung [39] |
DRACON | This intervention uses drama to develop cognitive understanding of conflict and bullying and to empower students to manage their own conflict, both personally and within the broader school community. | Australia | Primary and secondary school students | Implementation, mechanisms of change, acceptability and context | O’Toole [37] |
English classes (no name) | Teachers were trained and, working in pairs in the summer, they developed integrated health/English material, with a specific emphasis on the prevention of drug and alcohol use. | USA | Grades 8 and 9 | Fidelity, acceptability, quality and mechanisms of change | Holcomb and Denk [43] |
Hashish and Marijuana | The goal of the curriculum is to develop scientific knowledge of hashish and marijuana and to strengthen students’ problem-solving and decision-making skills through both didactic and participatory learning approaches. | Israel | Upper secondary school | Implementation | Zoller and Weiss [40] |
Infused-Life Skills Training (I-LST) | A substance abuse prevention and competency curriculum that focuses on social and psychological protective factors affecting substance use. It is integrated into the existing subject curriculum by the classroom teachers. | USA | Middle/junior high school | Fidelity, quality, dose and acceptability | Bechtel et al. [42] |
Kids, Adults Together (KAT) | The intervention aims to reduce drinking and antisocial behaviours in young people through a classroom curriculum, a parent evening and follow-up family activities. | UK | Grades 5 and 6 | Acceptability and satisfaction | Rothwell and Segrott [33] |
Fidelity, reach and mechanisms of change | Segrott et al. [44] | ||||
Peaceful Panels | Throughout art classes, students participated in anti-bullying lessons (from the Second Step programme for eighth grade students on empathy and communication in handling a grievance) and comic-making lessons. They then prepared artwork to demonstrate their understanding of how to resolve conflict. | USA | Grades 8 and 9 | Acceptability and satisfaction | Wales [45] |
Positive Action | Positive Action is a social-emotional and character development intervention aimed at encouraging positive behaviours through positive thoughts and actions. Lessons cover six units: self-concept; positive actions for mind and body; positive social-emotional actions; managing oneself; being honest with oneself; and continually improving oneself. | USA | Kindergarten to grade 12 | Coverage and acceptability | |
Acceptability and satisfaction | Beets [32] | ||||
USA | Implementation, fidelity, dosage and quality | Malloy et al. [41] | |||
Promoting Alternative Thinking Strategies (PATHS) | An intervention to reduce conflict by improving students’ social-emotional and thinking skills through a curriculum, the establishment of a positive classroom environment and generalised positive social norms throughout the school environment. | USA | Kindergarten to grade 5 | Quality, coverage (dose) and context | Ransford et al. [38] |
Roots of Empathy | An intervention that brings a visiting baby and their parent into a classroom as a springboard for learning empathy. Students learn messages of social inclusion, respect, how to build consensus, how to contribute to a safe and caring classroom and develop emotional literacy. | Australia | Grades 1–9 | Implementation, mechanisms of change and acceptability | Cain and Carnellor [34] |
Canada | Implementation and context | Hanson [30] | |||
UK | |||||
Steps to Respect | This is an anti-bullying intervention with both school-wide and classroom components. The School-wide components create new disciplinary policies for bullying and improve monitoring of and intervention in bullying. Classroom curricula positive social norms and improve social–emotional skills for better engagement with bullying. | USA | Grades 3–6 | Fidelity, context and acceptability | Low et al. [36] |
The Gatehouse Project | Through teaching a curriculum and establishing a school-wide adolescent health team, Gatehouse aims to: build a sense of security and trust in students; enhance skills and opportunities for good communication; and build a sense of positive regard through participation in school life. | Australia | Grade 8 | Coverage, quality and mechanisms of change | Bond et al. [35] |
Characteristics and quality of process evaluations
Intervention name | Site | Methods included steps to minimise bias in | Findings | Overall rating | |||||
---|---|---|---|---|---|---|---|---|---|
Sampling methods | Data collection | Data analysis | Supported by data | Have breadth and depth | Privilege young people’s perspectives | Overall reliability and trustworthiness | Overall usefulness answering our research questions | ||
4Rs (Reading Writing, Respect and Resolution) | New York, USA [39] | No Purposeful sampling but of only high-performing classrooms | Yes Different instruments piloted and used; findings triangulated | Yes Author verified data through ‘reflexive conversations’ and member-checking | Yes Clear results followed methods | Yes Very comprehensive data collected from a number of classrooms | No | Medium Selection of only high performers limits transferability of findings | High Detailed information about implementation provided |
DRACON | Brisbane, Queensland and New South Wales, Australia [37] | No No detail provided | No No detail provided on increasing rigour | No No details were provided | No No quotations present to support qualitative data and no links to questionnaire data | No Good breadth of findings, but limited depth | Yes | Low Limited data on methods and links to results | Low Limited detail on implementation |
English classes (no name) | Houston, USA [43] | No No detail provided | No No detail provided | No No detail provided | Yes Survey results followed clearly; qualitative results presented without supporting quotations | Yes Mixed methods enabled exploration of both breadth and depth | No | Low Limited detail on the rigour of methods used | Medium One of the few studies in which integration was core to the study’s design and some good detail around implementation is provided |
Hashish and Marijuana | Haifa, Israel [40] | No No detail provided | No No detail provided | No No detail provided | No Scant data were provided, and it was unclear how these were produced | No Minimal findings reported | Yes | Low Poor reporting of methods and minimal results | Low Lack of detail in findings restricted the use of this study |
Infused-Life Skills Training | PA, USA [42] | No No detail provided | Yes Multiple methods and instruments used; findings triangulated | No No detail provided | No No primary data provided, only authors’ accounts of the data | Yes Different aspects of implementation explored from students, teachers and administrators | No | Low Poor reporting of methods and minimal results | Medium Paper provides interesting insights and is the only one to compare with non-integrated curriculum implementation, but detail on methods is lacking |
Kids, Adults Together (KAT) | Southeast Wales, UK [33] | No No detail provided | Yes Multiple methods used at different data points to ensure comprehensive perspectives | Yes Comparative coding used to refine analytical framework | Yes Clear results followed methods | No Good depth around acceptability, limited detail on other aspects of implementation | Yes | Medium Insufficient detail to determine possible bias introduced in sampling, but data collection and analysis seem appropriate | Low Nothing about the integration of academic and health curricula in findings |
Southeast Wales, UK [44] | No No detail provided | Yes Comprehensive qualitative data was collected | Yes Data were triangulated; constant comparison of data was done; and authors increased validity of instruments | Yes Although actual quotations and results from process evaluation were limited | Yes Data were collected on many aspects of implementation | No | Medium A lack of data on methods makes reliability impossible to ascertain | Medium This study has interesting findings but would be better to see them grounded in primary data | |
Peaceful Panels | Athens, USA [45] | No Convenience sample drawn from the author’s classroom | Yes A range of methods used to collect data and an independent peer audited the author’s methods | Yes Author employed reflexivity, debriefs with peers, and member-checking to increase robustness | Yes Clear results followed methods | Yes Considerable detail on a number of implementation factors reported | Yes | Medium Convenience sampling and (opinion of the study team) less-robust than possible analyses may limit trustworthiness | Medium Detailed information about implementation processes, but limited information about influencing factors |
Positive Action | Hawaii [31] | Yes Sampling of schools was random, and there was an attempt to reach a census of all participating students | Yes Validated tools that collected data around a variety of measures of implantation were used | Yes Data were analysed using statistically appropriate methods | Yes Clear results followed methods | Yes Various features of implementation were explored in detail. These were generated from a large sample of diverse students. No qualitative data, however | Yes | High Methods were appropriate, efforts were made to increase rigour and the findings and interpretations lead clearly from the methods used | High This paper gives good information about important aspects of implementation |
Hawaii [32] | No Census of teachers attempted without success and no explanation provided | Yes High reliability of tools used | Yes Analysis were appropriate, and data were entirely quantitative | Yes Clear results followed methods | Yes Good range of process measures covered in considerable depth | No | High Study was well-conducted and statistically robust | High Useful discussion of key implementation factors including the perspectives of implementers | |
Chicago [41] | Yes Relevant sampling criteria used with a very high response rate | Yes Multiple data sources used and triangulated | Yes Analytical approach was appropriate and robust | Yes Clear results followed methods | Yes Multiple data sources provided information about many aspects of implementation, in detail, with description of relationships between these | No | Medium Although methods were robust, there was no qualitative data to answer the more useful ‘why’ questions, particularly behind the relationships between implementation factors | High Good data provided around multiple aspects of implementation | |
Promoting Alternative Thinking Strategies | PA, USA [38] | Yes Sufficient detail provided; high (85%) response rate | Yes Alpha-reliability coefficients acceptable and provided; other measures of validity lacking | Yes Data analyses were appropriate | Yes Clear results follow methods | No Study limited to teachers’ psychological factors | No | High This is a well-conducted study | Medium Although methodologically sound, comprehensive results are lacking |
Roots of Empathy | Western Australia [34] | No No detail provided | No No detail provided | No No detail provided | Yes Clear results followed methods | No Breadth around implementation from a teacher perspective, but little depth | No | Low A lack of methodological detail make trustworthiness questionable | Medium Useful data on some aspects of implementation provided, but lacking methodological rigour |
Western Canada and the Isle of Man, UK [30] | Yes Participants were from an ongoing RCT | Yes Reliability of instruments was good | Yes Data were merged to increase study power | Yes Constructs were well-defined and studied. Slight bias to Canadian results | No Lack of qualitative data | No | High This is a methodologically rigorous study | Medium Focus on teacher characteristics and implementation is valuable, but qualitative findings are limited | |
Steps to Respect | CA, USA [36] | Yes Participants are from an ongoing RCT; high response rate | Yes Questionnaire had high face validity and reliability | Yes Data analysis were appropriate | Yes Although qualitative exploration was lacking | No Concepts explored were limited | No | High This was a methodologically sound study | Medium Useful data, but qualitative findings are limited |
The Gatehouse Project | Victoria, Australia [35] | No No detail provided on how participants were selected | Yes Multiple methods used to collect data at multiple points in the year | No No detail provided | Yes Clear results followed methods | Yes Multiple aspects of implementation were explored from multiple stakeholder perspectives | No | Medium More detail on methodological rigour would be required to make a fair assessment of robustness | High Very useful data provided around implementation characteristics |
Thematic synthesis of process evaluations
Support from senior school staff
Ongoing practical support from leadership has been acknowledged as important for mainstreaming the promotion of emotional well-being through promoting greater connections between learning, classroom practices, and student well-being. [One teacher recalls], ‘The support of my principal has to come number one…getting the time on the timetable, setting up a team, [that] can’t happen unless you’ve got someone in administration that thinks it’s a great idea.’ [35], p.378
school leadership should develop a culture that encourages a shared and collective vision among staff and administration, is supportive of new innovations, and is aligned with the core values and concepts a given program is promoting … Perceptions of school climate were directly related to the beliefs teachers held about prevention/[social-emotional learning] and the attitudes teachers had towards [Positive Action]. [34], pp. 272–73
Because of the financial support of [the Department of Education and Training] and its coordination of training, the program was successfully implemented. It is essential, however, that there is a strong policy and resourcing commitment to effectively sustain [the program]. [34], p.68
Teachers’ immediate working environment
the first year participating teachers were especially helpful with recruiting and supporting new teachers in the program … they informally shared their experience with their fellow teachers, increasing interest and awareness of the program. They also gave examples of their lesson plans and discussed the importance of coaching and behavioral rehearsal in helping students master the life skills. [42], p. 224
teachers reported that the training was critical in adequately preparing them to integrate [life skills] components into their curriculum. They indicated that the training was especially effective in their development and implementation of infused lessons, and that the step-by-step process and manual were valuable in guiding the development of their lesson plans. [42], p. 224
research [to implement integrated curricula] consumed additional time in the teachers’ already busy schedules and required teachers to ‘learn’ some of the materials before presenting them to their students. Teachers’ lack of time or access to information, in some cases, may have limited the amount of health content applied to individual lessons. Thus, interdisciplinary lessons sometimes were not as detailed as they could have been. [43], p. S-39.
teachers who perceived their school administration as more supportive reported higher implementation quality, and positive perceptions of training and coaching were associated with higher levels of implementation dosage and quality. Teachers who reported the highest levels of burnout and the most negative perceptions of curriculum supports reported the lowest levels of implementation dosage and quality. [38], p. 510
Teacher attitudes towards intervention characteristics
teacher beliefs regarding their responsibility to teach [social-emotional learning] concepts were significantly…related to their attitudes towards Positive Action…[which] were positively related to the amount of the Positive Action curriculum delivered…and the amount of the curriculum delivered was positively related to material utilization in both the classroom…and school-wide. [32], p. 217
all participants were committed to the importance of [social and emotional learning] in their teaching…they considered [it] essential to the academic learning that underpinned the teaching philosophy of all participants. The pedagogical understandings in the…program were consistent with each participant’s philosophy of learning and teaching. [34], p. 63
A few [teachers] start with stronger reservations or resistance [to the programme], and some of these have chosen to withdraw from the project. These reservations are usually expressed as: not trusting drama to achieve its purpose, sometimes because it is perceived to potentially disrupt an orderly classroom, or to be too time-consuming in a full syllabus. [37], p. 279
teachers reported that the program’s greatest strengths were its flexibility, its infusion of new material into their classrooms, and its interest to students … Autonomy allowed by the program was a significant strength noted by all the teachers, not only for the convenience it provided, but for the respect it displayed for their professionalism. [43], p. S-39
Student attitudes towards intervention characteristics
although the students were not unanimous in positive feelings about the program, the great majority of them stated that they enjoyed it and that they felt that it helped them understand violence prevention … The students’ positive feelings implied that students enjoyed learning through comics and it is possible that this was this helped them retain what they learned. [45], p. 143
students responded with interest and enthusiasm to the infused approach, liked the integration of substance abuse prevention into other subject areas, and were more engaged and eager to participate in class. Moreover, their students especially liked the facilitative classroom environment and the hands-on approaches of behavioral rehearsal and role playing. [42], p. 224
Parental support
[an implementing teacher] … viewed inconsistency between the way students are taught at school and at home as an impediment. For example, whereas she taught children to ‘talk things out’ without using violence in a conflict, some parents encouraged their children to use violence as means of solving social conflicts at school. [39], p. 100