Replication of a whole school ethos-changing intervention: different context, similar effects, additional insights

Overall, the results from CORE were similar to those of the Gatehouse Project, that is, with no significant change in students’ depressive symptoms, but change in a cluster of risk behaviours notably, drinking, unprotected sex and poor health among girls. Two aspects of the findings were different. Gatehouse showed no change on connection to school (the purported intermediary variable). By contrast we found a change in connection to school for girls, but not boys. Gatehouse showed an impact on risk behaviours, with no gender differences (L. Bond personal communication) whereas, we only showed significant changes in the health and risk behaviours in girls. Linked longitudinal data would have allowed us to test changes in school engagement at the individual level (i.e., to determine if those students who increased their engagement to school also decreased the risk behaviours). We chose instead to reassure students by using no ID codes, to confirm anonymity and potentially maximise honesty of responses. To investigate one possible mechanism of action, we turned to social network analysis to get a better understanding of the dynamics of change. The network analysis revealed that staff and teachers became significantly more interconnected.

The Gatehouse Project was a cluster randomised control trial. By contrast, this before-and-after study in a single school is a weaker design, lacking a comparison group and limited in its capacity for causal inference. The results are also vulnerable to the possibility that students may have reported their behaviour more positively after 2 years as a consequence of perceived social pressure to do so, particularly as CORE was so prominent in the school. But this interpretation is unlikely. First, social desirability bias, if it were present would likely have had more students describing their connection to school as strong, as this was the obvious focus of CORE. If social desirability were present it appears to only operate through girls. It is unclear why girls would feel more compelled than boys to offer socially desirable responses. Second, students were unlikely to be sensitive to the risk behaviours of interest because prevalence data from baseline, was never fed back (the school only received data about connection to school and feeling safe) and the risk factors of interest were ‘buried’ in an omnibus survey that also asked about things like physical activity, employment and connection to neighbourhood which did not alter in response to CORE.

Further, both the original Gatehouse Project, and its replication (CORE), made no mention of drinking, smoking, unsafe sex or drug use. The intervention was entirely and exclusively focused on making the school a better place for everyone. This is an important point. As a consequence of it being misunderstood, the Gatehouse Project results have been recently criticised [55]. That is, the charge has been made that decreases in substance use were the consequence of social desirability bias, because program elements aimed to alert students to such risk factors and were the focus of the curriculum [55]. This is not true. Risk factors were not part of the curriculum or a focus at all. Further, this replication case study illustrates a similar pattern in a different context (older students, different school system) as a result of the same strategy. That is, a holistic intervention to enrich the social environment, with students and staff as active agents in addressing problems appears to be risk protective, with no attention paid to the risk behaviours themselves.

However, the view of what happened within the CORE intervention remains partial. Schools that have undertaken similar projects produce qualitative accounts of changes in self-regard and pride, engagement and aspiration, and a sense of security and support [16] as well as changes in the culture, feel and everyday language of the school [56]. This study shows that social network analysis methods may also be a promising means to capture the dynamic of change processes in schools with staff and potentially in future with students also. It is a different approach to social environment measurement than those that involve self-appraisal of a person’s engagement, commitment or belonging [57]. We found statistically significant increases in the density of all five social relationships among staff. The changes in the social network structures paint a picture of staff becoming more cohesive and collaborative. This type of social environmental change with staff has not been observed with the social network analysis in whole school interventions previously, to our knowledge. Whether this pattern also occurred in the Gatehouse Project is unknown. In this study we can speculate that a more connected staff may have been more capable, engaged and responsive to student concerns. The social network data have no comparison group. So we do not know the extent to which increases in densities across time are natural, versus attributable to the change processes under study. We also cannot discount the possibility that staff gave answers that they felt would be socially acceptable. On the other hand, if these scenarios were true we would expect higher densities after a year of intervention and less discrimination among the relationships measured. In other words, inflated answers would have been applied in all categories if staff felt obliged to say they were mixing or interacting more, rather than the pattern that was observed, where some network densities (such as socialising with) showed a small increase but remained low across two years.

Much of the change in the social networks was attributed to the decision taken to abandon the usual practice of ‘retreat style’ full day professional development sessions for staff, in favour of weekly one hour sessions throughout the year. In this sense, the objective (to increase skills in teaching for emotional literacy) achieved both an educational and structural purpose. It created a weekly “check in” procedure across the school that kept the goal of whole school change at the forefront of people’s minds and a new way of working together. It was also where the CORE facilitator implemented strict rules about mixing people together. This may have been critical to the change process. The interviews with key informants undertaken for process monitoring purposes (data not reported here) placed high value on the creation of this new professional space enabling staff to work together for school improvement, noting that some ancilliary or support staff were motivated to attend the after school sessions even though they were not paid to come or compensated with time off. Members of the CORE school action team told us that the mixing of staff possibly transferred skills from staff who were comfortable mixing with students to those who were not. CORE encouraged staff to show that they cared, even just in small ways, such as noticing students in the hallway, smiling, and saying hello. Staff also confessed that they were embarrassed by the extent to which (until CORE) staff did not know each other.

Depressive symptoms in students was a key variable of interest in the Gatehouse project design and in this replication. Overall, there was little change in depressive symptoms over a 2 year time frame. Prevalence for girls dropped slightly from 17% to 15% (not statistically significant) and boys remained at 11%. The Gatehouse Project also showed no reduction in depressive symptoms as have other similar studies [58]. The reasons why these interventions have not shown an impact on depression are unclear. It may be that depression is more intractable than other health risk behaviours; that depressive symptoms remain but response to these feelings changes (e.g. do not self-medicate via drug use) and/or there are incompletely understood causal pathways [20]. That is, while alienation from school predicts risk behaviours and depressive symptoms, action to promote engagement reduces risk behaviours but not depression. Universal classroom-based methods to prevent depression have shown small but consistent effects [59]. It should be noted that CORE contained no formal social emotional learning curriculum as such. A review of whole school ethos-changing interventions found no evidence for effectiveness on depression, noting that it may be difficult to make the relevant environmental changes in complex organisations and to measure them effectively [60]. So while specific curriculum approaches produce small effects for both depression (and risk behaviours), the large effects for risk behaviours from whole school approaches still provide encouragement to see if similar gains could be made for depression, given that whole school approaches do not compete for class time and have the potential to deliver multiple benefits [61]. It may also be that whole school approaches affect emotional well being, rather than depression, and so effects are not being adequately captured.

The differential impact of CORE on risk behaviours according to gender is interesting. No gender effects have been reported to date by a similar Gatehouse replication program of work in the UK by Bonell and colleagues [16,62]. However, in the USA a whole school ethos building intervention combined with a curriculum targeted at prosocial behaviours and emotional literacy in inner-city African-American schools was more successful than a curriculum-only approach with boys’ risk behaviours only. Those authors argued that the effect in girls may have been missed (a measurement issue in not capturing the types of aggression that are more prominent in girls) [63]. The other explanation offered was that in very hostile contexts girls maintain aggressive behaviours to present a tough image to protect themselves [63]. Other researchers have also reported differential effects by gender for interventions targeting risk behaviours [64]. We searched the journal kept by the CORE facilitator, the key informant interview records and all other documentary records of the intervention’s unfolding and could find no evidence that CORE was targeted preferentially to girls or that girls responded or engaged differently in activities. On the other hand, we did not structure reporting on boy/girl involvement formally into our procedures and we would recommend that this be done more systematically in future. Future social network analytic methods among students could investigate differential impacts on boys and girls.

Regression to the mean is a possible explanation for the change in girls’ risk behaviours in our study. Concern about sexually transmitted disease rates prompted contact with the research team, and the largest risk reductions were seen in unprotected sex and poor self-rated health among girls. However, comparison of our data with a nationally representative prevalence survey of Canadian adolescents suggests that this was not the driver of the effects we observed. Nationally, the prevalence of unprotected sex for Grade 10 girls in Canada remained constant between 2003 and 2005 at 31% [37]. The prevalence of unprotected sex in our sample was 18% in 2003, lower than the national average, and it moved further from the national trend to only 8% in 2005. School connection for girls also improved during CORE. (In contrast, the national survey indicates drinking alcohol reduced in Canada from 2002 to 2006 for both boys and girls [37], and this fitted the trend in our own data).

This reduction in risk behaviours in the town’s only high school occurred at a time when social turbulence in the town itself (associated with the expansion of the meat packing industry) saw crime rates increase and case loads of child and family service agencies double [32]. Note again that the CORE intervention made no mention of sexual health in any program materials, the way the intervention was talked about, or in the hiring and briefings of project team members. We also found no found evidence of any alternative concurrent intervention or explanation for the CORE effects from project records. This would seem to support the conclusion that something protective happened at the high school during this time, which impacted girls at least, and unprotected sex may have been more sensitive and more responsive to a whole school ethos-focused intervention in this school. While the results of the project were fed back to the school, we regret that we did not take the opportunity to formally research and document the local interpretations and explanations staff and students might give for the effects observed. We suggest that this occur in future repetitions of the intervention.

Ever since the early attention drawn to this field by Rutter [65], researchers have been alert to the special role of school context seen in “school level effects”, meaning differences in risk behaviours and student outcomes that are attributable to the school, rather than to the compositional characteristics of the students themselves [66]. In spite of this, a large portion of school level preventive programs continue to focus on specific risk behaviours, transferring identical risk factor reduction curricular from one school to another [67] paying scant attention to differences in context that might call for a different response. Even researchers who have advanced understanding of the clustering of adolescent risk behaviours and the creation of normative and maladaptive trajectories over time, appear to abandon their own ecological thinking when it comes to prevention, by urgently calling for ‘health education’ programs to address risks [68]. Alternative interventions, of the type described here are not well known.

This replication study and work by others [16,62] represents an intervention research agenda which mirrors the epidemiological research which suggests that we may have greater gains in prevention in the future by understanding how risks cluster, rather than by pursuing pathways that attempt to theorise alcohol, drug use, sexual behaviour, mental health separately [68,69]. This intervention research agenda seeks to enhance protective factors at school and reduce the general susceptibility to health risk [70,71]. It seems likely, therefore, that depending on the nature of the context, different risk factors will be more or less amenable to reduction. These may depend on the different compositional characteristics of the students, the characteristics of the schools, the number, initial prevalence and distribution of the risk behaviours. A further contributing factor may be the interconnectivity of the staff, and potentially the students. Experimenting with different risk behaviours is common in adolescence and considered one way in which adolescents try to establish autonomy and independence [72,73]. The extent to which a young person is able to successfully ‘slough off’ a risk behaviour they have simply ‘tried on’ may be affected by the availability of approachable adults at school and ease of contact with a variety of role models and ways of thinking/behaving. It may not simply be that adolescents need at least one connection with a supportive adult at school [74]. The adults themselves may need to be interconnected, valued and mutually supported. This thinking is consistent with an intervention being an event in a (complex) system, the effect and sustainability of which depends on how the dynamic properties of the system are harnessed [75]. These complex interdependencies are something which future studies should try to assess. Replication studies (sometimes derided as “me too!” studies) may therefore prove vital in building a stronger understanding of how context drives health outcome, and how context can be acted upon to change health outcome. This will enhance the field’s capacity to move beyond theories about individual behaviour and cognition to more fully understand and theorise schools as social environments [76].