The effect of school summer holidays on inequalities in children and young people’s mental health and cognitive ability in the UK using data from the millennium cohort study

Socio-economic differences in summer learning loss have been the subject of widespread concern among researchers, the public and policy makers since a landmark US study [1] found that a large fraction of total inequality in educational attainment between children from more and less affluent backgrounds was attributable to differences that emerge over the school summer holidays. The existence and seriousness of the problem appeared to be confirmed by a systematic review published in 1996 [2], but recent research has suggested that the apparent widening of inequalities is an artefact of the way children’s ability is measured and analysed [3, 4]. Reanalyses of previously published data and analyses of other datasets indicate a much smaller effect or no effect of summer learning loss on socio-economic inequalities in attainment [5‐7]. Since the review by Cooper et al. [2], analysis of socio-economic status and reading ability has not identified consistent evidence of inequalities between advantaged and disadvantaged students. While negative effects have been found for students with lower socio-economic status [8‐16], other studies found negative effects for higher socio-economic students [17, 18]. Many recent studies report conflicting results, such as different effects for different school grades, datasets, measurement scaling, or analysis models used [5‐7, 19‐22], or no difference between socio-economic groups [21, 23‐26].

Despite these recent findings, concerns persist over differential learning loss when schools are closed [27]. The National Summer Learning Association asserts on its website that the cumulative effect of summer learning loss is ‘a crisis in the making: by the fifth grade, summer learning loss can leave low-income students two-and-a-half to 3 years behind their peers.’ [28]. In the UK, concern over summer learning loss has been compounded by fears that many children experience ‘holiday hunger’ when schools close for the summer and stop providing free meals [29].

In the debates over whether and how long schools should be closed to help manage the pandemic, references to summer learning loss have been frequent. The finding from a rapid review of evidence by the Education Endowment Foundation [30] that inequalities could widen substantially if schools remained closed for 6 months was widely cited by senior policy makers and politicians in the UK [31‐33]. Against this backdrop of continuing debate about the effects of school closures on inequalities in attainment, and their contribution to managing the pandemic, we undertook the first analysis of UK-wide data on changes in socio-economic inequalities in children and young people’s cognitive ability and mental health over the school summer holidays.

The UK Millennium Cohort Study is a nationally representative, longitudinal survey of children born in the UK from September 2000 to January 2002 [34]. The survey uses a stratified clustered sampling design to oversample children living in Wales, Scotland and Northern Ireland, disadvantaged areas and, in England, areas with high proportions of ethnic minority groups. We used data gathered when the cohort members were aged 7 (n = 13,681), 11 (n = 13,112) and 14 (n = 11,564), with interviews spread over a year or more [34].

Since our aim is to investigate changes in mental health and verbal cognitive ability across the summer holidays, we restricted our analysis to cohort members surveyed in the 2 months (three in the case of Scotland) before and the 2 months after the summer holidays in each sweep (further detail is provided in the following section). Using the whole sample across the pre- and post-summer holiday months would have confounded the effects of the summer holiday with those of other holiday periods throughout the year, and with the effects of catch-up once cohort members return to school. In addition to the survey weights, we used stabilised inverse probability weighting to correct for the varying composition of the sample in the pre- and post-holiday months.

Table 1 shows the numbers of observations in the samples used in the regression analysis, broken down by period (pre- and post-summer holidays), UK country, survey Sweep and outcome variable. Further details of the weighting strategy and sample size broken down by calendar month are provided in the Additional file 1 Table A1a and b. Descriptive statistics for the outcome measures in the whole sample at each wave and a comparison of the covariate distributions in the unweighted and weighted samples are provided in Additional file 1 Table A2. Relevant flow charts were also drawn (Table A3a,b,c).

Our main findings in terms of odds ratios are presented in Table 2. Figure 1 depicts predictive margins using the margins command in Stata 17.0. Detailed results are presented in the Additional file 1, Table A5a, b, and c.

Table 2 presents the effect estimates for mother’s education (column 1) and the school summer holidays (column 2) on verbal cognitive ability and SDQ scores, adjusted for sex, ethnic and national differences in the composition of the pre- and post-holiday samples. The final column presents the coefficients of the interaction terms between school holidays and maternal education. The odds of borderline/abnormal total and prosocial SDQ scores were higher for cohort members with less educated mothers at each age. The odds of having a borderline/abnormal total difficulties score were higher among those surveyed after vs. before the school holidays at ages 7 and 14, but not at age 11. There were no substantial differences in pro-social behaviours among those surveyed before/after the school holidays. Verbal cognitive ability is lower for cohort members with less educated mothers at age 7, 11 and 14. At age 7, verbal cognitive ability scores were lower among those measured after the school holidays compared to those measured before. This was not the case at ages 11 or 14. Overall, the effects of maternal education on verbal cognitive ability and SDQ scores are greater than the effects of school summer holidays.

Figure 1 presents the socio-economic gradient (according to mother’s academic qualifications, X-axis) in verbal cognitive ability scores and risks of poor socio-emotional wellbeing (Y-axis), stratified by those surveyed before (solid line) and after (dashed line) the summer holidays.

As can be seen in Fig. 1 and Table 2 (Interaction column), the socio-economic gradient in total SDQ difficulties appeared to widen over the school holidays ages 7 [OR:1.4; 95% CI(0.6, 3.2)] and 14 [1.5(0.7, 3.2)], but the confidence intervals from the interaction term were wide. Inequalities in prosocial difficulties appear to narrow at age 7 [0.4(0.1, 1.5)] and slightly widen at age 14 [1.2(0.5, 3.5)] but again with wide confidence intervals. The socio-economic gradient in verbal cognitive ability does not widen over the school holidays. The negative coefficients from the interaction terms in Table 2 indicate a slight narrowing of inequalities at ages 11 [b:-0.7; 95%CI(− 4.3, 2.8)] and 14 [− 0.3 (− 10.4, 0.4)], but the changes were very small and the confidence intervals wide. Sensitivity analyses using non age standardised ability scores, (Additional file 1, Table A4) and income and neighbourhood deprivation in place of mother’s education (Additional file 1, Tables A6–7) found similar results.

This Millennium Cohort Study analysis is the first analysis of UK-wide data on the impacts of school summer holiday closures on inequalities in children and young people’s mental health and verbal cognitive ability. We observed marked inequalities in children and young people’s mental health and verbal cognitive ability according to maternal education and other measures of social position, but the evidence is mixed, with inconsistent results across different age groups. The increase in population-level SDQ total difficulties scores over the summer holidays was greater among disadvantaged groups, leading to a widening of inequality at ages 7 and 14, but not at age 11. Analysis suggests a relative decline in prosocial behaviour at age 14, while there was a relative improvement at age 7 and 11. We found younger children’s verbal cognitive ability declined by a small amount over the holidays regardless of socio-economic background. No decline was found in verbal cognitive ability over the holidays for other older age groups. We found no evidence that inequalities in verbal cognitive ability widened over the school summer holidays. As is often the case with interaction analyses, statistical power was limited, so the differences in the changes in mental health and verbal cognitive ability are imprecisely measured, and we cannot be confident that they reflect a real narrowing or widening of inequality.

These findings are consistent with the international evidence that has accumulated since the last systematic review was published in 1996. Several recent analyses of large USA datasets examining differences in cognitive ability after school summer closures between socio-economic groups have found different direction of results across school years [5‐7]. Recent studies have questioned some of the methods used in earlier research, emphasising that measuring inequalities in attainment is problematic, and that heterogeneity in reported findings reflects the diverse approaches taken to measurement and analysis [3, 4, 19]. A strength of our study is that this secondary analysis uses high quality nationally representative survey data, covering socio-emotional wellbeing as well as cognitive ability, among a cohort of children and young people who were measured at three separate ages covering both primary and secondary school-age. Importantly, we were able to compare pre- and post-holiday scores on the same measures within each age group, so our results should not be subject to some of the scaling problems that have affected earlier studies of summer learning loss [3, 4, 19].

A limitation of our study is that using the Millennium Cohort Study data from a restricted set of pre- and post-holiday months risks incorporating bias and we were unable to measure changes within cohort members. However, the rich socio-economic and demographic data and the sampling and attrition weights available within the Millennium Cohort Study allowed a weighting strategy which we believe has adequately compensated for this. As we only know the month in which each interview took place, rather than the exact day, some interviews at the end of each pre-holiday period and some at the end of the school holiday will be misclassified. Although we were able to compare pre- and post-holiday scores on the same measures within each age group, there remains a risk of under-estimating effects on inequalities due to floor or ceiling effects. The power of the study to detect small effects is constrained by the sample size, and with the numbers available we were unable to conduct analyses stratified according to levels of ability. Although our measures of cognitive ability were based on reading tests, the mental health measures were based on reports by parents. It is possible that reporting bias may have contributed to differences in scores given by those interviewed before and after the summer holidays. Moreover, SDQ ratings are based on the child’s behaviour over the previous 6 months so this might have biased our results; however, it is more likely parents would be replying based on the most recent behaviour of their child. We use outcome data collected between 2009 and 2016, so the results may not generalise to later periods if the experiences of children during school holidays have changed – for example, as a result of widening socio-economic inequalities or a rise in child poverty.

School closures were among the most widespread measures taken to control the spread of infection in the first wave of the COVID-19 pandemic, affecting 1.5 billion children in 190 countries in April 2020 [37], despite a lack of evidence from previous research [38], or support from modelling studies [39], to suggest that closing schools has a large effect on infection rates. Surveys conducted during the UK restrictions revealed wide differences in the nature and level of support for learning available to children from different backgrounds (for example, those attending state and private schools), and corresponding disparities in the ability of children from more or less affluent families to make use of the support provided [40]. Concerns over the impacts school closures on mental health, due to the reduction in pastoral care, and structure and routine, have contributed to calls for schools to prioritise mental wellbeing upon their return to school [41]. A shift in approach towards keeping schools open for as long as possible, and prioritising their reopening during the easing of restrictions, was evident during the second wave of the pandemic in the UK in Autumn and Winter 2020.

Our findings from the UK are consistent with other research evidence that suggests that adverse impacts do not automatically follow from the cessation of schooling over the summer. A large longitudinal study in England reported little change in pupil wellbeing across the first period of COVID-related school closures in Spring 2020 [42]. However, negative effects of school closures are being reported as further research emerges. For example, a USA study found pandemic related negative impacts on young people’s mental health [43], and learning and behaviour problems were found in a study of Italian school children [44]. These studies and a number of other surveys [45] conducted during the pandemic have reported widely differing experiences of schooling during the period when schools were closed, which may mean that pandemic related closures have a larger effect on inequalities in learning than we observe following routine summer closures when most children have a break from formal education. Evidence has also begun to emerge of attainment losses as a result of the lengthy closures in Spring and Summer 2020 [46, 47]. Future research should focus on establishing whether those early indications of adverse effects are real, what they mean for inequalities in health and educational attainment, and what are the mechanisms that cause them, so that interventions can be appropriately designed and targeted, in order to mitigate the adverse effects of future extended closures.

The results of our analyses of UK children are consistent with the mixed picture emerging from the international literature. They suggest that in normal circumstances, school summer holidays do not lead to significant additional educational disadvantage. School closures during the COVID19 pandemic may have a larger effect, because they last longer, because inequalities in support for learning have been more pronounced during the pandemic-related closures than they are during summer holidays in general, or because disadvantaged children tend to live in households more severely affected by the social and economic disruption of the pandemic. The possibility of such effects should be an important focus of future research and monitoring.