The Impact of Implementation Fidelity of a School-Based Multi-Component Smoking Prevention Intervention on Vocational Students’ Smoking Behavior: A Cluster-Randomized Controlled Trial

The Danish Vocational Education and Training system (vocational schools) educates students for skilled professions through programs of varying duration, consisting of a school-based basic part and a main part that shifts between apprenticeships and school periods. The basic program is divided into two parts; the first part is mandatory for the youngest students who completed lower secondary school within the last two years upon enrolment, and all other students can begin at the second part. For this reason, students’ age spans from 15 to 65 (Ringgaard et al., 2020). Preparatory Basic Education (preparatory schools) is intended for young people aged 15 to 25 years old who are not in education or employment and need professional, social, or personal support and qualifications to gain employment opportunities or proceed to upper secondary education such as vocational school.

The intervention was evaluated in a cluster-randomized controlled trial and conducted over two periods of a school semester, August to December, in 2018 and 2019. Randomization was stratified by school types: social and healthcare vocational schools, technical and commercial vocational schools, and preparatory schools. Eight schools were allocated to the intervention arm and six to the control group (continuing with usual practice). Questionnaire data were collected among students, teachers, and principals at all schools at baseline and follow-up after 4–5 months. Student questionnaires addressed, e.g., background characteristics such as sociodemographic factors, and smoking-related questions (more details under outcome variables and statistical analyses). Additionally for intervention schools, student, teacher, and principal follow-up questionnaires contained questions about the implementation of the intervention components as part of the process evaluation (presented in detail under exposure variables). One preparatory school in the intervention group partially withdrew from the study after randomization due to organizational changes and heavy workload (Jensen et al., 2023). This school received reduced versions of the implementation questionnaires and was therefore not included in the present study. Details on the study design, recruitment, and randomization procedures are published elsewhere (Jakobsen et al., 2021; Kjeld et al., 2023).

The “Focus” intervention was developed in 2017–2018 based on the stages and steps of the Behavior Change Wheel (Michie et al., 2011) including the Capability, Opportunity, Motivation, and Behavior model and Theoretical Domains Framework (Andersen et al., 2023). The intervention entailed six components: (1) the introduction and enforcement of smoke free school hours, a comprehensive STP stating that students, staff, and visitors are not allowed to smoke during the school day on or off school premises; (2) a course organized by the Danish Cancer Society for two to four staff members in short motivational counselling about smoking; (3) an “edutainment" event at the beginning of the school year delivered by a professional actor with a combination of information and entertainment to inform about the harms of smoking and challenge common misperceptions; (4) a class-based educational curriculum of eight sessions to challenge attitudes and beliefs, stimulate reflection, and support social activities not related to smoking; (5) a class-based “quit-and-win” competition to increase students’ mutual support and motivation to abstain from or quit smoking including a prize for the winning class; and (6) access to telephone-based support and counselling for smoking cessation or how to handle a smoke-free school day for all students and staff provided by The National Quitline. Detailed descriptions of each intervention component are published elsewhere along with a description of the intervention development (Andersen et al., 2023).

Two binary outcome variables were calculated from smoking status (“Do you smoke cigarettes?”) measured in questionnaires at baseline and follow-up: daily smoking (“Yes, everyday” vs. “Yes, at least once a week”, “Yes, more seldom than every week”, or “No, I do not smoke at present”) and regular smoking (“Yes, everyday” or “Yes, at least once a week” vs. “Yes, more seldom than every week” or “No, I do not smoke at present”). We included a third outcome variable, smoking during school hours, calculated from follow-up questionnaires (“How often do you smoke during the school day (with others or alone)?”): “Every day” or “Several days a week” vs. “Rarer” or “Never”.

We included students from the intervention (n = 656) and control group (n = 596) who responded to the follow-up questionnaire. We excluded 73 students with insufficient implementation data, 66 students with missing data on sex or age, and 1 student with missing data on all smoking outcomes, resulting in a final study population of n = 1112 students. Descriptive statistics and multilevel logistic regression models were performed in SAS 9.4, and we present the results as odds ratios (OR) with 95% confidence interval (CI) and p values. P values less than 0.05 are considered statistically significant. To address objective 1, we used 2-level logistic regression models with the GLIMMIX procedure (students nested within schools). We adjusted the models for self-reported sex (male or female), age calculated from the date of birth, socioeconomic status (SES) measured as student-reported family occupational social class, school type (social and healthcare, technical or commercial, and preparatory), and the outcome variable at baseline. The models with smoking during school hours as the outcome were adjusted for baseline regular smoking. We included school as random effects to account for clustering. As the cut-points for low, medium, and high implementation based on tertiles were arbitrary, we supplemented the models with graphic representations of fully adjusted restricted cubic spline regression models performed in RStudio (R Core Team, 2023) to be able to analyze all data points of the continuous variable and their relations to the outcome variables (Gauthier et al., 2020). To address objective 2, we stratified the logistic regression models by school type (social and healthcare students vs. technical, commercial, and preparatory school students). To address objective 3, we used the tertiles-based categorical variables of student-level implementation of the school tobacco policy, educational curriculum, and class competition components as exposures in multilevel logistic regression models. We only tested these three components separately because we did not expect the edutainment session to have an independent effect on the smoking outcomes as it was a one-time event at the beginning of the school year to support and raise awareness of the tobacco policy. Further, we did not have individual-level implementation measurements of the staff course, and only very few students reported use of the smoking cessation support offer. We performed additional analyses to decompose the student-level implementation construct into school-level fidelity (of the total intervention) and student responsiveness (of the smoke-free school hours component) and analyzed these variables’ relationships with the smoking outcomes. Finally, we compared the sociodemographic characteristics of the study population with the excluded students to assess potential selection bias.

Most students in the present study were from technical or commercial (48%) or social and healthcare (37%) vocational schools, while 15% were preparatory students (Table 1). The median age was 17 years, and most students had a middle-level SES of social class III to IV. Twenty-one percent of the students smoked daily at baseline, and 9% smoked weekly or less than weekly. Slightly more students in the intervention group smoked daily (25%) than in the control group (18%).

The student-level implementation scores were divided into tertiles: 32% of students had a low implementation level (0–23.3%), 33% of students had a medium implementation level (23.5–46.0%), and 35% of students had a high implementation level (46.5–100%) (Table 2). High implementation of the total intervention was associated with lower odds of regular smoking (OR 0.37, 95% CI 0.18–0.78), but not associated with daily smoking and smoking during school hours though the estimates were in the same direction (Table 2). The spline regression analyses (Fig. 1), however, showed that implementation levels between 60 and 80% were significantly associated with lower odds of daily smoking though insignificant at levels between 80 and 100% (Fig. 1b). Moreover, students with implementation levels of 60% or higher had significantly lower odds of regular smoking and smoking during school hours (Fig. 1c, d). Students with medium implementation had increased odds of all outcomes (Table 2) though not significantly, peaking at around 30% implementation (Fig. 1).

The associations between implementation fidelity and smoking outcomes differed when stratified by school types (Table 2). For students in social and healthcare schools, high implementation was associated with lower odds of smoking during school hours (OR 0.20, 95% CI 0.07–0.59). For students in technical, commercial, or preparatory schools, high implementation was not associated with smoking outcomes, but students in the medium implementation group had significantly higher odds of smoking daily (OR 2.74, 95% CI 1.09–6.92), regularly (OR 1.93, 95% CI 1.01–3.67), and during school hours (OR 2.75, 95% CI 1.10–6.86).

The impact of implementation of the tobacco policy, curriculum, and class competition components on the smoking outcomes was analyzed separately. High implementation of the tobacco policy was not significantly associated with the smoking outcomes though the estimates were in the intended direction, i.e., lower odds of smoking in the high implementation group (Table 3). Students in the low and medium implementation groups had higher odds of smoking outcomes though not significantly and most pronounced in the low implementation group. We detected no associations between implementation of the curriculum and competition components and smoking outcomes (Table 3).

When decomposing implementation into school-level fidelity and student responsiveness, our additional analyses showed that both fidelity and responsiveness were associated with the smoking outcomes (Supplementary Table S2). Fidelity was associated with lower odds of regular smoking among all students and lower odds of smoking during school hours among social and healthcare vocational students similar to the observed associations of the main analyses. Among all students, the associations between positive attitudes towards smoke-free school hours and lower odds of smoking, and between negative attitudes and higher odds of smoking, did not remain significant after adjustment. Among social and healthcare vocational students, positive attitudes were associated with lower odds of smoking during school hours, and among students at the other school types, negative attitudes were associated with higher odds of all smoking outcomes.

We compared background characteristics of the included study population (n = 1112) with those of students excluded from analysis (Supplementary Table S3). A slightly higher proportion of the included students were from high socioeconomic position compared to students lost to follow-up (2 percentage points), and slightly more excluded students had low, unclassifiable, or missing data on socioeconomic status.

This study investigated the association between implementation of the “Focus” smoking prevention intervention, i.e., the extent to which the intervention was delivered and received as intended, and smoking status and smoking during school hours after 4–5 months of intervention among Danish vocational and preparatory basic education students. We found that high levels of implementation were associated with lower odds of regular smoking and smoking during school hours, but not consistently with daily smoking. Stratified by school type, we found that high implementation was associated with lower odds of smoking during school hours in social and healthcare vocational schools, and no associations with decreased smoking in technical, commercial, and preparatory schools. On the contrary, a medium level of implementation was associated with higher odds of all smoking outcomes in these school settings. Lastly, we found that implementation of the intervention components was not associated with the smoking outcomes when analyzed separately.

In accordance with the present results, previous studies have demonstrated that the level of implementation matters for interventions’ effectiveness in general (Durlak & DuPre, 2008) and in smoking prevention interventions in elementary schools (Bast et al., 2016, 2019). To our knowledge, our study was the first to investigate the association between implementation of a smoking prevention intervention and smoking outcomes among vocational school students. Interestingly, the present study indicated a complex, non-linear relationship between level of implementation and smoking outcomes that differentiated between the school settings. A realist review by Schreuders et al. (2017) has similarly indicated that good quality implementation of STPs may lead to the intended effects of decreased smoking and that poor implementation could lead to reverse effects. Reverse effects have also been found in other studies (Poulin, 2007; Rozema et al., 2018). A Dutch study found that more adolescents started smoking at schools that implemented an outdoor school ground smoking ban compared to the control condition (Rozema et al., 2018). Schreuders et al. suggest mechanisms that either facilitate or counteract the impact of STPs on students’ smoking behavior, depending on the quality of the implementation process, in terms of enforcement practices, sanctioning, preventive efforts, etc. Examples of countervailing mechanisms include that students look for alternative places to smoke, develop new social meanings of smoking, undermine the harms of smoking, and alienate from school and the messages on smoking (Schreuders et al., 2017). Examples of facilitating mechanisms include that students feel they will get sanctioned for smoking, that students see less smoking during school hours and thereby feel less social pressure to smoke, and that students internalize anti-smoking beliefs (Schreuders et al., 2017). Similar countervailing and facilitating mechanisms activated in the implementation process might explain the complex relationship observed in the present study. Furthermore, such mechanisms might explain the variation across school types; the contextual conditions of the school types may have brought about different implementation processes that activated different mechanisms resulting in different outcomes. For example, low support of the tobacco policy and its enforcement from staff at technical vocational school could make students feel it was easy to avoid getting caught smoking and sensed a contradiction between the school’s anti-smoking message and actual enforcement practice. This could lead to students smoking at hidden locations instead of abstaining from smoking during the school day (Schreuders et al., 2017). Similarly, high support of the tobacco policy and its enforcement in line with an educational focus on health at social and healthcare vocational schools may have led to students internalizing the anti-smoking beliefs or believing they would get sanctioned for smoking (Schreuders et al., 2017). Students in social and healthcare schools may also be more motivated to quit or avoid smoking given their orientation towards health-related vocations. The Medical Research Council guidelines highlight that different overlapping research perspectives can be used in combination to explore the complexity of interventions, such as a theory-based perspective to inform an effectiveness evaluation (Skivington et al., 2021). Therefore, theory-based perspectives such as realist evaluation concerned with the interplay between context, mechanisms, and outcomes might be useful to further explore and unfold the complexities identified in the present study.

To our knowledge, few other studies have included component-specific implementation measurements in effectiveness evaluations of smoking prevention interventions (Bast et al., 2016, 2019). In the evaluation of a school-based smoking prevention intervention including, e.g., STP and curricular components, Bast et al. (2019) included number and combinations of components implemented and concluded that implementing all three intervention components mattered more for smoking prevention than implementing one or two components. Relatedly, our results indicate that high implementation of the overall intervention rather than the single intervention components affected students’ smoking behavior.

We found that overall implementation of the tobacco policy was not associated with our smoking outcomes. A longitudinal study among upper secondary school students across six European cities similarly found that a high total STP score considering enforcement, comprehensiveness, and communication, as reported from staff and students’ perspectives, was not associated with weekly smoking or smoking outside school premises but associated with less smoking on school premises (Mélard et al., 2020). However, the study also found that better enforcement of STPs was associated with less weekly smoking and less smoking on school premises (Mélard et al., 2020) indicating that enforcement is a significant aspect of implementing STPs in terms of changing students’ smoking behavior. We also found that implementation of the educational curriculum and class competition components did not affect the outcomes when analyzed separately, which is in line with the findings from Bast et al. (2019) concluding that exposure to curricular activities only, or any other single component, was not associated with less smoking (Bast et al., 2019). Using curricular materials and activities in school-based smoking prevention can be effective in preventing the onset of smoking and promoting attitude change (Mpousiou et al., 2021; Thomas et al., 2013). However, these interventions can be challenging to implement. As we observed in the “Focus” intervention, schools reported low levels of fidelity to the educational curriculum though students who received the sessions reported high acceptability (Jensen et al., 2023). Thus, we cannot conclude from this study that the educational curriculum or the class competition are redundant to the overall impact of the implementation on the outcomes. The intervention was designed to address multiple individual, interpersonal, and environmental determinants of students’ smoking behavior (Andersen et al., 2023) using a multi-component strategy (Coppo et al., 2014). In fact, it is recommended that STPs are accompanied by educational and preventive efforts to achieve the intended effects (Galanti et al., 2014; Schreuders et al., 2017). The unique contributions of the components on the smoking outcomes are difficult to detangle, and this task would require another study design such as a multi-arm trial allocating schools to implement single or different combinations of components. Instead, a possible interpretation of the findings is that the educational curriculum and class competition components are not essential to the effectiveness of the intervention, implicating that these components could be modified to fit local contextual conditions for scale-up or adaptation purposes without jeopardizing the outcomes (Blase & Fixsen, 2013).

A major strength of the present study was the measurement of implementation fidelity that was (1) underpinned by a comprehensive conceptual framework accounting for the implementation concepts' interaction (2) tailored to the specific program theory and assumptions of the “Focus” intervention, (3) component-specific which enabled analysis of how the intervention’s components influenced the outcomes, and (4) calculated as an individual-level construct that integrates school-level and student-level implementation. Individuals’ exposure and responsiveness are not always included as aspects of implementation fidelity (Moore et al., 2015). While some process evaluation or implementation frameworks incorporate whether and how participants come into contact with the intervention (i.e., “reach” or “coverage”) (Carroll et al., 2007; Linnan & Steckler, 2002), others view these as separate dimensions (RE-AIM; Glasgow et al., 1999) or as intervention outputs or intermediate outcomes (Moore et al., 2015), but there is no consensus on which to include (Moore et al., 2015). Moore et al. (2015) argue that evaluators should consider the extent to which participants’ engagement with the intervention is necessary to achieve effectiveness to determine whether reach should be included as an aspect of full implementation (Moore et al., 2015). More structural tobacco control interventions such as price policies could implicitly impact young people’s smoking behavior, but school tobacco policies, educational curricular activities, etc., require more conscious processes. Thus, in the case of “Focus”, we assume that the students would have to actively engage with the intervention components to achieve effectiveness, e.g., if students accept the policy to some extent, they will be more likely to comply with it. Furthermore, a key benefit of an individual-level measure is to avoid assuming that all students within the same school or unit are exposed equally to an intervention as it was delivered, thus considering each student’s uptake of the intervention (Ferm et al., 2018). Other strengths of the present study included the randomized study design enabling comparison of students’ smoking outcomes to a non-exposed control group, which has been requested in the literature on the effectiveness of STPs on students’ smoking to strengthen the evidence base (Galanti et al., 2014). The data collection took place during school hours to minimize non-response. The present study also has limitations. A limitation of the implementation construct is that we did not have appropriate data for all dimensions prescribed in the quantification model for all intervention components (Jensen et al., 2023). This implies that the intervention components’ constructs are not equally detailed and thus may vary in how well they capture the implementation process. Further, validated questionnaire items were not available from the literature. We used items from similar studies, where available, or developed new items tailored to the intervention components. We performed the analyses on a relatively small sample due to missing data which may have introduced type II error. Thus, the actual relationship between implementation of the intervention and students’ smoking might be stronger than estimated in this study. To assess potential selection bias, we compared background characteristics of the study population and students excluded from analysis. As differences were small, we concluded that selection bias was minimal, and we did not take further measures to handle missing data such as multiple data imputation. Another limitation is that the analyses that included smoking during school hours at follow-up were not adjusted for this variable at baseline. We did not include this item at baseline because the questionnaires were answered during the first weeks of the school semester, and many students would not have established a normal everyday school life at this point. Instead, we adjusted for baseline regular smoking because we assumed that students who smoked regularly at baseline would be likely to smoke during school hours at baseline.

The findings of this study carry important implications for research and practice. Our study indicated enhanced effectiveness of the smoking prevention initiatives at high implementation levels and potential reverse effects at lower levels. A policy priority should therefore be to support the implementation process of school tobacco policies actively, e.g., through the employment of implementation strategies tailored to the school settings (Powell et al., 2015, 2017) as our findings also suggest that the context of implementation plays a role for the effectiveness. The additional analyses decomposing the impact of school-level fidelity and student responsiveness on students’ smoking outcomes indicate that strengthening schools’ implementation is important across all school types to reduce smoking. The analyses also strongly indicate that it is important to address students’ attitudes towards smoke-free policies in technical, commercial, and preparatory schools to avoid reverse effects. Among social and healthcare students, negative attitudes did not seem to play an important role for smoking behavior, but positive attitudes were associated with less smoking. Thus, improving school-level fidelity seems to be more important in this school type. Further research is needed to understand the mechanisms behind the differential effects observed in this study. We found that high implementation of the intervention decreased smoking during school hours. While a key task of smoking prevention is to keep young people from smoking at all, changing the visibility of smoking in the school environment is an important step towards sustainable change because it decreases the social acceptability of smoking (Coppo et al., 2014; Galanti et al., 2014), especially in school environments such as vocational schools with a persistent history of high smoking prevalence (Ringgaard et al., 2020).

This study was the first to apply the model for quantification of implementation developed by Ferm et al. (2018) in an effectiveness analysis. We found the model to be a feasible tool to assess implementation and its relation to individual outcomes. However, operationalizing the constructs required a wide range of questionnaire items which is often not feasible in large intervention studies. To lessen participant burden, it would be interesting to know whether more limited indicators could capture the relationship between implementation and outcomes. A simple school-level construct was used in the primary effectiveness study (Kjeld et al., 2023) categorizing students according to their schools’ delivery of either school and class level components (full intervention) or parts of the intervention only (partial intervention). The results showed that exposure to full intervention delivery, but not partial intervention delivery, affected daily smoking (OR: 0.44, 95% CI: 0.19–1.02) and regular smoking (OR: 0.57, 95% CI: 0.32–1.01) (Kjeld et al., 2023). Thus, using a simple construct of delivery can provide an indication of the relationship. This might be sufficient in some studies to complement an effectiveness study. However, to analyze the relationship between the variables more closely, more dimensions of implementation must be included and should be tailored to the assumptions underpinning the intervention (Ferm et al., 2018).

From this study, we can conclude that high implementation fidelity of the “Focus” intervention was effective in reducing students’ regular smoking and smoking during school hours after 4–5 months. The relationship between implementation fidelity and students’ smoking was complex; high implementation raised the chances of decreased smoking while lower implementation could lead to reverse effects. These associations varied between the school types involved in the intervention. Thus, this study shows potential for enhancing and improving the quality of school-based smoking prevention among older adolescents and young adults. Our study adds to the knowledge base of the effectiveness of STPs on students’ smoking. It confirms that the existing evidence among younger age groups applies to school settings of older adolescents and young adults, i.e., that the implementation process is of key importance for interventions that include STPs to achieve their intended effects.