Examining the sustainability and effectiveness of co-created physical activity interventions in vocational education and training: a multimethod evaluation

For many young people, vocational education and training (VET) represents the first step toward working life. Despite their recent entry into the labor market, young workers are already exposed to increased health risks and are vulnerable to work-related diseases [1]. The health burdens of apprentices appear high in physically demanding occupations, such as in the automotive industry or the nursing care sector [2, 3]. Concurrently, insufficient physical activity (PA) behavior among apprentices has been reported [4]; this also applies to the automotive mechatronics and nursing care sectors [2, 5]. Although there is incontrovertible evidence of the lifelong health benefits of PA underlining the need for PA-promoting interventions [6, 7], these interventions are lacking in the field of VET [8].

In contrast to the overwhelming evidence on the benefits of PA, there is only limited evidence on the effectiveness of different intervention strategies for promoting PA [9, 10]. When developing interventions to promote PA among individuals, it is essential to understand why some people are more physically active than others [11]. The ecological model by Bauman et al. [12] provides a comprehensive framework to explain PA, suggesting that determinants at the individual, behavioral, social, environmental, and political levels play a contributing role in PA. Accordingly, the successful promotion of PA demands the consideration of different influencing factors at different levels [12]. Hence, the interaction between the individual and environmental levels also comes forward, as an effective behavioral change requires supportive environments and policies [13].

However, the success of an intervention relies on both effectiveness and sustained implementation. To achieve lasting intervention effects at the individual level, long-term implementation of the intervention at the institutional level is fundamental [14]. The extent to which interventions are maintained depends on different factors relating to the innovation itself (e.g., fit, adaptability, effectiveness), the context (e.g., climate, culture, leadership), the capacity (e.g., champions, funding, resources), and processes and interactions (e.g., engagement, shared decision-making, partnership) [15].

To ensure both the effectiveness and sustainability of interventions, co-creation approaches in which researchers develop interventions alongside relevant stakeholders seem to be promising. Co-created interventions can be tailored to the target group and given setting, allowing for the development of localized solutions [16, 17]. By involving the target group and listening to their voices, relevant determinants of PA can be identified and addressed [18‐20], thereby increasing the acceptability and effectiveness of an intervention [21, 22]. Moreover, a co-creation approach can facilitate contextualization of the new intervention for the specific setting by embedding it into established routines and structures, utilizing existing resources, and building new capacities [23, 24, 18]. The specific adaptations of the intervention to the setting promote the routinization of the intervention, which in turn increases the likelihood of its sustained implementation [25‐28].

Against this background, the research project Physical Activity-related Health Competence in Apprenticeship and Vocational Education (PArC-AVE), embedded in the research consortium of Capital4Health, addressed PA promotion in VET in the automotive mechatronics and nursing care sectors. The primary aim of the project was to develop and implement PA-promoting interventions tailored to the needs of the target group and the given setting in two German VET institutions using a co-creation approach involving members of the target group and other relevant actors from research, policy, and practice. During the participatory development and implementation of interventions, the focus was on both the individual level by promoting apprentices’ PA and physical activity-related health competence (PAHCO) [29], and on the institutional level by building capacities for a PA-friendly environment. A first evaluation of the project showed that the co-creation approach succeeded in developing and implementing PA-promoting interventions and thus in building new capacities for PA promotion on the institutional level [30]. However, the effectiveness and sustained implementation of these interventions have not been studied yet.

As interventions become more complex (in this case, co-created multi-component interventions tailored to the target group and the given context), their evaluation also becomes more challenging [31]. When evaluating complex interventions, it is important to know whether they work and how and why they work [32]. Therefore, evaluating both the effectiveness of the intervention and the context, including the underlying processes and factors affecting implementation, is recommended [31, 16, 33]. To cope with this complexity, pragmatic evaluation approaches characterized by theoretical flexibility, methodological comprehensiveness, and operational practicality are increasingly used [34‐36]. In this context, both qualitative and quantitative research methods are applied, as they answer different research questions on the one hand and provide a comprehensive evidence base by combining methods and data triangulation on the other hand [34, 37, 38].

Despite the growing popularity of co-creation approaches in developing interventions, the long-term evaluation of both the outcomes and underlying processes of these interventions is sparse [22]. Within the research project PArC-AVE, we aimed to address this issue and evaluated (1) the sustainable implementation of the multi-component interventions developed at the institutional level and (2) the effectiveness of specific components of these complex interventions at the individual level.

The present article provides new and comprehensive insights into the effectiveness and sustainability of participatively developed multi-component interventions in VET, specifically in the automotive mechatronics and nursing care sectors. First, we explored the sustainable implementation of multi-component interventions and the factors that contributed to sustained implementation. Second, two non-randomized controlled trials were used to examine the impact of single components on apprentices’ PA and PAHCO. While we found variability across sites in terms of sustained implementation, no difference was found in effectiveness.

At the institutional level, differences emerged between the implementation statuses of the multi-component interventions. While the long-term implementation of the multi-component intervention could not be registered in the automotive mechatronics institution, single components of the multi-component intervention were still being implemented in the nursing care institution. In this context, many factors influencing the likelihood of sustainable intervention implementation were identified, most of them congruent with the sustainability factors found in recent reviews of the literature [15, 46, 52, 28]: for example, legal framework at the outer contextual level, climate and culture, cooperation, embedment, decision-making, health-promoting leadership, relevance, resources, and strategic planning at the inner contextual level, acceptance, effectiveness, fit, and flexibility at the intervention level, alongside attitude and mindset, champion, qualification, and support at the personal level. In addition, new factors have been identified that have received little attention in the available sustainability literature. For instance, some emerging challenges, such as the outbreak of the COVID-19 pandemic and personnel changes, were mentioned as influencing factors. Further, new influencing factors identified were situational circumstances, such as openness of the sector to PA promotion and engagement, or outcomes, such as the emergence of ownership and empowerment. Although ownership and empowerment have rarely been discussed in the sustainability literature to date [15, 52, 46, 28], both are core concepts in participatory research [24], making their appearance less surprising given our chosen approach.

Other noticeable findings emerged from the differences in the number and availability of influencing factors between automotive mechatronics and nursing care institutions. Thus, not only a higher number of influencing factors were identified in the nursing care institution, but the facilitating factors were also available more frequently. Accordingly, the high number of facilitating factors that were not available could be a possible reason for failed long-term implementation of the intervention in the automotive mechatronics institution. These differences between institutions were most apparent at the inner contextual and personal level, as although approximately equal influencing factors were identified, many of those factors were not available in the automotive mechatronics institution (i.e., attitude and mindset, champion, commitment, embedment, strategic planning, health-promoting leadership, resources, support). Another remarkable result is that ownership at the inner contextual level and empowerment at the personal level were not even mentioned as factors influencing sustainability in automotive mechatronics, although both are core elements of participatory research [24]. As taking responsibility for continuing the intervention is often the consequence of empowerment and ownership [53], the lack of both factors might be a major barrier to successful long-term implementation in the automotive mechatronics institution. However, it remains unclear what contributed to the fact that empowerment and ownership were existent in nursing care, thus increasing the likelihood of sustained intervention implementation, while neither factor was mentioned in automotive mechatronics. With a higher number of influencing factors mentioned, the outer contextual factors seemed to play a greater role in the nursing care institution than in the automotive mechatronics institution. At the intervention level, there were no major differences between the two institutions, which is perhaps unsurprising given the co-creation approach used to develop interventions tailored to the target group and setting.

Examining the various influencing factors explicitly, interrelations between factors became visible, which could be another possible reason for the observed differences in the long-term implementation of the multi-component interventions in both settings. In nursing care institution, for example, the embedment of intervention components was favored by the law reform of the nursing professions. Indeed, this change in the legal framework, coupled with the overall openness of the sector to physical activity promotion, may have created a window of opportunity to place PA promotion in the VET of nursing care [54]. As interventions with relevance to existing aims and policies are easier to implement [55], involving relevant actors from policy and practice in co-creation strategies seems valuable for identifying existing policies, goals, structures, and practices to foster the embedment of an intervention [56, 57]. Furthermore, interrelations between the factors champion and decision-making or health-promoting leadership were found in nursing care; accordingly, the champions were part of the decision-making and health-promoting leadership simultaneously. In the automotive mechatronics institution, in contrast, both the champion and the health-promoting leadership had left the institution due to personnel changes. Since the importance of a champion and leadership in the implementation process predicts implementation success [58‐64], losing these important actors due to personnel changes appears to have challenged successful long-term implementation in automotive mechatronics institution. This parallels previous research indicating that personnel changes negatively influenced the long-term implemenation of interventions [64]. Moreover, links between the relevance of the issue of PA promotion and the commitment of actors could be determined in automotive mechatronics; PA promotion was perceived as an additional task and, thus, met with little response and interest from the individuals. These results could also depend on the characteristics of both settings. Therefore, notably, VET in Germany is organized in a dual apprenticeship system combining school-based learning and company-based training. While the development and implementation of the multi-component intervention in the nursing care sector took place at school, in the automotive mechatronics sector, it was conducted at the workplace. In our case, the nursing care school was characterized by a flattened decision-making hierarchy (i.e., the champion was part of the health-promoting leadership and decision-making), and strong existing commitment. In contrast, the automotive company had a more hierarchically decision-making structure, in which the champion was not embedded in the leadership structure, and PA promotion was of low relevance, so that no commitment was demonstrated.

Overall, the differences between the two institutions regarding sustained implementation and the associated influencing factors could also be related to organizational readiness, as the latter is considered a key predictor for successful implementation [65]. According to Scaccia et al. [66], “readiness refers to the extent to which an organization is both willing and able to implement a particular innovation” (p. 485), and it includes the components of the organization’s motivation to adopt an innovation, general organizational capacities, and innovation-specific capacities. Thus, lack of commitment, support, and relevance, alongside the shortage of resources and the absence of a champion could have resulted in lack of readiness to implement the intervention in the automotive mechatronics institution over the short and long-term. In contrast, the actors of the nursing care institution were highly motivated (e.g., commitment, attitude and mindset, support) and utilized existing organizational (e.g., resources) and intervention-specific capacities (e.g., champion), and were thus prepared to implement the interventions in the long-term. In this context, assessing organizational readiness for change from the outset seems worthwhile to identify those institutions that are willing and able to implement the interventions, or otherwise to prepare those institutions that are not yet ready for change by addressing deficits in readiness [67]. Finally, a co-creation approach may be more appropriate for some institutions than others. In the nursing care sector or school setting, participatory intervention development appeared promising, as it was related to sustainable implementation at the institutional level. Even if the readiness for change is present, there must also be a readiness for participation where actors’ participation is important. If an institution is completely closed to the actors’ participation, it would be unsuitable for a co-creation approach [68].

At the individual level, the effectiveness of the multi-component interventions for changing apprentices’ PA behavior and PAHCO, evaluated on the basis of one intervention component per institution, could not be demonstrated. In contrast with previous findings reporting low volumes of PA among nursing care and automotive mechatronics apprentices, our results indicated that these two groups were achieving a very high amount of PA. These results are in line with other recent studies [69‐71], each reporting similarly high PA volumes in the automotive mechatronics and nursing care sectors. Regarding the results of the sustainability evaluation, implementation failure could be one of the main reasons for the missing effectiveness in the automotive mechatronics institution. The fact that the interventions implemented were not typical researcher-developed evidence-based interventions implemented in a real-world setting after efficacy had been demonstrated [72‐75], but rather co-created interventions based on elements of evidence-based behavior change techniques and tailored to the specifics of the target group and given setting without prior evaluation of their effectiveness under ideal conditions, may be another reason for the lack of intervention effects. This parallels the findings of a recent systematic review reporting that participatively developed interventions tended to improve the relevant psychological factors associated with PA, but not PA levels per se [8]. Although synergizing the scientific world with the real-world is considered a key benefit of co-creation [17], a key challenge appears to be involving the target group to develop target group-specific interventions without neglecting theory and an evidence-based approach.

With respect to the methodological approach, it remains a challenge to evaluate the effectiveness and sustainability of complex multi-component interventions developed in a participatory manner. Although evaluating effectiveness at the individual level and sustainability at the institutional level using a pragmatic evaluation approach provided us with a deep and comprehensive insight into the impact of a co-creation approach, clarifying the role of co-creation requires methodologically complex and elaborate evaluation designs. To meet this claim, a hybrid effectiveness-implementation trial in the context of a cluster-randomized trial seems worthwhile [76, 77]. On the one hand, a hybrid effectiveness-implementation trial allows for a closer look at the interplay between effectiveness and implementation, as information on the intervention effects at the individual level and the effects of the intervention strategy for improving intervention implementation at the institutional level are collected simultaneously, rather than consecutively, with a time lag, as in our case. On the other hand, a cluster-randomized trial provides the opportunity to compare participatively and non-participatively developed interventions and thus clarify the role of co-creation in the effectiveness and implementation or sustainability of these interventions.

Presently, the question of the role of co-creation regarding effective and sustainable PA promotion in VET cannot be answered conclusively. Only in the nursing care sector or school setting, a co-creation approach appeared promising for sustained intervention implementation. However, sustainable intervention implementation also depended on a variety of influencing factors that need to be considered from the outset. Hence, it might be useful to assess setting-specific factors, such as organizational readiness or other important influencing factors, a priori to utilize existing sustainability factors or to focus on deficient factors when developing and implementing interventions and thus to foster their sustainable implementation. To demonstrate effectiveness at the individual level, future intervention projects need to be designed with methodologically elaborate and comprehensive evaluation over a longer time period. In this context, future research should consider cluster-randomized hybrid-implementation trials to clarify both the role and impact of a co-creation approach.