Implementing Intervention Research into Public Policy—the “I³-Approach”

Researchers are used to identify scientific problems and desiderates for new insights. However, in the case of an integrative approach to intervention/prevention and implementation research, the focus is not primarily on problems arising in basic research but on (mostly social) problems in society. Consequently, in order to identify such problems and in order to be generally willing to take action, researchers must not only be curiosity-driven but also mission-driven, combining the quest for fundamental understanding with a consideration of practical use (Stokes 1997). In other words, if scientists intend to contribute to this field of research, the first step requires socio-political responsibility as a basic mindset.

The availability of robust and sound scientific knowledge and evidence is a fundamental precondition for working on an identified problem. Moreover, it is a prerequisite for any kind of transfer (Spiel et al. 2009). Consequently, researchers have to be experts in the relevant field with excellent knowledge of theory, methods, empirical findings, and limitations. This also includes the political dimension of research in the sense of defining and financing corresponding research topics.

The identification of a problem and the availability of relevant insights for initiating changes are not enough if one does not succeed in identifying promising starting points for interventions and their implementation. This must be emphasized, as a wide body of research has made it clear that many intervention programs and measures do not work everywhere and at all times (Meyers et al. 2012). Here again, a necessary condition is high expertise in the relevant scientific field. However, this alone will not do. It must be combined with a differentiated view of prevailing cultural and political conditions. Researchers need knowledge and experience in the relevant practical field and its contextual conditions. This also includes knowledge about potential problems and limitations.

This step is a very crucial one for several reasons. Successful development and implementation of evidence-based intervention in practical settings involves various stakeholders and requires cooperation, persistence, time, and money. In order to conduct integrative intervention and implementation research, stable alliances with the relevant policymakers are necessary, which many researchers traditionally have not established. However, as research often follows its own, very intrinsic logic, which clearly differs from political thinking, a very deliberate process of establishing cooperation and building alliances is necessary. Among other things, this includes more awareness of policymakers’ scope of action. Researchers have to consider that there are other influences on government and policy, beyond evidence. These include values, beliefs, and ideology, which are the driving forces of many political processes (Davies 2004, 2012). Habits and traditions are also important and cannot be changed quickly. Furthermore, researchers’ experience, expertise and judgment influence policymaking, and media; lobbyists and pressure groups also play a role. Researchers have to keep in mind that policymaking is highly embedded in a bureaucratic culture and is forced to respond quickly to everyday contingencies. Last but not least, as resources are limited, policymaking is always a matter of what works at what costs and with what outcomes (Davies 2004, 2012). Consequently, researchers have to find ways to integrate evidence with all these factors. However, again, this step also addresses a certain basic attitude of researchers: It requires that researchers make their voice heard, and sometimes, they have to be very insistent.

This step is the centerpiece and a long process rather than a step. The coordinated development has to be performed in a theory-driven, ecological, collaborative, and participatory way. This means that researchers have to include the perspectives of all relevant stakeholders (practitioners, policymakers, government officials, public servants, and communities) in this development process, communicate in the language of these diverse stakeholders, and meet them as equals. Therefore, researchers again have to consider parameters for their research work that differs from many traditional approaches: working together right from the beginning is not common in many fields and also requires new conceptions of e.g., research planning (regarding things like the duration of project phases; see Meyers et al. 2012). Here, the big challenge is to find a balance between wide participation and the maintenance of scientific criteria and standards of evidence, as well as between the freedom of science and research on demand. Consequently, researchers must have theoretical knowledge and practical experience in their very specific field of expertise, but the required profile for a successful “integrative intervention and implementation researcher” is much wider.

For this final scale-up step, several models and guidelines have been proposed by implementation science (see “Theoretical Background”). Therefore, we will only make a reference to them (e.g., Fixsen et al. 2013; Meyers et al. 2012; Spoth et al. 2013).

In sum, none of the PASCIT steps refer to a completely new consideration or demand (see also Glasgow et al. 1999, who developed the RE-AIM framework for the evaluation of public health interventions), but they have to be seen as one coherent approach. The sound, consistent integration of intervention and implementation research with the goal of introducing changes to policy also requires a (re)differentiation of our scientific identity and the creation of a new, wider job description for researchers in this field.

To illustrate the application of the PASCIT steps of the “I³-Approach”, the following sections introduce the ViSC school program, which seeks to reduce aggressive behavior and bullying and to foster social and intercultural competencies.

Since 1997, violence in schools has gained widespread public attention in Austria, and a number of reports in the media have addressed this topic. As a consequence, a joint statement issued by four federal ministers declared the government’s intention to set up initiatives to prevent violence in several social domains. The government provided financial support for a number of individual intervention and prevention projects in schools. However, as shown in a national report (Atria and Spiel 2003), most of the initiatives taken to prevent violence in schools were organized by individual teachers, researchers were not involved in the planning and organization of these projects. Therefore, these projects and programs were not theoretically based, project goals were imprecisely formulated, and programs were rarely documented and evaluated. In sum, they were a far cry from aligning with the standards of evidence.

Many prevention and intervention programs have been developed by researchers to take on violence at school. They have been evaluated in numerous efficacy and effectiveness trials (e.g., Ferguson et al. 2007; Fox et al. 2012; Smith et al. 2004; Ttofi and Farrington 2009, 2011), and many resources of national institutions have been invested into the implementation of research-based programs in several countries (e.g., Kennedy 1997; Nutley et al. 2007; Petrosino et al. 2001; Shonkoff 2000). It has become apparent that there are at least two key features necessary for program success: (a) to approach the school as a whole and to incorporate a systemic perspective and (b) to conduct activities at school level, class level, and individual level (Smith et al. 2004; Ttofi and Farrington 2009, 2011). However, the deployment of research findings in applied settings has remained slow and incomplete (e.g., Slavin 2002, 2008). It turns out that national strategies actively supported by governments are needed for sustainable violence prevention (Olweus 2004; Roland 2000; Spiel et al. 2011a, b, c).

Consequently, the best means for dealing with violence in Austrian schools was the development of a national strategy with policy and advocacy as important pillars (Roland 2000). Violence prevention programs which take into account the key factors identified for success (Smith et al. 2004; Ttofi and Farrington 2011), comply with the standards of evidence (e.g., Flay et al. 2005) and consider cultural and situational conditions (Datnow 2002, 2005; Shonkoff 2000) should be conceptualized as central parts of this national strategy.

Therefore, we tried to convince officials at the Federal Ministry for Education and the Minister herself of the need for a strategy at a national level by advocating for evidence-based programs and explaining the advantages of a strategic plan as opposed to individual initiatives. Based on several previous projects with the Federal Ministry for Education, we had established an open and honest line of communication.

At the beginning of 2007, in the wake of a quick succession of dramatic, well-publicized incidents in schools and the public discussion of the high rates of bullying and victimization in Austria that followed reports of the results of the Health Behaviours in School-aged Children (HBSC) survey (Craig and Harel 2004), we received a mandate from the Federal Ministry for Education to develop a national strategy for violence prevention in the Austrian public school system. In preparing the national strategy, we had to cope with the challenge that there was no existing system of collaboration among various stakeholders actively involved in violence prevention and intervention (e.g., school psychologists, social workers, teacher unions) and the lack of knowledge concerning scientific standards. It was therefore our intention to systematically integrate the perspectives of these stakeholder groups in strategy development and to specifically consider the application of theory-based and empirically evaluated prevention and intervention programs (Spiel and Strohmeier 2011).

We developed the strategy between January and November 2007 in a continuous dialogue with officials responsible for this issue at the Federal Ministry of Education and in intensive exchange with international colleagues who had been involved in similar national strategies in their own countries (for details, see Spiel and Strohmeier 2011). In December 2007, the Federal Minister decided to implement the strategy and presented this decision and the strategy plan in a major press conference. For strategy management and implementation, a steering committee was established at the Federal Ministry, with Christiane Spiel as an external member responsible for research issues. In 2008, the national strategy became part of the coalition agreement between the two governing parties and was designed to last up to the end of the legislation period in September 2013. As a consequence, money was devoted to the national strategy and the activities within the strategy. The officers of the Federal Ministry and the Federal Minister herself were very much committed to the national strategy and very keen on getting positive results. The implementation of the strategy continues during the legislation period from 2013 to 2018 (for details about the national strategy and its development, see Spiel and Strohmeier 2011, 2012; Spiel, et al. 2012). As one part of the national strategy, it was possible to expand the so-called ViSC class project (Atria and Spiel 2007), which had previously been developed and applied in rather controlled contexts, into a school-wide program, to develop an implementation strategy for Austrian schools, and to conduct a large-scale evaluation study.

In accordance with the national strategy, the main goals of the ViSC program are to reduce aggressive behavior and bullying as well as to foster social and intercultural competencies in schools (Strohmeier et al. 2012). The ViSC program was designed to focus on the school as a whole and to incorporate a systemic perspective. The prevention of aggression and bullying was defined as a comprehensive school development project over the duration of an entire school year. Activities were designed to operate on three different levels: the school as a whole, the classrooms, and the individual (for details, see Spiel and Strohmeier 2011; Strohmeier et al. 2012). In an in-school training for all teachers, basic knowledge on bully-victim behavior and its development was presented. Based on this shared knowledge, the school principal and all of the school’s teachers were encouraged to jointly develop (a) a shared definition of aggression and bullying, (b) shared principles on how to handle aggression and bullying, and (c) commonly agreed-upon measures to sustainably reduce aggression and bullying on the school level. Furthermore, teachers were trained to conduct talks with bullies, victims and their parents in accordance with shared, standardized procedures, in reaction to critical incidents.

One unique feature of the ViSC program concept is that it includes a theory-based project at the class level, which was developed in recognition of the importance of the class context for the prevalence of bullying and victimization. The ViSC class project consists of 13 training units (for details, see Atria and Spiel 2007; Spiel and Strohmeier 2011; Strohmeier et al. 2012). Taking the culture of Austrian schools into account, the ViSC class project is well structured, but open for adaptation in terms of materials and activities used. Before expanding the ViSC class project to the ViSC school program, it had been implemented four times by different researchers in Austrian and German schools. Summative and formative program evaluations confirmed promising results (Atria and Spiel 2007; Gollwitzer et al. 2006, 2007). In particular, a high implementation quality could be found. Out of a total of 52 training units (4 classes × 13 units), only one training unit was deemed not to be in accordance with the project goals. Twenty-nine units were structured exactly according to the training manual and 22 units were in accordance with the project goals, but used adapted materials (Gollwitzer et al. 2006).

The ViSC program’s implementation model was developed concurrently by the same researchers who developed the ViSC program. It needed to take the context and culture of the Austrian school system as well as the concrete situation at each specific school into account (Datnow 2002, 2005). Therefore, and to avoid overburdening teachers and principals, we developed a cascaded train-the-trainer model for the implementation of the ViSC program: researchers train multipliers, multipliers train teachers, and teachers train their students. To train multipliers—known as ViSC coaches—ViSC courses were offered by the program developers. The ViSC courses consisted of three workshops—mostly held at the University of Vienna—and the implementation of the ViSC program in one school. ViSC coaches were required to hold in-school trainings for the entire staff at their assigned school and to supervise and coach them throughout the implementation process. The ViSC coaches also held an additional in-school training for those teachers who planned to conduct the ViSC class project and offered them three supervision units during the implementation of the class project.

The primary target groups for recruiting ViSC coaches are teachers at educational universities and psychologists. From 2008 to 2014, 55 coaches were trained. To support schools in implementing the ViSC program, many materials were provided on a website, which was presented and explained to teachers by the ViSC coaches. Furthermore, a manual was created to serve as a guide for teachers.

In 2009/10, the ViSC program was intensively evaluated in terms of implementation quality and effectiveness. All secondary schools located in the capital city of Austria were invited to participate in the ViSC program. Out of 155 secondary schools in Vienna, 34 schools applied for participation, out of which 26 schools fulfilled the necessary requirements (participation of the whole school staff in the program, providing time resources for all ViSC trainings, taking part in the evaluation study; Gradinger et al. 2014; Strohmeier et al. 2012). Applying a randomized intervention-control group design, 13 schools were randomly assigned to the intervention group. Out of the remaining 13 schools, only five agreed to serve as control schools. Data from 2042 students (1377 in the intervention group, 665 in the control group) from grades 5 to 7 (47.3 % girls), mean age 11.7 years (SD = 0.9), and attending 105 classes were collected at four points in time. In addition, 338 teachers participated in a survey at the pre-testing and post-testing points.

Our first analyses focused on the short-term effectiveness of the program with respect to aggressive behavior and victimization (Strohmeier et al. 2012). A multiple group latent change score model (LCS) to compare the control and intervention group was applied, with gender and age as covariates. The multiple group LCS model, imposing strict measurement invariance, fit the data well. Results indicated a decline in aggressive behavior (the latent mean of the aggression change score in the intervention group, M = −0.23, differed significantly from 0; p = 0.13), but no change in victimization. Boys scored higher on aggression at time 1 and had lower decreases over time. Age did not have any effects (for details, see Strohmeier et al. 2012). A further analysis using the Handling Bullying Questionnaire (HBQ) showed that teachers in the intervention group used more non-punitive strategies to work with bullies and more strategies to support victims compared to teachers in the control group (Strohmeier et al. 2012). We also investigated the short-term effect of the ViSC program on cyberbullying and cybervictimization. Results of a multiple group bivariate LCS model, controlling for traditional aggression, traditional victimization, and age showed effectiveness for both cyberbullying (latent d = 0.39) and cybervictimization (latent d = 0.29; for details, see Gradinger et al. 2014).

The evaluation of the implementation quality of the ViSC program focused on implementation fidelity (the number of conducted units documented by the ViSC coaches) and participant responsiveness (participation rates of the teaching staff; Schultes et al. 2014). There was a high variability for both scores: implementation fidelity ranged from 0.4 (conduction of 40 % of the prescribed training units) to 2.0 (conduction of twice as many training units as prescribed in the curriculum); the range of participation rates lay between 30 and 100 %. Multilevel analyses showed that teachers’ self-efficacy was significantly more enhanced in schools where the ViSC program had been implemented with high fidelity and that only teachers with high participant responsiveness significantly changed their behavior in bullying situations (for details, see Schultes et al. 2014). We used these results to adapt the training of the ViSC coaches. In a participatory approach together with coaches and school principals, we worked out what conditions are necessary for implementing the ViSC program with high fidelity and high participant responsiveness. In further analyses, implementation fidelity and participation rates will be considered. Currently, the ViSC program is being implemented in Romania and Cyprus by local researchers. Initial evaluations show promising results.

Our intention was not only to implement a program to prevent violence, but also to enable principals and teachers to assess and interpret violence rates in their schools and classrooms as well as to evaluate the effectiveness of interventions against violence without depending on the presence of researchers for data collection, analysis, and interpretation. Therefore, we developed the self-evaluation tool AVEO (Austrian Violence Evaluation Online Tool), which provides information about violence rates from the perspective of both, students and teachers (Schultes et al. 2011; Spiel et al. 2011a, b, c; see also Spiel et al. 2012). The teacher and school perspectives were systematically integrated into the development of the self-evaluation tool and its implementation was carefully evaluated.