Introduction
Evidence-based prevention and health promotion interventions offer great promise for improving the well-being of a population. Accordingly, much progress has been made in developing and evaluating health-promoting interventions addressing a range of disorders yielding a substantial portfolio of effective public-health programs. By contrast, less progress has been made in assuring their introduction and sustained use in an effective manner in the community [
1]. The past decades have witnessed the development of robust HIV prevention intervention portfolios as illustrated by the Center for Disease Control’s identification of 44 “best evidence” HIV prevention programs through its “Prevention Synthesis Project” [
2]. Despite significant declines in AIDS-related deaths and lower rates of new HIV infection, the potential impact of evidence-based behavioral prevention programs has been muted because of the difficulties in sustaining effective delivery of these interventions in real-life settings as opposed to effectiveness trials. For interventions addressing children and adolescents, school-based programs are especially appealing given their wide reach to a “captive” audience. Initiating implementation of effective programs within a school system is a critically important and complex first step and sustaining these efforts even more so, and sustaining them with fidelity has proven to be a daunting task [
3-
6].
Sustainability of evidence-based interventions has been identified as a major research priority area for many years, producing some instructive findings [
7,
8]. The extant literature indicates that sustained implementation of evidence-based behavioral interventions is low [
6]. Factors reported to influence sustainability of implementation within a school system include the following: program’s acceptability and perceived relative importance to school administrators and teachers, perceived effectiveness of the intervention, feasibility to implement on an ongoing basis, and flexibility and adaptability [
9-
13].
Studies assessing fidelity of implementation of effective programs (using a range of definitions, criteria, and cutoffs) likewise find “low” levels of fidelity [
14]. Factors associated with
fidelity of implementation include ease of integration of the program into their existing curricular approach/routine [
8,
15] and teacher training in the curriculum [
16-
18]. Teacher characteristics found to be related to fidelity include the following: teacher’s positive attitude towards prevention programs [
19,
20], shorter duration of time as a teacher [
6], and confidence in one’s ability to teach interactive methods [
6]. Teacher perceptions that are important include the following: the intervention reflects the values of and is “from” their community [
21], students’ are engaged in the intervention curriculum, and students are positively impacted by the intervention [
15,
22].
There is evidence that the more similar the intervention delivery is to that which was demonstrated to be effective, the more likely it is that positive outcomes will be realized. A recent systematic review reveals that a majority of studies have found that higher implementation dose and/or implementation fidelity are associated with better program outcomes [
23]. Blakely and colleagues conducted a comprehensive examination of fidelity and effectiveness assessing seven nationally disseminated education and criminal justice projects and found that implementations conducted with high-fidelity were more effective than low-fidelity implementations [
24]. Other studies have found that suboptimal quality delivery of evidence-based programs can result in minimal or null effects [
14]. An evaluation of 14 school-based anti-bullying programs found that the majority of programs yielded non-significant outcomes on measures of self-reported victimization and bullying [
25]. Derzon et al. assessed intervention effectiveness of drug prevention programs and found no significant effect on drug use among program participants [
26]. These studies suggest that inadequate program delivery may overshadow the potential impact of prevention programs [
25,
26]. However, Blakely’s research noted above [
24] did suggest that if adaptation of the curriculum occurs in the form of local
additions to the program, effectiveness appeared to be increased or at least not decreased, supporting the view that
some adaptation may be important in enabling appropriate fit of the intervention in a new setting.
The literature systematically identifying implementation patterns by teachers of effective HIV prevention programs in school settings is sparse, but some relevant studies do exist. Klingner et al. categorized teachers who participated in the implementation of four research-based programs into three implementation groups [
27], and Shin et al. identified five teachers’ delivery profiles based on teacher engagement and delivery techniques [
28]. While informative, these studies are limited by the qualitative nature of data, the modest sample sizes, and the non-systematic methods for identification of the implementation clusters. The current study seeks to fill this research gap by identifying teachers’ implementation patterns using cluster analysis and relating implementation patterns to student outcomes.
Understanding whether and why teachers did and did not implement an intervention as it was designed is important. Critical to improving outcomes is the ability to identify teachers who are at increased risk of not implementing an intervention in a manner likely to bring greatest benefit before teaching the curriculum. Therefore, in the present analyses, we sought to address four questions relevant to implementation of an effective HIV risk prevention program. First, do teachers vary in their degree of implementation of an HIV risk reduction intervention in terms of implementation dose (e.g., how much of the intervention do they deliver) and integrity of implementation (e.g., do they adhere to the activities and presentations as described in the intervention manual)? Second, are groups of teachers who vary in their degree of implementation identifiable prior to intervention delivery? If so, what are the characteristics of these groups? Third, do these characteristics change in response to intervention delivery? Fourth, do student outcomes vary according to either or both implementation dose and fidelity of implementation?
Methods
Study site
From 2011 through 2013, all 80 government elementary schools in the Commonwealth of The Bahamas participated in national implementation of Focus on Youth in the Caribbean (FOYC), including the research component described herein. The 80 schools are located on 14 of the major islands constituting The Bahamas, where more than 98% of the population resides. Forty-six schools were located on the three most populated islands: 25 schools on Island #1 (I-1), 11 schools on Island #2 (I-2), and 10 schools on Island #3 (I-3). The remaining 34 schools were located on eleven small islands. The 80 participating schools housed 208 grade six classes and teachers: I-1 housed 122 (58.7% of the total) teachers; I-2, 27 (13%) teachers; I-3, 16 (7.7%) teachers; and the six small islands housed 43 (20.7%) teachers. The research protocol was approved by the Wayne State University Human Investigation Committee and the Institutional Review Board of the Bahamian Princess Margaret Hospital, Public Hospitals Authority.
HIV prevention program
FOYC was adapted from Focus on Youth (FOY), a ten-session sexual risk reduction program targeting mid-adolescents. FOY and Informed Parents and Children Together (CImPACT) are currently part of the Centers for Disease Control and Prevention’s “Diffusion of Effective Behavioral Interventions (DEBI)” Portfolio. FOYC and FOY are based on a social cognitive model, the Protection Motivation Theory [
29]. The intervention includes games, interactive discussions, stories, and exercises to reinforce main messages and increase knowledge and skills regarding sexual risk avoidance, effective communication, negotiation, and condom use [
30]. Multiple exercises in communication and a decision-making model is applied in most of the sessions illustrating the steps that should guide a child in selecting a course of action. Intervention evaluations showed that the intervention significantly increased Bahamian youth’s HIV/AIDS knowledge, perceptions of their ability to use condoms, and condom use intention [
31,
32] with evidence of increased condom use [
33].
Teacher training
The Bahamian Ministry of Education (MOE) launched 11 teacher training workshops following the protocol used by the DEBI program to train future interventionists in the delivery of FOY including two 4-day-, eight 2-day-, and one 1-day-long workshops. Duration was based on available time in the MOE workshop training schedules. Regardless of the length of the workshop, the teacher training covered the following: 1) review of the need for HIV prevention in The Bahamas, 2) overview of FOYC including the research showing its effectiveness, 3) a walk-through of each of the sessions of FOYC with participation and “role-play” of the core activities considered to be critical to the success of FOYC, and 4) a didactic question-and-answer period regarding contraception and condom use. The longer workshops (4 days compared to 2 days compared to 1 day) provided a more detailed curriculum experience, longer didactic session, and more teacher involvement in role-plays and discussions. All 208 teachers (regardless of attendance at a workshop) were given a copy of the FOYC teacher training manual.
Nearly half of the teachers (49.2%) attended some or all of a FOYC training workshop in 2011/2012, 30.6% completed the workshop, and 23 (11%) teachers reported that they had attended a FOYC workshop before 2010 when they participated in the original FOYC intervention effectiveness trial. Overall, 60.2% of the teachers received training supporting their delivery of the FOYC curriculum, with the remainder receiving no training.
Measures
Implementation dose and fidelity of implementation
To assess implementation, all teachers were asked to complete a Teacher Implementation Checklist specific for each of the eight sessions of FOYC after they had taught the session. The checklist includes all 46 activities in the FOYC curriculum, 30 of which were identified by the developers as “core elements”. The teachers indicated which activities they had and had not taught in each session; implementation dose was defined as the number of the 30 core activities that were taught. For those core activities that they taught, the teachers recorded whether they had modified the format of the activity as outlined in the manual to determine fidelity of implementation. Teachers also recorded the total number of all activities (from among 46) that they taught. The number of sessions taught was calculated by summing the sessions in which a teacher taught one or more core activities; if teachers taught any core activity within a session, it counted as “teaching that session”.
Factors associated with implementation: teachers’ perceptions of the FOYC curriculum content, their prior training experience, and other factors shown to impact intervention implementation and fidelity
At pre- and post-intervention delivery, all participating teachers were asked to complete questionnaires assessing factors described in the prior research summarized in the “Introduction” section as influencing fidelity of intervention implementation. Both pre-implementation and post-implementation questionnaires assessed the following: teachers’ perceptions of the importance of prevention programs, HIV prevention, and FOYC intervention (very important, somewhat important, or not important) for grade six students in their community or schools; teacher’s confidence in teaching the FOYC intervention; teacher’s sense of “ownership” of the curriculum (i.e., perceiving it as a “Bahamian intervention”); and the relative importance of their time spent in teaching FOYC compared to the time spent teaching reading skills in grade six (less important, about the same, more important) [
6,
14,
19,
21].
The
pre-implementation questionnaire also collected information on the teacher’s level of formal education, years as a teacher/guidance counselor, teacher’s attendance at FOYC training workshop, training in interactive teaching, and prior experience of teaching FOYC or other HIV prevention programs. The
post-implementation questionnaire further assessed the role of competing priorities as a reason for not being able to complete delivering the FOYC curriculum and perceived student benefits from FOYC curriculum (very much, somewhat/not at all). Hypothetical relationships among these variables are discussed in a prior publication [
34].
In bivariate analyses, responses were grouped into two or three collapsed categories due to low frequencies in some categories: years as a teacher/guidance counselor (1–5 years, 6–10 years, >10 years), training in interactive teaching (a lot/some, a little/none), perceptions of the importance of HIV prevention (very important and somewhat/not important), and confidence in teaching FOYC (very comfortable and somewhat/not comfortable). The Teacher Implementation Checklist described above asked teachers to record their degree of comfort in teaching the lesson (very comfortable, rather comfortable, or not comfortable) and how many students (most, some, few) appeared to be engaged in the lesson.
Student outcomes
An anonymous curricular assessment instrument (with identifying information only at the level of the school and teacher), adapted by the MOE from a version of the Bahamian Youth Health Risk Behavioral Inventory (BYHRBI) [
30], was administered by the MOE to grade six students at the beginning of the grade six before receipt of FOYC and at the end of grade six. The instrument assessed HIV/AIDS knowledge and preventive reproductive health skills, as well as some perceptions, intentions, and self-reported behaviors. A 15-item scale including true and false statements was used to assess the level of
HIV/AIDS knowledge. The internal consistency of the scale was high (Cronbach’s
α = 0.85). Correct responses were scored 1 and incorrect 0, resulting in a summary score of 0 to 15 for each participant.
Preventive reproductive health skills were assessed through an adaptation of the Condom-use Skills Checklist [
35]. The validated scale includes true and false statements describing the steps of correct condom use from opening a condom pack for use to disposal after use. This six-item scale demonstrated adequate internal consistency (Cronbach’s
α = 0.83). Correct responses were scored 1 and incorrect 0, resulting in a summary score of 0 to 6 for each participant. A three-item scale was used to assess
self-efficacy for using pregnancy/STI prevention methods. All three items employed a yes/no response scale, with one point assigned for each “yes” response. Individual item scores were added to yield a summary score (range 0 to 3). The internal consistency of the scale was 0.81.
Intention to use protection was measured using the question, “if you were to have sex in the next six months, how likely is it that you would use a condom (to prevent yourself from getting HIV)?” Youth rated the likelihood on a five-point Likert scale ranging from 1 (very unlikely) through 5 (very likely).
Analysis
Cluster analysis was used to identify teachers who were at high risk for inadequate implementation FOYC
before they taught the course. Cluster analysis is a person-oriented approach which identifies homogenous groups within a sample of diverse individuals based on similarities across a set of attributes [
36]. A two-step cluster analysis in SPSS 22 based on the two variables was used to define implementation dose (i.e., number of core activities completed) and fidelity of implementation (i.e., percentage of core activities being changed during the implementation). The optimal number of clusters was determined automatically by the statistical software based on Schwarz’s Bayesian information criterion, and the overall quality of the cluster solution was investigated based on the Silhouette measure of cohesion and separation and each variable’s level of importance to the cluster solution [
37].
The differences in teachers’ education, total years as a teacher, attendance of training workshop, training in interactive training, prior experience of teaching FOYC or other HIV prevention programs, and perceptions of importance of HIV prevention or FOYC to grade six youth between the clusters were examined using chi-square test. The number of all activities completed and number of sessions taught in the classroom were compared across the clusters using ANOVA test.
Teachers’ post-implementation perceptions of the importance of HIV prevention, program ownership, student benefits from FOYC intervention, comfort level in teaching intervention curriculum, relative importance of the time spent in teaching FOYC and teaching reading skills, competing priorities, and student engagement were compared across the clusters, using chi-square test (for categorical variables) and ANOVA test (for continuous variables). Differences in teachers’ perceptions of the importance of HIV prevention, program ownership, and comfort level in teaching FOYC at pre-intervention and post-implementation were assessed to determine if these factors changed during the course of teaching the curriculum using McNemar’s test.
To assess differential intervention effects on student outcomes (HIV/AIDS knowledge, preventive reproductive health skills, self-efficacy, intention to use protection) by teachers’ implementation cluster membership, the change over time within each group of students, as well as the differences in change scores between the clusters, were compared using the Student t test and ANOVA with the Tukey honestly significant difference post hoc tests. The association of teachers’ patterns of implementation (High, Moderate, and Low Implementation Groups, with the Low Implementation Group serving as the reference group) with student outcomes was examined using mixed-effects modeling, adjusting for clustering effects of classroom and/or school. Independent variables included teachers’ implementation clusters (high/moderate/low implementers), student’s age, sex, and baseline student outcomes. School and class were included as random effect variables in the model. Regression coefficients were calculated for all variables. All the analyses except the two-step cluster analysis were performed using SAS 9.3 statistical software package (SAS Institute Inc., Cary, NC, USA).
Discussion
That teachers’ cluster into three identifiable groups according to their levels of implementation dose and implementation fidelity and that these groups are significantly associated with student performance outcomes are important findings. Nearly one third of the teachers were identified as high implementers. These teachers taught over 80% of core activities in their classroom and adhered to the format for 86% of the activities outlined in the manual. Low implementers, representing approximately one sixth of the teachers, taught less than one third of core activities and modified most of these taught. About half of the teachers were identified as moderate implementers; these teachers taught less than half of all core activities but they adhered to the format of most of the activities that they taught.
Teachers’ implementation group membership is associated with both prior teaching experience and intervention training experience. Teachers in the High Implementation Group were less likely to have been teaching for more than 10 years than teachers in the other two groups and were more likely to have received training in interactive teaching. These findings are consistent with previous research suggesting that fewer years of teaching experience and confidence in using interactive methods in intervention delivery are positively associated with fidelity of implementation [
6,
39].
Teachers’ full attendance in the FOYC training workshop is associated with High (and Moderate) Implementation Group membership. By contrast, no training or attending only part of the assigned workshop is associated with Low Implementation Group membership. Duration of training was not in and of itself the important factor. This observation is not consistent with previous research suggesting that longer training is associated with higher quality implementation [
18]. Rather,
full attendance was the critical component, suggesting that a workshop can be streamlined provided that the critical components of the intervention are retained in the training. This is important in long-term implementation, as multi-day training programs are expensive for government-supported educational systems and difficult for teachers to accommodate in their schedules.
Teachers’ High Implementation Group membership is associated with teachers’ perceptions of the importance of the FOYC intervention for grade six students, student benefits from the intervention, and program ownership at post-intervention, which are consistent with prior studies [
5,
14,
19,
21]. Pre-intervention perceptions regarding the importance of FOYC and program ownership did not differ by implementation group membership; pre-intervention, all three groups expressed endorsement of the importance of such educational material for students. Teachers in the High Implementation Group retained this perspective, while those in the Moderate Implementation Group experienced some erosion and those in the Low Implementation Group experienced a marked decrease of this endorsement. This finding suggests that something occurred in the latter two groups to undermine their confidence in the curriculum. Whether this decrease resulted in the sub-optimal implementation or their sub-optimal implementation resulted in their changed perception is not known but requires further exploration as potential remedies will vary. It is noteworthy that teachers in the High Implementation Group did not perceive a conflict in terms of other priorities competing with teaching FOYC. Whether this reflects actual differences in the scheduled courses or other obligations of teachers in the High and Low Implementation Groups or it reflects differing perspectives requires further evaluation as implications for intervening will differ.
Teachers in the Low Implementation Group exhibited several distinguishing characteristics pre-intervention compared to teachers in the two higher performing clusters: less training in interactive teaching, limited prior exposure to the FOYC curriculum, failure to attend the full training FOYC workshop, and low levels of comfort in teaching FOYC lessons. Lack of prior training experience and low levels of comfort in teaching intervention activities may increase teachers’ anxiety which in turn has an impact on implementation quality, particularly when prevention programs are perceived as competing with other priorities [
40]. Missing part of a training workshop may contribute to low-quality implementation and/or may reflect disinterest or discomfort (or both) with the curriculum. These warning signs are easily identifiable pre-intervention or during early implementation and should alert the need for additional training or support. More investigation is needed to better understand the reasons for and resolutions to inadequacies in implementation of FOYC. In the interim, strategies could include increasing teachers’ awareness of adolescent risk behaviors and health consequences, engaging in discussions with teachers about the importance of HIV prevention programs in their schools/communities, reviewing the data with teachers regarding curriculum impact on student outcomes, and enhancing teachers’ competency in teaching the intervention curriculum.
Regardless of teachers’ implementation clusters, students’ HIV/AIDS knowledge, reproductive health skills, self-efficacy, and intention to use protection significantly increased from baseline to follow-up. These changes are similar to those found in previous randomized controlled trials [
31,
32]. Our study reveals that quality of implementation was significantly related to student outcomes (better implementation leads to better outcomes). Students whose teachers were in the High and/or Moderate Implementation Groups demonstrated greater improvements in three student outcomes (knowledge, skills, self-efficacy, and intention) compared to students whose teachers belonged to the Low Implementation Group. Students whose teachers were in the High Implementation Group demonstrated greater improvements in HIV/AIDS knowledge and reproductive health skills compared to students whose teachers belonged to the Moderate Implementation Group. These findings are consistent with previous research suggesting that implementation dose and implementation fidelity influences program outcomes [
23].
There are several potential limitations in this study. First, our findings are based on teachers’ self-reports of their extent of implementation of the FOYC intervention. It is possible that teachers over-reported their level of implementation and provided responses that they thought would be more appropriate. In the current study, trained observers independently observed and assessed approximately 20% of each teacher’s classes. The teacher and observer reports on activities covered in these sessions were compared to determine the level of agreement; in general, we found that the observer-teacher agreement was high (over 80%), indicating that teachers’ self-reports of their implementation of the intervention curriculum in their classrooms are reliable. Second, school-level factors such as support by the principal and school administrator’s perception of importance of HIV intervention were only collected from about half of the participating schools. Thus, these data were not included in the present analysis.
Our study adds to the sparse but emerging literature on the implementation of evidence-based interventions in school settings. Findings regarding inconsistent implementation, three teachers’ implementation groups, and differences in teacher characteristics, teaching, and training experiences and pre- and post-implementation perceptions by implementation groups provide a greater understanding of the barriers and facilitators impacting large-scale implementation of effective intervention programs in schools. The results suggest that additional training or assistance should be provided to those teachers who received minimum training in interactive teaching and in teaching the intervention curriculum, who were not confident in teaching intervention curriculum or held less favorable attitudes towards intervention programs at pre-intervention. Finally, while the questions asked pre-intervention clearly identify at-risk teachers, additional information is needed to identify remedies to the sub-optimal teaching performance. Findings from this study can help interventionists or health practitioners develop better approaches to promote the implementation of effective HIV prevention interventions in schools.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
BW, BS, LD, and XL conceived the initial research questions. BW and BS analyzed the data and drafted the manuscript. LD, SL, LK, SM, XL, and GR helped in interpreting the data. MP, VK, and RA were fully involved in data acquisition, data entry, and maintaining the data base. BS obtained the funding and overall oversight of study. All authors were involved in the critical revision of the paper for intellectual content. All authors read and approved the final manuscript.