Evaluating school capacity to implement new programs

Abstract

An eight-factor survey-based Bayesian model (Bridge-It) for assessing school capacity to implement health and education programs was tested in secondary analyses of data from 47 schools in the Texas Tobacco Prevention Initiative (TTPI). Bridge-It was used during the pre-implementation phase and again at mid-course of the TTPI 2 years later. Achieved implementation status was evaluated in follow-up almost 4 years after the start of the TTPI. The Bridge-It score aggregated across all eight of the capacity factors predicted both quality of adherence to the Guidelines for School Programs to Prevent Tobacco Use and Addiction and quantity of implementing activity. The school-based leadership factor was an independent predictor of quality of adherence whereas the facilitation processes factor predicted quantity of implementing activity. Integration of Bridge-It, or comparable multi-attribute tools, into the planning and evaluation of school-centered programs can increase understanding of factors that influence implementation and provide guidance for capacity building.

Introduction

Implementation is the “bridge” between a school program and its impact on students and their families (Berman & McLaughlin, 1976; also cited in Dusenbury, Brannigan, Falco, & Hansen, 2003). No matter how effective it proved in the research lab or field trial, a school-centered health or education program cannot produce its intended benefits until it is effectively integrated into actual procedures and practices at the campus level.

Effective implementation, however, is elusive. Too often, promising programs fall into an implementation gap, adopted but not used or only partially integrated into actual practice. Inspection of the research literature suggests as many as half of schools into which innovative and evidence-based programs are introduced fail to implement with sufficient scope or fidelity to expect the promised benefits will accrue to students, families, and communities. This challenge becomes more acute when evidence-based practices and guidelines include program components that span multiple aspects of school functioning, such as policy change and enforcement, classroom instruction, counseling support, and family involvement. For examples, see Berman & McLaughlin, 1978; Buston, Wight, Hart, & Scott, 2002; Elliott & Mihalic, 2004; Eslea & Smith, 1998; Gingiss, Roberts-Gray, & Boerm, 2006; Gottfredson & Gottfredson, 2002; Hahn, Noland, Rayens, & Christie, 2002; Hallfors & Godette, 2002; Kumpfer, 2002; Pentz, 2003; Ringwalt et al., 2003; Taggart, Bush, Zuckerman, & Theiss, 1990).

Part of the explanation for failed or low levels of implementation is lack of capacity at the campus level to implement the new program or technology (Fullan, 2005; Johnson, Hays, Center, & Daley, 2004; Kaftarian, Robinson, Compton, Davis, & Volkow, 2004). Elements of school capacity to implement new programs and policies include support by the principal; the teachers’ sense of efficacy; implementers’ ability to communicate with program participants; staff morale; the general school culture; quality of leadership; availability of time, money, or other resources; presence or amount of turbulence in the implementing environment; and other organizational characteristics (Bosworth, Gingiss, Pothoff, & Roberts-Gray, 1999; Dusenbury et al., 2003; Kallestad & Olweus, 2003). Program characteristics such as complexity and relative advantage; facilitation processes such as adaptive planning and implementer training; and factors external to the school such as state- and district-level policies and mandates are other elements to be considered in understanding and building school capacity to implement a new health or education program (Berman & McLaughlin, 1976; Blake et al., 2005; Boerm, Gingiss, & Roberts-Gray, 2007; Bosworth et al., 1999; Greenberg & Walls, 2003; Han & Weiss, 2005; Payne, Gottfredson, & Gottfredson, 2006; Ringwalt et al., 2003).

To enable and encourage successful implementation of school programs for prevention of tobacco use and addiction, evaluation and planning for the Texas Tobacco Prevention Initiative (TTPI) included a focus on schools’ capacity to implement best practices identified in the Guidelines for School Health Programs to Prevent Tobacco Use and Addiction (school guidelines, Centers for Disease Control and Prevention (1994), Centers for Disease Control and Prevention (2004)). Tobacco use is one of the six health behaviors that contribute most to the leading causes of mortality in the United States (Kann, Brener, & Allensworth, 2001). Frequently these behaviors are established during youth. Nationwide, 22.3% of high school students and 8.1% of middle school students are current cigarette smokers (CDC, 2005). School health education programs play an important role in reducing adolescent tobacco use by increasing student knowledge, positive attitudes and peer resistance skills, and therefore lowering the levels of youth smoking (Engquist et al., 1994). When properly implemented, school programs can lower smoking prevalence by 25–60% (Meshack et al., 2004; National Cancer Policy Board, 2000; Rhode et al., 2001). The school guidelines were designed as a set of recommendations for ensuring a quality school program to prevent tobacco use. These recommendations constitute a “bundle” of program components which necessitate school adaptation and accommodation on multiple dimensions and at multiple levels.

To build schools’ capacity to implement programs consistent with the school guidelines, the TTPI, which is administered through the Texas Department of State Health Services (TDSHS) using funds from the Texas Tobacco Settlement, awarded small ($2000) competitive grants to reimburse program expenses and also provided initial training, guidance, and materials to selected schools in East Texas (TDSHS, 2001). Evaluation of the TTPI included studies to monitor schools’ capacity for achieving and sustaining successful implementation of tobacco prevention and control programs and studies to track implementation status (Gingiss, Boerm et al., 2006; Gingiss, Roberts-Gray et al., 2006; Boerm et al., 2007).

The Bridge-It system (Bosworth et al., 1999) was adopted to assess capacity to implement school programs in the TTPI. Bridge-It originally was developed using an expert panel and an empirical group process technique to integrate the wide range of influential variables into a model to help schools plan for and monitor key elements of the implementation process for school programs. The system includes an 8-factor, 36-item survey to analyze capacity for program implementation and a companion Bayesian model that uses the survey data to estimate the likelihood of implementation success.

The current study uses results from TTPI evaluation studies conducted at three points in time. The baseline or pre-implementation study was conducted to document the then current amount of implementing activity for components of the school guidelines and assess schools’ capacity to achieve and maintain proper implementation of the recommended best practices. Two years later, mid-course assessments were conducted to assess capacity for continued implementation success. Almost 4 years after the pre-implementation assessment, a more comprehensive follow-up assessment of implementation status was conducted using the survey tools adapted from the School Health Education Profile—Tobacco Module (SHEP-TM, Centers for Disease Control and Prevention, 2001). Data from the sequence of three evaluation studies were submitted to secondary analyses to examine the utility and predictive validity of the Bridge-It system. Our hypotheses were that (1) the Bridge-It model for assessing school capacity to implement new health and education programs provides valid predictors of implementation status measured nearly 4 years after program start-up; and (2) Bridge-It's multi-factor approach for measuring capacity has utility because different factors are predictive of qualitative versus quantitative aspects of achieved implementation status.