A state-wide quasi-experimental effectiveness study of the scale-up of school-wide Positive Behavioral Interventions and Supports

Abstract

The three-tiered Positive Behavioral Interventions and Supports (PBIS) framework promotes the development of systems and data analysis to guide the selection and implementation of evidence-based practices across multiple tiers. The current study examined the effects of universal (tier 1) or school-wide PBIS (SW-PBIS) in one state's scale-up of this tier of the framework. Annual propensity score weights were generated to examine the longitudinal effects of SW-PBIS from 2006–07 through 2011–12. School-level archival and administrative data outcomes were examined using panel models with an autoregressive structure. The sample included 1316 elementary, middle, and high schools. Elementary schools trained in SW-PBIS demonstrated statistically significantly lower suspensions during the fourth and fifth study years (i.e., small effect size) and higher reading and math proficiency rates during the first two study years as well as in one and two later years (i.e., small to large effect sizes), respectively. Secondary schools implementing SW-PBIS had statistically significantly lower suspensions and truancy rates during the second study year and higher reading and math proficiency rates during the second and third study years. These findings demonstrate medium effect sizes for all outcomes except suspensions. Given the widespread use of SW-PBIS across nearly 26,000 schools in the U.S., this study has important implications for educational practices and policies.

Introduction

Positive Behavioral Interventions and Supports (PBIS; Sugai and Horner, 2002, Sugai and Horner, 2006; Sugai, Horner, & Gresham, 2002) is a school-based, multi-tiered prevention framework that integrates data to inform decisions about practices and systems needed in the school to promote positive student behavior. At the universal (tier 1) level, referred to as school-wide PBIS (or SW-PBIS), there is a focus on shifting all staff toward a proactive and positive approach to behavior management and ensuring consistent implementation across all school settings (i.e., classroom and non-classroom). As described in greater detail below, prior PBIS efficacy research has largely focused on SW-PBIS outcomes in elementary schools and has demonstrated significant effects (a) across a range of student behavioral, social emotional, and academic outcomes; (b) student need for additional supports; and (c) school climate (e.g., Bradshaw, Koth, Bevans, Ialongo, & Leaf, 2008; Bradshaw, Koth, Thornton, & Leaf, 2009; Bradshaw, Waasdorp, & Leaf, 2012; Horner et al., 2009). Although the evidence base for PBIS continues to grow, less is known about the effects of SW-PBIS in regular practice when scaled up by a state. Moreover, much of the prior SW-PBIS scale-up research has used correlational designs and lacked a comparison group. Yet, the issue of effectiveness is particularly salient, given that SW-PBIS has been widely disseminated to nearly 26,000 schools across the United States (Horner et al., 2014; Sugai, Horner, & McIntosh, 2016) and internationally. The current study aimed to fill this gap by examining the real-world outcomes of SW-PBIS when scaled-up across a state, using a quasi-experimental non-equivalent control group design (Shadish, Cook, & Campbell, 2001).

PBIS is based on behavioral, social learning, and organizational behavioral principles which, taken together, suggest that shifting the school environment can shape student behavior in a positive way. As adults model positive behaviors, more students will engage in such positive behaviors. As mentioned above, PBIS is a three-tiered prevention framework, where a universal system of supports is integrated with targeted (tier 2) and intensive (tier 3) preventive interventions for students displaying a higher level of need (O'Connell, Boat, & Warner, 2009). Consistent with a public health approach, it is expected that 80% of students within the building will respond to this universal system of behavioral supports, and that the data and systems will be used to identify the roughly 15% of students with a need for more targeted or group intervention and the 1–5% of students in need of individualized and intensive supports (O'Connell et al., 2009). This same tiered framework is commonly used to promote academic learning, whereby the universal curriculum and supports are provided to meet the needs of the majority of the students, and more intensive academic supports are provided at tiers 2 and 3 for students needing greater assistance to develop their skills (Arden, Gruner Gandhi, Zumeta Edmonds, & Danielson, 2017).

Training in multi-tiered PBIS has a strong emphasis on data, systems, and practices across the intervention continuum. SW-PBIS training specifically focuses on data collection regarding implementation of core features of the model, data on behavioral infractions, as well as other data points that can be used as a means for assessing when students respond positively to the universal behavioral supports or may need additional targeted or intensive supports (Horner, Sugai, & Anderson, 2010; Horner, Sugai, Todd, & Lewis-Palmer, 2005). Systems that allow for consistent implementation and collection and analysis of data are needed. This, in turn, informs data-based decision making, the selection and evaluation of discrete teacher practices, and the provision of on-going professional development that the SW-PBIS team provides to all school personnel. The intersection of data, systems, and practices would be expected to be mirrored in any advanced tier implementation efforts as well.

The current study focused on the scaled-up implementation of the universal, school-wide component of PBIS, which explicitly targets the school's systems and procedures to prevent and respond to disruptive behavior, with an emphasis on clarity and consistency. Through training in SW-PBIS, school staff learn to set and teach clear behavioral expectations, implement a system to respond to the meeting of behavioral expectations (i.e., as a means for proactively encouraging desired and preventing undesired behaviors), and create and implement a consistent response system to behavioral infractions for all students across all school settings (Sugai and Horner, 2002, Sugai and Horner, 2006). In doing so, the expectation is that students will engage in fewer disruptions and receive fewer classroom removals, and thus will experience increased time for instruction and learning, which will translate into improved academic performance (Sugai et al., 2002). Thus, improvements in behavior are expected to be proximal outcomes and academic outcomes are expected to be more distal. At the time of data collection for the current study, training and support for the advanced tiers (i.e., 2 or 3) was not systematically or widely available within the state.

A series of randomized controlled trials (RCTs) testing PBIS in elementary schools has provided an evidence base for its efficacy (also see Horner et al., 2010). Specifically, two RCTs conducted in elementary schools provide evidence that tier 1 SW-PBIS was associated with reduced student office discipline referrals and suspensions, improved school climate (Bradshaw, Koth, et al., 2008; Bradshaw, Koth, et al., 2009; Bradshaw, Mitchell, & Leaf, 2010) and improved student academic achievement (Horner et al., 2009). More specifically, the RCT with papers authored by Bradshaw and colleagues demonstrated that the overall referral rate was reduced by approximately 18% in SW-PBIS schools and students in SW-PBIS schools were 33% less likely to receive a referral than students in comparison schools. Further, small- to medium-sized effects were evinced (i.e., ds of 0.10 to 0.30) on measures of climate. Schools implementing SW-PBIS also rated their students as needing fewer specialized support services (Bradshaw, Waasdorp, & Leaf, 2012) and as having fewer behavioral problems (e.g., aggressive behavior, concentration problems, bullying, rejection; Bradshaw, Waasdorp, & Leaf, 2012; Waasdorp, Bradshaw, & Leaf, 2012). The effect sizes for these outcomes also were in the small range. A generalizability study (Stuart, Bradshaw, & Leaf, 2015) leveraged data from this Maryland-based RCT and demonstrated that the positive effects generalized when schools in the trial were weighted to match the characteristics of schools within the state.

A third elementary school RCT involved schools all trained in SW-PBIS and the intervention schools further incorporated an external coach to provide tailored training and implementation support for the student support teaming process and for the implementation of targeted behavioral and engagement interventions (i.e., Tier 2). The aim of this RCT was to examine the effects of multi-tiered PBIS. In this 42-school RCT, teachers in the intervention schools reported small improvements in student need for special education, student academic performance, and their own self-efficacy (see Bradshaw, Pas, Goldweber, Rosenberg, & Leaf, 2012).

In a fourth RCT, in a high school, an external coach assisted in the integration of school climate survey data into the data-based decision-making of PBIS. The coach also offered training and on-going supports in evidence-based programs targeting the universal prevention of bullying or substance use and targeted interventions to improve student engagement or experiences of trauma. Student surveys regarding safety (i.e., weapon carrying, being threatened to be injured with a weapon, skipping school because of a fear of safety) and overall engagement across multiple domains improved by the end of the first year of implementation (see Bradshaw et al., 2014).

State-wide program evaluations of SW-PBIS effectiveness have generally shown promising findings, indicating trends of lower office discipline referrals and suspensions in implementing schools (i.e., no comparison group; Barrett, Bradshaw, & Lewis-Palmer, 2008; Childs, Kincaid, George, & Gage, 2016; Freeman et al., 2016; Muscott, Mann, & LeBrun, 2008). In Maryland, studies regarding the scale-up have most recently focused on the dissemination process and implementation fidelity rather than effectiveness. Specifically, in examining the characteristics of schools and PBIS training and adoption, findings indicated that schools with more suspensions were more likely to be trained in PBIS and schools with greater student mobility and poorer student academic proficiency were more likely to be trained in and to adopt PBIS (i.e., implement and submit implementation data to the state consortium; Bradshaw & Pas, 2011). PBIS implementation fidelity scores were highest in schools that had (a) implemented for a greater number of years and (b) had more certified teachers working in the building, as measured by the Implementation Phases Inventory (i.e., IPI; Bradshaw, Debnam, Koth, & Leaf, 2009; Bradshaw & Pas, 2011). Scores on the IPI were also associated with school-level student outcomes in elementary and middle schools, such that higher IPI scores were associated with higher academic proficiency rates on state standardized math and reading assessments as well as lower rates of truancy (Pas & Bradshaw, 2012).

To our knowledge, there have been two dissemination studies using methodological approaches that have taken steps toward drawing causal inferences (e.g., by minimizing threats to validity such as selection bias) about the effectiveness of SW-PBIS when disseminated within a state in conjunction with district partners. The first was a study conducted in Minnesota among a relatively small sample of trained schools (i.e., 32 elementary and 34 middle schools; Ryoo, Hong, Bart, Shin, & Bradshaw, 2018). A second recent study was conducted across the state of Florida, matching schools implementing SW-PBIS with fidelity with those never trained in SW-PBIS (Gage, Grasley-Boy, George, Childs, & Kincaid, 2019). The Florida study demonstrated that schools implementing SW-PBIS with fidelity had lower suspension rates than non-PBIS schools. However, the Florida study focused solely on one year's data and only examined school discipline outcomes. Thus, the current study fills important gaps in extant literature regarding the effects of SW-PBIS when scaled-up throughout a state and across a wide range of high stakes student outcomes.

Training for PBIS implementation in the United States is provided by the federal Office for Special Education Programs, and the costs of implementing PBIS are relatively low (Horner et al., 2012), which may explain its expansive scaling. Nearly all states in the United States have developed a state- or district-level infrastructure to support its implementation; several other countries are also scaling up PBIS (e.g., Canada, Australia). In Maryland, where the current study was conducted, a coordinated system for implementation of SW-PBIS has been developed over nearly two decades, through collaboration between the Maryland State Department of Education, Sheppard Pratt Health System, and Johns Hopkins University (Barrett et al., 2008; Bradshaw et al., 2014; Bradshaw & Pas, 2011), or the state management team. This collaborative, called the PBIS Maryland Consortium, also has a state leadership team with a representative from each of these agencies as well as from the 24 local education agencies (i.e., school districts) in the state. There is ongoing data collection and evaluation of implementation and outcomes data by the state management team (for details, see Barrett et al., 2008; Bradshaw et al., 2014). During the time frame of this study, there were annual, two-day state-wide offerings of initial SW-PBIS trainings for new teams and booster trainings for returning teams, quarterly full-day state leadership meetings to train district contacts and ensure that state-wide trainings were aligned to their needs, and quarterly full-day SW-PBIS coaches trainings provided to school-based PBIS coaches throughout each school year; all training efforts were led by the PBIS state-level management team (see Barrett et al., 2008). School-based coaches and district leaders (Rogers, 2002; Schoenwald & Hoagwood, 2001) help to promote fidelity and on-going implementation of SW-PBIS. For example, districts offer their own monthly or quarterly coaches' meetings for additional professional development support.

In total, there are currently about 1100 Maryland schools (i.e., pre-k, elementary, middle, high, alternative, special education) trained in SW-PBIS and 855 schools actively implementing SW-PBIS and providing data to the statewide collaborative. The state is now beginning to disseminate training and webinars about implementation of PBIS at the more advanced (i.e., targeted and intensive) tiers for students not responding to SW-PBIS. The data regarding the training status and implementation levels for the current study come from the state's evaluation efforts.

Taken together, extant efficacy research has suggested significant positive effects of PBIS on a range of behavioral and academic outcomes. There has been less consideration of the effectiveness of PBIS within the context of state-wide scaling; however, a recent state-wide study in Florida examined discipline outcomes and reported effects on suspensions (Gage et al., 2019). Most of the available scale-up studies have lacked comparison groups and suffer from threats to validity, including selection bias (Barrett et al., 2008; Bradshaw, Pas, Barrett, et al., 2012; Bradshaw & Pas, 2011; Childs et al., 2016; Freeman et al., 2016). Additional rigorous research that takes steps toward eliminating such threats to validity, and gets closer to drawing causal inferences about the impacts of PBIS when widely disseminated is needed. The current study was designed to fill this important gap in the effectiveness research on PBIS by examining the effectiveness of PBIS on a range of student outcomes when scaled-up within the state of Maryland. Our first aim was to examine the levels of implementation achieved among SW-PBIS schools as a means for confirming that training status in SW-PBIS did in fact lead to school-based implementation and for contextualizing what “regular practice” in Maryland was (i.e., whether adequate fidelity was the norm within the state). Our second aim was to determine whether training in SW-PBIS was associated with improved student outcomes.

A quasi-experimental non-equivalent control group design (Shadish et al., 2001) was selected, in which we leveraged existing archival data and used propensity score weights to approximate a control condition comprised of non-trained schools (Rosenbaum & Rubin, 1983). In other words, although there are differences between schools that selected to be trained and not selected to be trained in SW-PBIS, these differences could be measured and observed. By accounting for the differences in these observed variables and the likelihood that any school would be in the intervention group, we could then weight the data from each school to either contribute more or less information in the outcome analysis. Using propensity score weights also allowed for all schools in the state to remain in the outcome analysis; other approaches would result in the dropping of schools that were too dissimilar from other schools, therefore biasing the sample. The data for this study came from the state-wide scale-up and evaluation of SW-PBIS in Maryland public schools, as implemented by existing school personnel. We focused on PBIS training and implementation which occurred in 2006–07 through 2011–12 among public elementary and secondary schools. We hypothesized that schools trained in SW-PBIS would demonstrate lower rates of suspensions and truancy and higher levels of academic proficiency, based both on the findings of prior RCTs (Bradshaw et al., 2010; Bradshaw, Waasdorp et al., 2012; Horner et al., 2009) and non-experimental dissemination studies (Bradshaw & Pas, 2011; Bradshaw, Pas, Barrett, et al., 2012; Childs et al., 2016; Freeman et al., 2016). Based on the conceptual model for change, we also hypothesized that improvements in suspensions may emerge in the earlier years, as this is the most proximal outcome for PBIS, whereas the truancy and academic effects would emerge later.