Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools: a cross-sectional observational study

Participants included 67 Kindergarten-through-third-grade special education teachers and 85 classroom staff (assistants and aides) from 67 classrooms in 52 schools in northeastern United States. Self-contained autism-support classrooms typically have one teacher whose role is to lead all instructional activities and several classroom staff who support instructional activities, which is why there were more classroom staff than teachers. Demographic information for teachers and classroom staff is presented in Table 1.

Seventy-five schools were invited to participate in the study because they had a Kindergarten-third grade autism support classroom. Of these 75 schools, 18 declined to participate and 5 had fewer than three staff working in their autism support classroom or provided substantially missing data (i.e., > 30%) on study measures that prevented data aggregation, resulting in a final sample of 52 schools. Sixty-seven out of 70 teachers (96% response rate) and 85 out of 96 classroom staff (89% response rate) completed data collection.

The university institutional review boards and school district approved the study. The research team met with a school district official to obtain a list of schools with Kindergarten-through-third-grade autism support classrooms. The research team met with the principal at each school to discuss the study activities and obtain a letter of agreement to conduct research. Recruitment materials were distributed to each school, and the research team met with interested participants to inform them about the study and their role as a participant. Once informed consent was obtained, the research team asked participants to complete all study measures. Participants were compensated a $50 check for their time.

Ongoing training was provided to teachers and classroom staff in each EBP (discrete trial training, pivotal response training, visual schedules) for children with autism as part of their standard curricula by a purveyor organization. Didactic group training occurred as part of teachers’ professional development, and monthly in vivo coaching (2 h per/session, once per month) was provided to each classroom in all three EBPs across the school year. Coaches observed teachers’ use of all three EBPs and provided didactic training, modeling, and feedback. For more details on the training, see Pellecchia and colleagues [43]. Discrete trial training is a highly structured, one-on-one instructional strategy that uses massed trials to teach skills where the teacher provides a cue to elicit a response from the child [44]. Pivotal response training uses a naturalistic, play-based approach to address pivotal skills such as motivation and responsivity to the environment to improve language and social skills [46]. Visual schedules involve the use of visual supports to transition students from activity to activity [46]. Although these EBPs are all based on applied behavior analysis [44], the presentation, implementation, and execution of each of these EBPs varies in the classroom [46].

Study data were managed using Research Electronic Data Capture (REDCap), a secure, web-based application designed to support data capture for research studies [48]. We first examined the distribution of and correlations among variables. We examined unadjusted regression models in which the individual and organizational variables were used individually to predict EBP intensity of each practice. We also examined models that adjusted for demographic covariates (teacher education level and teacher age) that predicted EBP intensity with a p value of < .20 [49]. In the adjusted models, teacher education was used as a covariate for predicting discrete trial training intensity and teacher age and teacher education were used as covariates for predicting visual schedules intensity. No demographic factors met criteria to be used as a covariate for predicting pivotal response training intensity.

The a priori sample size calculation was based on 150 participants with α = 0.05, which would allow us to detect relatively small associations (Cohen’s f² = 0.12). Thirty-eight of the schools had 1 teacher in the sample, 12 schools had 2 teachers in the sample, and 2 schools had 3 teachers in the sample. Because the majority of schools only had one autism support teacher in the sample, we used linear regression to examine associations between attitudes toward EBPs and EBP intensity and each organizational-level factor and EBP intensity. Effect sizes were calculated using Cohen’s ƒ² [50].

The average intensity of discrete trial training use was .72 (SD = .74), pivotal response training was .54 (SD = 1.20), and visual schedules was 1.88 (SD = 1.64). Classrooms on average reported using discrete trial training and pivotal response training less than one time per week and visual schedules only for a “few transitions” throughout the week.

In the unadjusted models, the Divergence subscale on the EBPAS accounted for 10% of the variance in discrete trial training intensity, with lower scores significantly associated with higher intensity (β = − .31, R² = .10, p = .02). In the unadjusted models, the Appeal subscale on the EBPAS accounted for 10% of the variance in discrete trial training intensity, with higher scores significantly associated with higher intensity (β = .31, R² = .10, p = .02). No significant associations were found between each individual factor (i.e., Divergence or Appeal) and pivotal response training or visual schedule intensity. The Openness and Requirements subscales on the EBPAS were not significantly associated with intensity of any of the EBPs (see Table 3).

Results were similar in the adjusted models. Lower scores on the Divergence subscale were significantly associated with higher discrete trial training intensity (β = − .34, R² = .15, p = .009, f² = .18). Higher scores on the Appeal subscale were significantly associated with higher discrete trial training intensity (β = .41, R² = .18, p = .002, f² = .22). The Requirements subscale was not significantly associated with discrete trial training intensity (β = .24, p = .07).

In the unadjusted models, the ILS total score was not significantly associated with discrete trial training (β = − .24, R² = .06, p = .06), pivotal response training (β = − .23, R² = .05, p = .07), or visual schedules intensity (β = − .15, R² = .02, p = .24). In the adjusted models, the ILS total score was not significantly associated with discrete trial training (β = − .21, R² = .07, p = .10), or pivotal response training (β = − .23, R² = .05, p = .07). In the unadjusted models, the ICS total score was not significantly associated with discrete trial training (β = − .17, R² = .03, p = .17), pivotal response training (β = − .04, R² = .00, p = .74), or visual schedules intensity (β = − .10, R² = .01, p = .41), and the adjusted model also was not significantly associated with discrete trial training (β = − .15, R² = .05 p = .24) (see Table 3).

Several limitations should be noted. First, the relatively small sample size precluded our ability to examine mediation or moderation of implementation leadership and climate. Mediation or moderation models may help us understand the nuanced relationships between these constructs and the complications of autism EBP implementation in schools. Second, intensity of EBP use was measured using teacher-report, and there was no measure of other aspects of fidelity such as the quality of intervention delivery, which may be related to practitioner attitudes and implementation leadership and climate. Third, while the implementation leadership and climate scales in this study had a minimum of three raters per school, most raters were teachers and classroom staff from special education settings, which may not represent the majority of the school, the broader perspectives of non-special education employees. Special education classrooms represent a small proportion of the overall school—in our study, 38 schools had one classroom represented; therefore, we were limited in the number of raters per school. Further research is needed to explore how organizational constructs can be more broadly and reliably measured in schools [40]. Fourth, the school district in which these data were gathered is one of the largest school districts in the US and represents a racially/ethnically and socioeconomically diverse population of families and students, which may limit the generalizability to other US school districts. Fifth, multiple analyses were conducted for each set of variables, which may capitalize on chance and lead to increased error. Lastly, while the Domitrovich and colleagues [59] framework guided the study aims, district-level variables that may predict successful implementation and sustainment (e.g., policy, financial constraints) were not measured. This was beyond the scope of the current study but should be considered in future research.