Evaluation of Daily Physical Activity (DPA) policy implementation in Ontario: surveys of elementary school administrators and teachers

Two cross-sectional online surveys were conducted between March and June, 2014 – one among school administrators and the other with teachers, based on a representative random sample of Ontario elementary schools and classrooms. Ethical approval to conduct the study was received from PHO’s Ethics Review Board (ERB) in November 2013 (ERB ID: 2013-039.01). Administrators and teachers were required to provide informed consent in order to participate in the surveys.

A stratified and nested random sampling strategy was used to identify the study sample. The sampling frame included all publicly funded Ontario elementary schools, from which a random sample of 532 schools was selected. The random sample of schools was designed to be representative of Ontario elementary schools based on the following characteristics: school board language (French versus English); school board type (public versus Catholic); school location (urban versus rural, based on postal code) [46]; and school enrolment size (small - up to 200 students, medium - 201-400, and large – more than 400) [47]. Specifically, the sample was stratified to reflect the proportion of Ontario schools within each category of the four characteristics. School administrators responded on behalf of their schools, in which one school administrator (principal or delegated vice-principal) responded per participating school.

Nested within schools, classrooms were randomly sampled from each of grades 3, 5 and 7, where possible (i.e., classrooms were only sampled from schools in which a school administrator had completed the survey). One homeroom teacher from each of the three grades responded on behalf of their respective classrooms. The three grades were selected to ensure representation from primary, junior and intermediate levels.

Participant recruitment occurred in three stages. First, approval was requested from school boards to conduct the study at the schools sampled within their board. Upon receiving school board approvals (30 of 40 school boards = 75.0 %), principals’ approvals to conduct the study within their schools were requested. One school administrator from each of the 228 schools that provided approval (228 of 532 schools = 42.9 %) was invited to participate in the survey through mailed information letters, followed by emailed invitations. Subsequently, similar mailed information letters and email invitations were sent to one teacher from each of grades 3, 5 and 7 (n = 508) in schools where a school administrator had completed a survey (n = 209). These teachers were randomly selected from a complete list of teaching staff provided by principals or school websites upon approval of the study.

To enhance recruitment and study quality, a stakeholder engagement strategy was deployed. Relevant stakeholders were identified based on their interest in, influence on and importance to the study. These key stakeholders included the Ontario government, our Study Advisory Committee, school board and school staff, principal and teacher associations, and public health units (PHUs) in Ontario. Areas of opportunity or concern for each stakeholder were then noted and this information was organized into a matrix. This matrix determined the relative level of engagement needed with each group of stakeholders and their potential influence on the study. Using this information, activities were developed for each group, including ongoing discussions with the EDU and sending question-and-answer sheets about the study to principal and teacher associations.

Two similar instruments for the respective school administrator and teacher surveys were developed and informed by reviewing relevant items from existing survey instruments, the adapted Chaudoir framework [45], results from previous DPA studies [20], and DPA guidelines and other resource documents from the EDU [48‐51].

The primary outcome was a measure of fidelity to DPA policy implementation [45]. Fidelity was further defined in the context of DPA implementation as six components, based on the six requirements of the DPA policy: 1) duration (minimum of 20 min); 2) frequency (each school day); 3) scheduling (during instructional time); 4) intensity (MVPA); 5) continuity (sustained physical activity for 20 min); and 6) inclusivity (including children with special needs). To measure the outcome, participants were asked to indicate whether DPA had been implemented at least once within their school (for school administrators) or classroom (for teachers) during the 2013-14 school year. If participants responded “Yes”, additional questions about DPA implementation within their school or classroom were asked, including six specific questions regarding fidelity to the six policy requirements described above.

The six fidelity questions asked participants to indicate the frequency (or number of days per week) that the DPA policy requirement in question was typically met or implemented in their school or classroom. Response options for each item ranged from 1–5: 1 = Never/1 day; 2 = Rarely/2 days; 3 = Sometimes/3 days; 4 = Often/4 days; and 5 = Always/5 days. The response option chosen for each individual item represented the fidelity score (out of five) for its corresponding DPA policy requirement. Internal consistency reliability (based on Cronbach’s alpha) of the six initial fidelity scale items was .90 at the school level and .98 at the classroom level, suggesting that internal consistency was high.

Using this scale, consisting of the six policy items, a composite score measuring overall implementation fidelity (out of 30) was calculated. This summed each participant’s fidelity score across the six items. Participants who answered “No” on the initial screening question (“Is DPA currently being implemented in your school/classroom?”), indicating DPA had not been implemented at least once in their school/classroom in 2013-14, were also included in analyses related to DPA implementation fidelity. In those cases, they received a composite score of zero for overall implementation fidelity.

For analysis and reporting purposes, the overall implementation fidelity scores were grouped into two categories. Meets DPA policy requirements (scores 24–30) included participants who indicated that DPA is currently being implemented in their school or classroom and often or always meets policy requirements (averaged a score of at least 4 across each of the six items). Does not meet DPA policy requirements (scores 0–23.9) included participants who indicated that DPA is not currently being implemented in their school or classroom (i.e., score of 0), or is currently being implemented but never, rarely or sometimes meets policy requirements (averaged a score of less than 4 across each of the six items).

Several predictor variables were examined in relation to implementation fidelity, including: school sample characteristics (school board type, school board language, school location, school size); awareness of DPA policy requirements; perceptions of the DPA policy; scheduling and monitoring DPA; organization and instruction of DPA; use of DPA resources and supports; perceptions of barriers to implementing DPA; self-efficacy to carry out DPA activities; and personal characteristics of the respondents (gender, years of experience, prioritization of physical activity in daily personal life). School sample characteristics were obtained from the EDU, while personal characteristics were measured using multiple choice items on the survey instruments. Most other predictor variables were measured on the survey instruments using five-point Likert-type items. For analysis and reporting purposes, similar categories (e.g., agree/strongly agree) were combined in predictor variables with five response categories to make three categories.

Dillman’s Tailored Design Method [52], which has been shown to enhance participation in surveys administered by mail, internet, and mixed methods [35, 52‐54] was applied for data collection. To encourage study participation, a number of procedures for contacting potential participants (i.e., email, mail, and phone) were used. Participants were also provided a $10 gift card to a bookstore in the original letter inviting them to participate.

Both the school administrator and teacher surveys were conducted using FluidSurveys [55]. Each survey instrument consisted primarily of closed-ended questions (e.g., multiple choice and Likert-scale questions) and was designed to take approximately 15 min to complete.

School administrator surveys were administered in multiple waves between February and April 2014, while teacher surveys were administered in waves from April to June 2014. Both school administrator and teacher surveys remained open for 5 weeks, from the first day in which it was sent to each wave. Reminder letters were sent (by email or letter) to participants at weeks two and four of each wave.

School administrators and teachers received similar surveys. They were asked to respond to the same questions regarding DPA implementation; school administrators from a school-level perspective and teachers from a classroom-level perspective. The school administrator survey also included additional questions regarding DPA planning at the school level. Respondents could choose to complete the survey instruments in either official language, English or French.

School administrator and teacher survey data were imported into IBM SPSS version 21 (SPSS Inc., Chicago, IL, USA) for analysis. Univariate analyses captured descriptive characteristics of the sample and determined the distribution of responses for the outcome and selected predictor variables. Binary logistic regression was used with school administrator survey data to examine the relationship between various predictors and implementation fidelity at the school level. For teacher survey data, generalized linear mixed models (GLMM) were used to adjust for school-level clustering effects, when examining the relationship between predictors and implementation fidelity at the classroom level.

Significant p-values (p < 0.05) from the logistic regressions and GLMMs were reported to demonstrate the association between predictor and outcome variables at the school and classroom levels, respectively. All findings were reported at an aggregate (provincial) level.

Of the 532 school administrators invited to participate in the study, 209 responded to the school administrator survey for a response rate of 39.3 %. Of these respondents, the majority were female (67.9 %), and more than half (59.0 %) have had 6–15 years of experience in their role. Most (76.2 %) reported having little or no training in HPE, although 60.2 % said physical activity was a high priority in their daily life. Of the 508 teachers invited to participate in the teacher survey, 307 responded, yielding a response rate of 60.4 %. Most were female (71.9 %), and about half (49.6 %) have had 6–15 years of experience in teaching. While most (80.8 %) reported having little or no previous PE training, just under two-thirds (62.5 %) reported physical activity as being a high priority in their personal lives. The distribution of participant and school characteristics, for both school administrator and teacher responses, is shown in Table 1. The distribution of the final analytic sample was found to be similar to the distribution of all publicly funded elementary schools in the province, based on school board language, school board type, school location, and school size.

Findings from the surveys of administrators and teachers indicated incomplete and inconsistent DPA implementation. Notably, a higher percentage of school administrators (61.4 %) indicated that their school was meeting the DPA policy requirement than teachers at the classroom level (50 %). These apparent differences occurred overall and for most of the specific components of the policy (Fig. 2). Reported implementation fidelity, at both the school and classroom levels, was lower for both intensity (MVPA) and continuity (sustained physical activity), compared to other components. At the school level, 58.0 % of school administrators reported meeting the intensity component, while 44.4 % reported meeting the continuity component. Similarly, 59.2 % and 42.2 % of teachers reported meeting these two components, respectively, at the classroom level. In addition, 57.2 % of teachers reported meeting the duration component.

Upon further assessment of implementation fidelity in relation to potential influencing factors (predictors), several significant associations were found. Table 2 presents the overall distribution of school administrator and teacher responses for each predictor explored and their associations with overall implementation fidelity at the school and classroom levels.

The number of years as a school administrator was significantly related to implementation fidelity at the school level (p = 0.017). Specifically, administrators who have worked more than 16 years were more likely to report meeting the guidelines than those who have worked less than 5 years (OR = 5.07, 95 % CI = 1.34–19.24) (not shown in Table 2). However, the strength of this relationship should be interpreted with caution due to the large confidence intervals. At the classroom level, grade level was significantly related to implementation fidelity (p = 0.043). Specifically, grade 5 teachers were more likely to report meeting the guidelines compared to grade 3 teachers (OR = 1.82, CI = 1.02–3.24) (not shown in Table 2). Significant differences in overall implementation fidelity at the school level (p = 0.012) were also found based on board type, with higher fidelity in public schools compared to Catholic schools (OR = 2.22, 95 % CI = 1.19–4.14).

Among school administrators, 81.1 % reported awareness of more than half of the DPA policy components, while awareness of more than half was reported by 62.6 % of teachers. Awareness was significantly related to implementation fidelity at both the school (p = 0.036) and classroom (p = 0.048) levels. Administrators with higher awareness of the DPA policy requirements were more likely to report implementation fidelity in their schools compared to those with lower awareness (OR = 2.17, 95 % CI = 1.05–4.46). This pattern was also found for teachers reporting at the classroom level (OR = 1.63, 95 % CI = 1.00–2.65). This pattern was replicated, in several instances, when comparing awareness with implementation fidelity for specific policy components (for example, awareness of the duration component in relation to implementation fidelity for that specific component).

Approximately two-thirds of school administrators (66.5 %) and teachers (67.0 %) reported that DPA was scheduled in teachers’ timetables. A key finding was the significant relationship between scheduling DPA in teachers’ timetables and overall implementation fidelity at both school (p < 0.0005) and classroom (p < 0.0005) levels. Both school administrators (OR = 4.39, 95 % CI = 2.30–8.39) and teachers (OR = 3.38, 95 % CI = 1.99–5.73) were more likely to report meeting the policy requirements if DPA was scheduled in teachers’ timetables.

Another important finding was the level of DPA monitoring in schools and its significant relationship with overall implementation fidelity at both school (p < 0.0005) and classroom (p = 0.001) levels. Based on survey findings, 72.3 % of administrators and 69.9 % of teachers reported no monitoring procedures for DPA at the school level. However, schools having a DPA monitoring procedure were more likely to have implementation fidelity at the school (OR = 4.73, 95 % CI = 2.09–10.75) and classroom (OR = 4.89, 95 % CI = 2.01–11.90) levels, compared to schools that did not have such a procedure.

The study findings indicated non-significant associations between form of DPA delivery and overall implementation fidelity.

Both school administrators and classroom teachers reported very infrequent use of DPA resources (i.e., learning tools) and supports (i.e., individuals or organizations). Among school administrators, 11.3 % and 8.7 % reported often or always using DPA resources and supports, respectively. Similarly, 10.6 % and 5.3 % of teachers reported often or always using DPA resources and supports, respectively. Use of DPA resources and supports was significantly related to implementation fidelity at the school and classroom levels. A consistent pattern was that those using DPA resources and supports often or always (or even occasionally in some cases) were more likely to report implementation fidelity at both the school and classroom levels (Table 2). However, it is important to note that, while the odds ratios are relatively high, the strengths of these relationships may be weaker since the 95 % confidence intervals are quite large. With respect to communicating with public health units (PHUs) about DPA, 69.9 % of administrators and 94.7 % of classroom teachers indicated that they never or rarely do so. Also, analysis indicated no significant relationships between communicating with PHUs and DPA implementation fidelity at the school or classroom level.

Only 56.0 % of administrators and 43.0 % of teachers perceived the DPA policy to be realistic and achievable. This perception was significantly related to implementation fidelity at both the school (p = 0.001) and classroom (p < 0.0005) levels. Specifically, those agreeing/strongly agreeing that the policy is realistic and achievable were more likely to report greater implementation fidelity at the school (OR = 3.29, 95 % CI = 1.68–6.44) and classroom level (OR = 8.61, 95 % CI = 4.85–15.27) than those disagreeing/strongly disagreeing that the policy is realistic and achievable.

Both administrators and teachers perceived that DPA is associated with a number of benefits, including improvements in students’ physical, social, and emotional well-being, academic outcomes, conduct, and physical activity habits. However, perceptions about these benefits, with one exception (impact on social well-being), were not significantly related to implementation fidelity at the school and classroom levels.

Many school administrators (76.4 %) and classroom teachers (78.5 %) agreed or strongly agreed that competing curriculum priorities were barriers to policy implementation in schools and classrooms respectively. This pattern was also apparent in the case of lack of time, with 61.4 % of administrators and 78.8 % of teachers agreeing/strongly agreeing that this was a barrier to implementation. However, several administrators (47.3 %) agreed/strongly agree that lack of teacher readiness was an important factor, while 63.1 % of classroom teachers agreed/strongly agreed that lack of space was a prominent barrier (Table 3).

In addition, many of the barriers assessed were significantly related to meeting the DPA policy requirement, especially at the classroom level (Table 3). The dominant pattern was that those disagreeing or strongly disagreeing with the prominence of a barrier were more likely to report meeting the DPA requirement compared to those agreeing/strongly agreeing with the prominence of a barrier. For example, administrators (OR = 3.63, 95 % CI = 1.31–10.02) and teachers (OR = 5.54, 95 % CI = 2.29–13.38) who disagreed/strongly disagreed that competing curriculum priorities were a barrier to policy implementation were more likely to report implementation fidelity at the school and classroom level respectively. Similarly, administrators (OR = 5.75, 95 % CI = 2.51–13.19) and teachers (OR = 7.23, 95 % CI 3.02–17.31) who disagreed/strongly disagreed that lack of time was a barrier to policy implementation were more likely to report implementation fidelity at the school and classroom levels respectively. These general patterns were replicated for both administrators and teachers in the case of several additional barriers: lack of teacher readiness, lack of space, and bad weather. In addition, the pattern was replicated specifically for teachers in the case of: lack of equipment, lack of resources, lack of school board support, and lack of amenities.

According to Bandura’s concept of self-efficacy (as part of Social Cognitive Theory), individuals who have confidence in taking action, or in overcoming barriers to take action, are more likely to implement the related behaviour [56]. Findings from the study supported this claim in the case of classroom teachers. Teacher self-efficacy has been shown to be an important predictor of physical activity policy/curriculum implementation in previous studies [27, 30, 39].

Findings from the study also confirm the importance of some additional perceptions that administrators and teachers had, which were significantly related to implementation fidelity. One explanation for the lower perceived priority of DPA among teachers, as compared to administrators, is that teachers may experience more direct constraints on what can be accomplished in their classroom. For example, Patton’s study of DPA implementation found that administrators very much favoured DPA programs while teachers held a more instrumental view, maintaining that implementation lacked logistical supports [38]. In any case, it is interesting to note that study findings revealed that implementation fidelity was higher in classrooms where teachers agreed with the notion that DPA was equally important as other curriculum requirements, while this was not a significant relationship among school administrators.

In developing the study and measurement instrument, resources and supports were conceptualized as facilitating factors in relation to implementation fidelity, as previous studies have indicated their importance [26‐28]. However, the findings indicated that they need to be considered as potential rather than actual facilitators since administrators and teachers reported infrequent use of these. It is unclear whether this was due to a lack of (or lack of awareness of) available resources and supports. In the specific case of resources, perhaps they were not current, or newer resources and learning opportunities were offered on potentially more prominent topics. In any case, the significant association of using resources and supports with implementation fidelity indicates that they could have an important potential role in achieving higher levels of implementation.

A key finding was that scheduling and monitoring were found to be important predictors of DPA implementation fidelity in both schools and classrooms. In, particular, despite low levels of reported monitoring at the school and classroom levels, it was a significant predictor of implementation fidelity. In a prior study of DPA’s development and implementation in Ontario [20], key informants indicated that regular monitoring was a central factor in assessing DPA policy implementation and effectiveness. An important tool for examining policy status, regular monitoring is considered crucial to assessing accountability [14]. It can also contribute to ongoing information for those responsible for implementing DPA, such as school boards, school administrators and classroom teachers.

Some of the most consistent findings were related to perceived barriers to DPA implementation in schools and classrooms such as: competing curriculum priorities; lack of time; lack of teacher readiness; and lack of space. Moreover, perceived barriers were negatively associated with implementation fidelity. A number of similar perceived barriers and related factors, described earlier in this paper, have also been shown to be associated with physical activity policy/curriculum implementation in previous studies [26‐28, 30, 32, 38].

Research in school settings presents a number of challenges and this study was no exception. While DPA is a policy requirement for school boards and schools, research on this topic is not always seen as a high priority among these groups. School boards, school administrators, and classroom teachers are extremely busy and face many requests for research and other projects to consider in addition to their core responsibilities. In the recruitment phase of this study, some administrators expressed hesitation about having their school participate, citing concerns about competing priorities and overburdening staff.

Given these systemic challenges, the research team designed and conducted a study intended to enhance participation and produce the best data possible. In order for study findings to be considered potentially generalizable to the underlying distribution of publicly funded schools in Ontario, a stratified random sample design was used for the school surveys. The final analytic sample compares quite favourably with the underlying distribution of publicly funded elementary schools in the province, providing some evidence that the findings may be representative.

The school-level response rate, based on survey completions by school administrators, was similar to, or better than, response rates achieved in a number of other online school surveys [26, 57]. However, it limits the extent to which findings can be generalized regarding implementation fidelity. It could be argued, for example, that non-participating schools from the sample drawn would be less likely than participating schools to reflect higher DPA implementation fidelity. Thus, findings may be overestimates of implementation fidelity. This would, in effect, underestimate the extent of the implementation problem (although this was not determined).

The school-level response rate resulted in lower analytic sample sizes than planned at the school and classroom levels. This influenced the statistical significance of associations between a number of factors and overall implementation fidelity. Study findings reflect a large number of significant associations between factors (e.g., awareness, scheduling, monitoring, barriers) and overall implementation fidelity. Still, some other relationships may have reached significance with a larger analytic sample size.

To address the challenges of recruiting school administrators and teachers, a number of strategies were applied, including proactive use of the Dillman approach for online surveys [52]. For example, small gifts of appreciation were provided during recruitment, and reminder messages were sent using different modes of communication (e-mail, regular mail, courier, and personal telephone calls). These approaches, along with a stakeholder engagement strategy, contributed to some extent in increasing survey participation.

An additional limitation was that the study was based on self-reported data from school administrators and classroom teachers. More specifically, measures of implementation fidelity were not validated with information from external sources. However, based on a review of existing survey measures, a set of parallel questions was also included to assess information similar to the DPA policy requirements. For example, the survey contained separate questions on the number of minutes of MVPA in DPA sessions. Within the scope of the current study, it was not feasible to have direct observation of DPA activities, objectively measure components such as the duration and intensity of DPA activities, or obtain classroom timetables.

It is assumed that, for the most part, participating school administrators and classroom teachers provided informed and accurate information and assessments. They should be very familiar with these issues. However, it is possible that some participating teachers, for example, may not themselves be delivering DPA for their students. Instead, DPA may be delivered by a PE teacher or someone else. In such cases, the classroom teacher participating in the survey may not have been the most informed about the specific information requested regarding DPA for students in their classroom.

Due to the cross-sectional design of the surveys, the relationships between predictors and implementation fidelity should not be interpreted as causal. Finally, the findings reported in this overview paper were based on descriptive and independent bivariate analyses. Future analyses to be conducted by the research team, once data from the two surveys are merged, will examine these associations together using advanced regression models, including multi-level analysis.