Background
Evaluation framework
-
Characteristics of the implementation tool. Although in CFIR, this domain applies to interventions, for this study, we applied it to characteristics of implementation tools. Examples of this category’s constructs include “evidence strength and quality,” “adaptability,” and “complexity.”
-
Outer setting. This domain includes an organization or system’s political and social context. Examples of relevant constructs include “external policy” and “cosmopolitanism” (i.e., the extent to which organizations are networked with other organizations.)
-
Inner setting. This domain includes the structural, political, and cultural context within an organization. Examples of relevant constructs include “leadership engagement” and “available resources.”
-
Characteristics of individuals. In this study, “individuals” include those providing support for the tools (i.e., state and district staff) as well as those using the tools to implement EBIs (school staff). Examples of constructs include “knowledge” and “self-efficacy.”
Methods
Design
Sample
State staff survey
Characteristic | Total (N = 69) | Department of Health (N = 42) | Department of Education (N = 27) | |||
---|---|---|---|---|---|---|
n
| % |
n
| % |
n
| % | |
Highest degree attained | ||||||
Bachelor’s level | 20 | 29.0 | 15 | 35.7 | 5 | 18.5 |
Master’s level | 41 | 59.4 | 25 | 59.5 | 16 | 59.3 |
Advanced degree | 7 | 10.1 | 2 | 4.8 | 5 | 18.5 |
Not reported | 1 | 1.4 | 0 | 0.0 | 1 | 3.7 |
How long in current position | ||||||
Less than 1 year | 9 | 13.0 | 7 | 16.7 | 2 | 7.4 |
More than 1 year, less than 2 years | 10 | 14.5 | 6 | 14.3 | 4 | 14.8 |
More than 2 years, less than 5 years | 20 | 29.0 | 14 | 33.3 | 6 | 22.2 |
More than 5 years, less than 10 years | 15 | 21.7 | 9 | 21.4 | 6 | 22.2 |
10 years or more | 15 | 21.7 | 6 | 14.3 | 9 | 33.3 |
How long in current agency | ||||||
Less than 1 year | 3 | 4.3 | 1 | 2.4 | 2 | 7.4 |
More than 1 year, less than 2 years | 7 | 10.1 | 4 | 9.5 | 3 | 11.1 |
More than 2 years, less than 5 years | 20 | 29.0 | 14 | 33.3 | 6 | 22.2 |
More than 5 years, less than 10 years | 17 | 24.6 | 11 | 26.2 | 6 | 22.2 |
10 year or more | 22 | 31.9 | 12 | 28.6 | 10 | 37.0 |
How long in school health field | ||||||
Less than 1 year | 11 | 15.9 | 8 | 19.0 | 3 | 11.1 |
More than 1 year, less than 2 years | 4 | 5.8 | 3 | 7.1 | 1 | 3.7 |
More than 2 years, less than 5 years | 10 | 14.5 | 9 | 21.4 | 1 | 3.7 |
More than 5 years, less than 10 years | 15 | 21.7 | 13 | 31.0 | 2 | 7.4 |
10 years or more | 29 | 42.0 | 9 | 21.4 | 20 | 74.1 |
State staff interviews
District interviews
Measures
Survey
Interview guides
Analysis
Survey
Interviews
Results
What types of promotion, training, and technical assistance are support systems using?
Promotion
In the last 12 months, has your state engaged in… | CSPAP | HECAT | P4HS | SHG | ||||
---|---|---|---|---|---|---|---|---|
n
| % |
n
| % |
n
| % |
n
| % | |
… marketing/communications related to: | 36 | 83.7 | 15 | 34.9 | 15 | 34.9 | 26 | 60.5 |
... training and/or technical assistance related to: | 34 | 79.1 | 11 | 25.6 | 8 | 18.6 | 17 | 39.5 |
Training
Technical assistance
I think sometimes when we come into the schools and we wanna do a program, we’re saying that they’re missing something. If we can approach them and say you’re doing an awesome job, and you’re doing X, Y and Z, but how about we just take it up a notch.
For what purposes are support and delivery system staff using the CDC’s tools?
Support system use of tools
When used to provide “how to” guidance for implementing EBIs in schools, the tools were drawn on, for example, to guide schools in creating an action plan or communicating with parents.When the physical activity guidelines and all that came out… What it did is, it allowed us to continue moving forward our district policy, wellness policies, but also, it gave it more credibility, because the CDC was saying ‘Look, here it is. Now, here’s how you go about doing it.’
Delivery system use of tools
What contextual factors influenced support system staffs’ use of the CDC’s tools?
CFIR construct | Themes | Exemplar quotations |
---|---|---|
Support system | ||
Outer setting | ||
Cosmopolitanism | Interactions and partnerships with other organizations working to improve school health | “I will say that’s probably our strongest asset we have in [state] is that our partners in all of those different groups, we know and we work and collaborate and communicate on a pretty regular basis.” |
External policy | • Shift from No Child Left Behind to Every Student Succeeds Act • CDC funding requirements and alignment with other federal agencies • State standards/regulation | “The areas that have caused us setbacks would be unintended consequences of No Child Left Behind, in terms of a decrease in physical activity and within our schools. Hopefully now we’ll start seeing that uptick with ESSA and with physical education, health education being considered a well-rounded subject.” |
Characteristics of individuals | ||
Knowledge and self-efficacy | • State staff had greater knowledge and self-efficacy for 2 tools. | For somebody, like me, who has attended multiple trainings and oversees school health as an umbrella, I think it’s a little bit easier to grasp. |
Delivery system | ||
Outer setting | ||
Student and family needs and resources | Parent support for school health | “Sometimes it’s the community or parent part of it. Cuz if you have parents that are really gung ho about making sure their kids are doin’ healthy lifestyles kind of stuff, then they can drive the administration.” |
Inner setting | ||
Culture of wellness | A school’s health-related norms and values | “I think some of it is culture within the school, but also within the community.” |
Relative priority | How nutrition and physical activity were prioritized in relation to academics and other concerns | “Whether it’s bullying or suicide prevention, or tobacco prevention, our guides around nutrition and physical activity are gonna be just one in a whole pool of guides.” |
Readiness for implementation | • Leadership Engagement: Commitment and involvement of district and school leadership • Resources: A paid district wellness coordinator. • Access to knowledge and information. Teachers’ access to professional development related to the tools | “I see a principal as a gatekeeper and if that gatekeeper gets it, lots of these things are gonna be very impactful and effective.” “…being able to have something that works and put programs in place, and then following up to really see the results that come out of that. That’s through those health and wellness coordinators.” “If there isn’t a commitment to providing professional development, or dedicating dollars for training, or doing some sort of stipend for teachers, or covering for substitutes to get teachers trained… Then that policy’s not gonna go anywhere.” |
Characteristics of individuals | ||
Knowledge, beliefs and self-efficacy | School staffs’ limited knowledge of CDC tools | “…your teachers and staff who don’t know about them [CDC tools], don’t know how to use them, or don’t know how to access them.” |
Other personal attributes | School staffs’ motivation, particularly champions | “I think there’s a lot of intrinsic motivation that’s going on with the people who are champions within our region. I think it just takes a certain element of resiliency…” |
Outer setting
Characteristics of individuals
CSPAP (N = 69) | HECAT (N = 69) | P4HS (N = 69) | SHG (N = 69) | |
Aware of tool | 91.3% | 85.5% | 50.7% | 89.9% |
CSPAP (N = 60)* | HECAT (N = 53)* | P4HS (N = 32)* | SHG (N = 51)* | |
Very good or excellent knowledge of tool | 68.3% | 26.4% | 31.3% | 54.9% |
Confident or highly confident to train on tool | 61.6% | 26.4% | 28.1% | 60.8% |
Have received training on tool | 40.0% | 7.5% | 0.0% | 23.5% |
What contextual factors influenced delivery system staffs’ use of the CDC’s tools?
Inner setting
Outer setting
Characteristics of individuals
What characteristics of the CDC’s tools influenced their use?
Characteristics of tools | ||
---|---|---|
Credibility of source | Value of CDC as the source of the tools | “I really appreciate the tools that come out of CDC with them being research-based and best practice. Because with prevention, that’s really all we have behind us is to tell people that it works.” |
Evidence strength and quality | • The tools’ guidance is supported by evidence | “SHG it’s always, for me, been a go-to document. It’s very rich. Rich with the evidence and the strategies” |
Compatibility | • Congruence with state, district, and school priorities, resources, and needs and with tools already using | “In terms of the education component, we never had much of a role in supporting health education. It was always around physical education, because clearly it’s the one area that supports not only physical activity, but physical education and the components of physical education. Also the link to cognitive improvement and academic achievement through physical activity and movement. That’s been our primary focus” |
Complexity | • Tools are long and complicated • State- and district-level staff appreciated tools’ comprehensiveness • Major barrier to school staffs’ use of tools | “It depends on who your end user is, I guess. For somebody, like me, who has attended multiple trainings and oversees school health as an umbrella, I think it’s a little bit easier to grasp. For some of my school people, who are really just looking for the biggest bang for their buck, “What can I do?”, boots on the ground, I think it’s a little bit less user-friendly.” |
Relative advantage | Benefit (or lack of) that CDC tools provided relative to other tools | SPARK book is described as more useable than CDC tool because it “is set up to where you can basically just read it off and have instant activities. You don’t even have to see the lesson. You just look at it be like ‘This is what we’re doing.’ Boom…[SPARK is] super ready to go. You just take one page and one page, put ‘em together.” |
Adaptability | Value of being able to extract and use one or more components of a tool | I really tried to educate our local health departments to go out into the schools and have them do the assessments of the components, and then approach and say it looks like you’re doing three out of the five components of this framework of CSPAP. How can we incorporate or get another component, work on a component for this year? |
Design quality and packaging | Perceptions of how well the tool is assembled and presented | “the user friendly of how it looks, how it can be pieced together, how it can be paired together, [and] the sections.” |