Background
Difficulty implementing evidence-based teen pregnancy and STI prevention programs
Getting To Outcomes—an implementation support intervention
Previous efforts to evaluate implementation support
Contributions of the EQUIPS study
Methods
Design overview
Study sites
Making proud choices—an evidence-based pregnancy and STI prevention program
Making proud choices implementation supported by GTO
GTO step | What the GTO manual provides for each step | Practices BGC club staff carried within each GTO step |
---|---|---|
1. Needs: What are the needs to address and the resources that can be used? | Information about how to conduct a needs and resources assessment | Club leaders reviewed data about the needs of a their membership |
2. Goals and outcomes: What are the goals and desired outcomes? | Tools for creating measurable goals and desired outcomes | Each site developed their own broad goals and “desired outcomes”—statements that specify the amount and timing of change expected on specific measures of knowledge, attitudes, behavior |
3. Best practices: Which evidence-based programs can be useful in reaching the goals? | Overview of the importance of using evidence-based programs and where to access information about them | Club leaders reviewed options and choose Making Proud Choices as the evidence-based program to implement |
4. Fit: What actions need to be taken so the selected program fits the community context? | Tools to help program staff identify opportunities to reduce duplication and facilitate collaboration with other programs. | Each site reviewed Making Proud Choices for how it would fit within their club and made adaptations to improve fit |
5. Capacity: What capacity is needed for the program? | Assessment tools to help program staff ensure there is sufficient organizational, human and fiscal capacity to conduct the program | Each site assessed their own capacity to carry out Making Proud Choices and made plans to increase capacity when needed |
6. Plan: What is the plan for this program? | Information and tools to plan program activities in detail | Each site conducted concrete planning for doing Making Proud Choices (e.g., who, what, where, when) |
7. Process evaluation: How will the program implementation be assessed? | Information and tools to help program staff plan and implement a process evaluation | Each site collected data on fidelity, attendance, satisfaction to assess program delivery and reviewed that data immediately after implementation |
8. Outcome evaluation: How well did the program work? | Information and tools to help program staff implement an outcome evaluation | Each site collected participant outcome data on actual behavior as well as on mediators such as attitudes and intentions |
9. Continuous quality improvement: How will continuous quality improvement strategies be used to improve the program? | Tools to prompt program staff to reassess GTO steps 1–8 to stimulate program improvement plans | Each site reviewed decisions made and tools completed before implementation and data collected during and after implementation and made concrete changes for the next implementation |
10. Sustainability: If the program is successful, how will it be sustained? | Ideas to use when attempting to sustain an effective program | Each site took steps such as securing adequate funding, staffing, and buy-in, to make it more likely that Making Proud Choices would be sustained |
Measures and data collection
Amount of GTO implementation support
Program performance
MPC fidelity
Analyses
Overview
Amount of GTO implementation support
Program performance
MPC fidelity
Support for GTO logic model
Type I error control
Results
Amount of GTO implementation support
GTO steps | Year 1b
| Year 2b
| Change from year 1 to year 2 | ||||||
---|---|---|---|---|---|---|---|---|---|
M(SD) | Hedges’ g (95 % CI) | M(SD)b
| Hedges’ g (95 % CI) | Hedges’ g (95 % CI) | Generalized omega-squaredc
| ||||
Control | Intervention | Control | Intervention | Control | Intervention | Difference of differences | |||
1. Needs assessment | 0 | 1.0(0.8) | NA | 0.1(0.1) | 0.52(0.8)* | 0.87 (0.12, 1.61) | NA | NA | NA |
2. Goals | 0 | 0.9(0.7) | NA | 0.1(0.2) | 0.89(0.8)** | 1.43 (0.63, 2.23) | NA | NA | NA |
3. Best practicesa
| 4.5(7.6) | 13.8(7.2)** | 1.25 (0.48, 2.02) | 7.8 (10.7) | 20.2(19.6)* | 0.88 (0.14, 1.63) | 0.35 (–0.36, 1.06) | 0.46 (–0.27, 1.18) | −0.019 |
4. Fit | 1.4(1.3) | 3.4(1.7)*** | 1.87 (1.02, 2.71) | 0.7 (1.2) | 2.1(1.8)* | 1.03 (0.27, 1.79) | −0.87 (−1.61, −0.14) | −1.48 (−2.29, −0.67)*** | 0.004 |
5. Capacity | 0 | 3.7(4.6) | NA | 0.1 (0.2) | 3.7(4.2)** | 1.18 (0.41, 1.96) | NA | NA | NA |
6. Planning | 2.2(2.0) | 6.2(2.6)*** | 1.87 (1.02, 2.71) | 1.2 (1.3) | 6.8(4.4)*** | 1.80 (0.97, 2.62) | −1.19 (−1.96, −0.43) | 0.17 (−0.54, 0.89) | 0.011 |
7. Process evaluation | 0.1(0.1) | 0.7(0.7)** | 1.27 (0.50, 2.04) | 0.1(0.3) | 1.3(1.7)* | 0.92 (0.16, 1.67) | 0.90 (0.17, 1.64) | 0.55 (−0.18, 1.28) | 0.008 |
8. Outcome evaluation | 0 | 0.3(0.6) | NA | 0.1(0.1) | 0.9(1.2)** | 1.10 (0.34, 1.87) | NA | NA | NA |
9. Continuous quality improvement | 0 | 4.8(4.4) | NA | 2.3(2.1) | 15.5(6.4)*** | 2.77 (1.77, 3.77) | NA | NA | NA |
10. Sustainability | 0 | 0.0(0.1) | NA | 0 | 0.4(1.1) | NA | NA | NA | NA |
Total | 8.2(7.9) | 34.8(12.6) *** | 2.63 (1.67, 3.59) | 13.1(13.8) | 64.2(26.7)*** | 2.40 (1.46, 3.34) | 0.35 (−0.36, 1.06) | 1.33 (0.54, 2.12)*** | 0.12*** |
Impact of GTO on performance and fidelity
Program performance
GTO steps | Year 1a
| Year 2a
| Change from year 1 to year 2 | ||||||
---|---|---|---|---|---|---|---|---|---|
M(SD) | Hedges’ g (95 % CI) | M(SD) | Hedges’ g (95 % CI) | Hedges’ g (95 % CI) | Generalized omega-squaredb
| ||||
Control | Intervention | Control | Intervention | Control | Intervention | Difference of differences | |||
1. Needs assessment | NA | NA | NA | NA | NA | NA | NA | NA | NA |
2. Goals | 1.9(0.3) | 2.9(0.6)*** | 2.23 (1.32, 3.14) | 1.7(0.5) | 3.6(0.8)*** | 2.95 (1.85, 4.04) | −0.56 (−1.28, 0.16) | 1.23 (0.39, 2.07)** | 0.20** |
3. Best practices | NA | NA | NA | NA | NA | NA | NA | NA | NA |
4. Fit | 2.1(1.1) | 2.9(0.9)* | 0.88 (0.14, 1.62) | 1.4(0.6) | 3.0(1.6)** | 1.28 (0.46, 2.09) | −0.72 (−1.44, 0.01) | 0.00 (−0.74, 0.74) | 0.003 |
5. Capacity | 1.6(0.8) | 2.9(1.0)*** | 1.49 (0.69, 2.28) | 1.5(0.6) | 3.7(0.9)*** | 2.97 (1.88, 4.07) | −0.19 (−0.91, 0.53) | 1.04 (0.25, 1.83)* | 0.077 |
6. Planning | 1.5(0.6) | 3.0(1.1)*** | 1.70 (0.88, 2.52) | 1.5(0.7) | 3.3(1.4)*** | 1.62 (0.75, 2.49) | 0.01 (−0.71, 0.72) | 0.39 (−0.36, 1.14) | −0.013 |
7. Process evaluation | 1.4(0.6) | 2.7(0.9)*** | 1.70 (0.88, 2.52) | 2.5(0.9) | 3.2(1.2) | 0.74 (−0.03, 1.50) | 1.75 (0.92, 2.58)* | 0.51 (−0.24, 1.27) | 0.025 |
8. Outcome evaluation | 1.2(0.8) | 2.5(0.8)*** | 1.73 (0.90, 2.55) | 1.1(0.5) | 3.2(0.8)*** | 3.05 (1.96, 4.14) | −0.08 (−0.79, 0.62) | 0.88 (0.10, 1.66)* | 0.040 |
9. Continuous quality improvement | 1.6(0.8) | 2.3(0.7)* | 1.00 (0.25, 1.75) | 1.6(1.1) | 2.6(1.0)* | 0.91 (0.13, 1.69) | 0.07 (−0.63, 0.78) | 0.38 (−0.37, 1.13) | −0.019 |
10. Sustainability | 1.4(0.5) | 2.1(0.6)* | 0.77 (0.03, 1.52) | 2.0(1.2) | 3.3(1.2)* | 1.10 (0.30, 1.90) | 0.62 (−0.10, 1.34) | 2.05 (1.12, 2.98)** | 0.022 |
Total | 1.6(0.4) | 2.7(0.6)*** | 2.14 (1.25, 3.02) | 1.7(0.4) | 3.2(0.9)*** | 2.29 (1.34, 3.25) | 0.20 (−0.51, 0.90) | 1.33 (0.51, 2.16)* | 0.035 |
Fidelity
Year 1 | Year 2 | Change from year 1 to year 2, odd ratio (95 % CI) | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Adherence: How well was the MPC activity completed? | Control | Intervention | Control | Intervention | Control | Intervention | Difference of differences | |||||||
Freq | % | Freq | % | Freq | % | Freq | % | 0.97 (0.62, 1.50) | 8.65 (4.64, 16.11)*** | Logistic b = 2.19 (1.43, 2.95)*** | ||||
Fully | 145 | 55.7 | 165 | 57.1 | 74 | 55.0 | 156 | 92.0d
| ||||||
Partially | 83 | 31.9 | 113 | 39.1 | 48 | 36.0 | 12 | 7.0 | ||||||
Not at all | 32 | 12.3 | 11 | 3.8c
| 12 | 9.0 | 1 | 1.0 | ||||||
Number of activity observations | 260 | 100 | 289 | 100 | 134 | 100 | 169 | 100 | ||||||
Quality of delivery (1 = least to 7 = most)a
| M (SD) | Hedges’ g (95 % CI) | M(SD) | Hedges’ g (95 % CI) | Hedges’ g (95 % CI) | Generalized omega-squaredd
| ||||||||
Control (N = 46) | Intervention (N = 51) | Control (N = 25) | Intervention (N = 29) | Control | Intervention | Difference of differences | ||||||||
Classroom control | 4.8 (1.4) | 4.7 (1.5) | −0.08 (−0.48, 0.31) | 4.7 (1.29) | 5.4 (1.10)e
| 0.68 (0.13, 1.23) | −0.13 (−0.62, 0.36) | 0.51 (0.05, 0.97) | 0.066 | |||||
Student interest | 5.2 (1.4) | 5.1 (1.1) | −0.06 (−0.46, 0.34) | 4.9 (1.06) | 5.6 (0.95)e
| 0.71 (0.16, 1.27) | −0.08 (−0.57, 0.42) | 0.89 (0.41, 1.37) | 0.051 | |||||
Facilitator enthusiasm | 5.0 (1.1) | 4.7 (1.2) | −0.22 (−0.62, 0.18) | 4.8 (1.00) | 5.5 (0.69)e
| 0.77 (0.21, 1.32) | −0.28 (−0.77, 0.21) | 0.39 (−0.07, 0.85)** | 0.015** | |||||
Objectives met | 5.0 (1.3) | 5.1 (1.3) | 0.09 (−0.32, 0.49) | 4.9 (1.07) | 6.0 (1.09)e
| 1.18 (0.59, 1.76) | −0.13 (−0.62, 0.36) | 0.70 (0.23, 1.17)** | 0.058** | |||||
Dosageb
| Control (N = 121) | Intervention (N = 200) | Hedges’ g (95 % CI) | Control (N = 103) | Intervention (N = 146) | Hedges’ g (95 % CI) | ||||||||
Percent modules attended, M (SD) | 0.77 (0.24) | 0.73 (0.27) | −0.13 (−0.36, 0.09) | 0.74 (0.29) | 0.69 (0.29) | −0.18 (−0.43, 0.08) | −0.0004 (−0.27, 0.26) | −0.09 (−0.31, 0.12) | −0.025 |
Variable | Measures | Significant results |
---|---|---|
How much GTO was received? | ||
Implementation support | TA hours delivered in year 1 | Intervention group received more TA hours (total and for all GTO steps) than the control group |
TA hours delivered in year 2 | Intervention group received more TA hours (total and for all GTO steps) than the control group | |
Differences between years 1 and 2 | Total TA hours increased for both groups (except for hours spent on GTO step 4), but increased more for the intervention group | |
What impact did GTO have on sites performance and fidelity? | ||
Performance | Year 1 interview scores | Intervention group scored higher on performance (total and for all GTO steps) than the control group |
Year 2 interview scores | Intervention group scored higher on performance (total and for all GTO steps except step 7) than the control group | |
Differences between year 1 and 2 | • Total scores (and scores for steps 2, 5, 8, and 10) increased for both intervention and control groups | |
• Step 2 increased more for the intervention group | ||
• Step 7 increased more for the control group | ||
Fidelity | Year 1 adherence, quality of delivery1, and dosage | • Intervention group had fewer activities “not at all” completed |
• No differences between groups on quality of delivery1 or attendance | ||
Year 2 adherence, quality of delivery1, and dosage | • Intervention group had more activities completed in full and higher ratings of all quality of delivery variables | |
• No differences between groups on attendance | ||
Differences between year 1 to 2 | • Intervention group had more activities rated as completed in full an increase in the quality of delivery (two dimensions: facilitator enthusiasm and objectives met) than in year 1 | |
• Intervention group had a greater increase in activities rated as completed in full and quality of delivery (two dimensions: facilitator enthusiasm and objectives met) than the control group | ||
Is there empirical support for the GTO logic model? | ||
GTO predicts performance (intervention group only) | Year 1 TA hours and interview scores | Not significant |
Year 2 TA hours and interview scores | • TA hours predicted performance of GTO step 9 and total score | |
• All other GTO steps were not significant | ||
Change in TA hours and interview scores between year 1 and 2 | Not significant | |
Performance predicts fidelity (both intervention and control group) | Year 1 performance scores and adherence | Higher performance predicts an increase in the odds of an MPC activity being rated “fully” or “partially” |
Year 2 performance scores and adherence | Higher performance predicts an increase in the odds of an MPC activity being rated “fully” | |
Differences between year 1 and 2 | Not significant |