Contributions to the Literature
-
Despite widespread enthusiasm for evidence-informed programs (EIP), most EIP implementations fail. Measuring implementation process fidelity during the Pre-Implementation and Implementation Phases—via the Universal Stages of Implementation Completion (SIC)—could tell us why.
-
Sites with high Pre-Implementation fidelity were more likely to deliver EIP services. Implementation fidelity (later in the implementation process) was positively and significantly associated with achieving competency in program delivery, but only if sites had high Pre-Implementation fidelity.
-
To support implementation of EIPs, funders and allies of EIP adoption should accommodate the resources necessary for sites to conduct strong Pre-Implementation.
Background
The role of implementation process fidelity
Pre-Implementation: a critical period
The present study
Method
Measures
Universal Stages of Implementation Completion (SIC)
Phase | Stage | Example activities | |
---|---|---|---|
1. Pre-Implementation | 1 | Engagement | Interest Indicated |
Initial Cost/Resource Info. Sent | |||
2 | Consideration of Feasibility | Feasibility Questionnaire | |
Program Champion Identified | |||
3 | Readiness Planning | Community Partner Meeting | |
Communication Plan | |||
2. Implementation | 4 | Staff Hired and Trained | First Supervisor Selected |
Supervisor Training | |||
5 | Fidelity Monitoring in Place | Fidelity Training Conducted | |
Recording Equipment Tested | |||
6 | Services and Consultation Begin | First Intake Assessment | |
First Intervention Session | |||
7 | Ongoing Program Delivery and Fidelity Monitoring | Supervisor Development Plan | |
Achievement of Intervention Fidelity | |||
3. Sustainment | 8 | Competency (certification) | Rated Competent for Sustainment |
Sites and programs
Program | Sample size | Start-up ratea | ||||
---|---|---|---|---|---|---|
Characteristic | Type | Programs | Sites | Mean | Min | Max |
Focus | Behavioral Health | 15 | 695 | 0.61 | 0.13 | 1.00 |
Child Welfare | 5 | 476 | 0.63 | 0.07 | 1.00 | |
Otherb | 3 | 99 | 0.38 | 0.00 | 1.00 | |
Criminal Justice | 2 | 7 | 1.00 | 1.00 | 1.00 | |
Substance Use | 2 | 10 | 0.61 | 0.22 | 1.00 | |
Population | Adult | 12 | 310 | 0.58 | 0.00 | 1.00 |
Child | 9 | 830 | 0.51 | 0.13 | 1.00 | |
Organizational | 4 | 124 | 0.89 | 0.77 | 1.00 | |
Family | 2 | 23 | 0.80 | 0.60 | 1.00 | |
Setting | Child Welfare | 6 | 203 | 0.67 | 0.07 | 1.00 |
Education | 5 | 214 | 0.50 | 0.13 | 0.79 | |
Justice | 5 | 162 | 0.75 | 0.16 | 1.00 | |
Healthcare | 4 | 105 | 0.37 | 0.00 | 1.00 | |
Community | 3 | 14 | 0.74 | 0.22 | 1.00 | |
Multiplec | 2 | 480 | 0.45 | 0.26 | 0.65 | |
Substance Use | 2 | 109 | 0.88 | 0.77 | 0.98 |
Data analysis strategy
Results
Descriptive statistics
Stage 1 | Stage 2 | Stage 3 | Phase 1 | Phase 2a | ||
---|---|---|---|---|---|---|
Implementation Status | N | Mean (SD) | Mean (SD) | Mean (SD) | Mean (SD) | Mean (SD) |
Start-Up Status | 1287 | 0.72 (0.37) | 0.38 (0.45) | 0.34 (0.42) | 0.41 (0.39) | |
Discontinued | 832 | 0.59 (0.39) | 0.11 (0.26) | 0.06 (0.16) | 0.15 (0.18) | |
Achieved | 455 | 0.97 (0.13) | 0.89 (0.21) | 0.85 (0.23) | 0.88 (0.19) | |
Competence Status | 1105 | 0.69 (0.38) | 0.30 (0.41) | 0.24 (0.38) | 0.33 (0.36) | 0.22 (0.38) |
Discontinued | 943 | 0.63 (0.39) | 0.19 (0.35) | 0.14 (0.28) | 0.23 (0.28) | 0.10 (0.26) |
Achieved | 162 | 0.99 (0.06) | 0.89 (0.22) | 0.87 (0.22) | 0.89 (0.17) | 0.91 (0.12) |
Prediction models
Program start-up and competency by program
Implementation process fidelity and program start-up
Outcome | Model | Predictor | Fixed effects | Variance componentsb,c | ||||||
---|---|---|---|---|---|---|---|---|---|---|
β |
SE
|
p
| 95% CI |
Var
|
SE
| |||||
Start-Upd | ||||||||||
Phase 1
| ||||||||||
Intercept | -1.73 | 0.74 | .019 | [-3.17, -0.29] | 8.86 | 3.83 | ||||
Proportiona | 13.78 | 1.10 | < .001 | [11.63, 15.94] | ||||||
Stage 1
| ||||||||||
Intercept | -0.97 | 1.09 | .373 | [-3.10, 1.16] | 8.70 | 6.08 | ||||
Proportiona | 8.33 | 3.39 | .014 | [1.68, 14.98] | 63.19 | 59.55 | ||||
Stage 2
| ||||||||||
Intercept | -1.51 | 0.38 | < .001 | [-2.25, -0.77] | 1.92 | 0.87 | ||||
Proportiona | 7.87 | 0.47 | < .001 | [6.94, 8.79] | ||||||
Stage 3
| ||||||||||
Intercept | -1.23 | 0.83 | .139 | [-2.87, 0.40] | 11.98 | 5.23 | ||||
Proportiona | 13.33 | 1.11 | < .001 | [11.16, 15.50] | ||||||
Competencee | ||||||||||
Phases 1 & 2
| ||||||||||
Intercept | -4.53 | 0.45 | < .001 | [-5.41, -3.65] | 1.1138 | 0.7959 | ||||
Phase 1a | 3.07 | 0.83 | < .001 | [1.46, 4.69] | ||||||
Phases 1 × 2a | 8.22 | 1.42 | < .001 | [5.45, 11.00] |