Contributions to the literature
-
This paper shows how existing methods can be utilized singly and in combination, which is of use to researchers studying translation of evidence-based interventions to practice.
-
This paper illustrates, using a case example, specific methods used in combination to highlight ways to understand implementation through examination of adaptations and their descriptive components.
-
This paper adds to the literature on methods for studying adaptations, which may be useful for better characterizing which combination of adaptation components is associated with successful implementation outcomes.
Introduction
Methods
Case example: the Invested in Diabetes study
Overview
Data collection
Interviews
Fidelity observations
Facilitator field notes
Data analysis
Step | Reason |
---|---|
1. Primary documents (transcripts, notes) were analyzed; all adaptations found were enumerated. | Allowed us to find all adaptations to the implementation process described |
2. Adaptations were entered into a spreadsheet, and each FRAME component was described. | Allowed us to be able to break down and review reasons why adaptations occurred and their intended consequences |
3. Adaptations within each practice and data source were de-duplicated. | Quantitizing adaptations allowed us to gather information on how often certain adaptation components occurred and grouped together. Since adaptations were collected through qualitative methods, there was inherent inconsistency in how much any adaptation was identified within data sources. De-duplication removed the issue of conflating number of mentions with number of adaptations as certain interviews could mention the same adaptation multiple times. Keeping de-duplication within each data source allowed us to understand how adaptations occur in each source. |
4. Adaptations were compared between data sources. | Allowed us to make recommendations on which types of data collection to use and for what scenarios and intended outcomes |
5. Adaptations and their components were enumerated across data sources. | Allowed us to see raw numbers of adaptations/adaptation components discovered in the data |
6. Adaptation components were compared using three approaches: co-occurrence, k-means clustering, and taxonomic analysis. | Allowed us to see groupings of adaptations and adaptation components in order to be able to tell an implementation story |
Identifying adaptations from multiple data sources (step 1)
Categorizing adaptations using a tracking log (steps 2 and 3)
FRAME element | FRAME component choices (as noted from the model) | Use of FRAME components for this study |
---|---|---|
Process | ||
When did the modification occur? | • Pre-implementation/planning • Implementation • Scale-up • Maintenance/sustainment | • As our data were primarily from the implementation phase, we did not see data categorized as pre-implementation, scale-up, or maintenance • Split pre-implementation and implementation into “When did it occur” and “For how long did it occur” to distinguish permanence of adaptation |
Were adaptations planned? | • Planned/proactive (proactive adaptation) • Planned/reactive (reactive adaptation) | • No modifications |
Who participated in the decision to modify? | • Political leaders • Program leaders • Funders • Administrator • Program manager • Intervention developer/purveyor • Researcher • Treatment/intervention team • Individual practitioner • Community members • Recipients | • Changed to reflect relevant roles (researchers, patients, study-involved staff at practice, non-study-involved staff at practice, both researchers and practice staff, other) |
What was modified? | • Content • Contextual • Training and evaluation • Implementation and scale-up activities | • Expanded to reflect study changes (program content, who is involved, recruitment, time devoted, follow-up or tracking, scheduling, reimbursement, resources, other) |
At what level of delivery (for whom/what is the modification made?) | • Individual • Target intervention group • Cohort of individuals • Individual practitioner • Clinic/unit level • Organization • Network system/community | • Modified slightly to reflect relevant players (individual- patient, individual- practice member, practice, study-initiated for intervention arm, study-initiated for entire project) |
Contextual modifications are made to which of the following? | • Format • Setting • Personnel • Population | • Added N/A option |
What is the nature of the context modification? | • 15 selection choices including tailoring, packaging, loosening structure, and “drift” | • Answer choices did not fit well with our study, so we did not categorize, opting to capture as part of the open-ended “What was adapted” |
Relationship fidelity/core elements? | • Fidelity consistent • Fidelity inconsistent | • As we were comparing two study arms, further expanded “Fidelity inconsistent” to outside protocol (condition specific) for when one study arm was altered to look more similar to the other study arm • Added “Became within protocol” for situations where data reflected an outside of protocol change that was brought back within protocol |
Reasons | ||
What was the goal? | • Increase reach or engagement • Increase retention • Improve feasibility • Improve fit with recipient • Address cultural factors • Improve effectiveness/outcomes • Reduce cost • Increase satisfaction | • “Address cultural factors” was merged with “Improve fit with recipient” due to similarities and low number of cultural changes • Added “Outside factors/just happened” and “Other or N/A” |
Reasons (sociopolitical, organization/setting, provider, recipient) | • Sociopolitical • Organization/setting • Provider • Recipient | • Not categorized, captured as part of open-ended “Why was it adapted” with free text |
Not in FRAME | ||
What we added (not in FRAME) | • N/A | • Free text around implications for time, cost, expertise, etc. (i.e., the impact of the adaptation) • Free text around what made the change go well or not go well |
Comparing the data sources (step 4)
Adaptation domain — what was adapted (FRAME) | Degree of concordance/discordance across data sources | Summary of reasons for disparities |
---|---|---|
Follow-up or tracking (Ex: began contacting patients before sessions) | Mean score: 2.3 Counts Completely different (1): 8 Some similarities (2): 2 Mostly similar (3): 6 Same data (4): 4 | • Interview and facilitator note data was similar in sharing stories of changes to participant follow-up and data collection • The most descriptive data was in the facilitator notes with regard to understanding and asking for permission on patient-reported outcomes • This was generally not covered in the observations unless a survey was to be completed during the specific class session in which it could be seen if this was done or not |
Program content (Ex: not all curriculum content covered during session) | Mean score: 2.2 Counts Completely different (1): 4 Some similarities (2): 10 Mostly similar (3): 4 Same data (4): 2 | • Found in all data sources but different emphasis and perspectives • The interview data provided more perceptions of the curriculum and why changes were made, the facilitator notes reported difficulties with the content and the observations noted when content had been altered but not why |
Recruitment (Ex: expanded focus of recruitment past patients with high A1c) | Mean score: 2.4 Counts Completely different (1): 7 Some similarities (2): 2 Mostly similar (3): 7 Same data (4): 4 | • Almost all practices had adaptations in their recruitment strategies, ranging from small changes to completely different strategies. This was discussed in interviews and in facilitator notes, with differences in levels of detail • Almost entirely missing from observations, except one key point (type 1 diabetic patient found) |
Resources (Ex: began utilizing whiteboard) | Mean score: 2.6 Counts Completely different (1): 9 Some similarities (2): 1 Mostly similar (3): 0 Same data (4): 10 | • Mostly, no data reported in any source (4), but when it was reported, tended to come out in either interviews or facilitator notes or both • Observations were lacking in this data |
Scheduling (Ex: changed to weekly sessions from monthly sessions) | Mean score: 2.1 Counts Completely different (1): 9 Some similarities (2): 2 Mostly similar (3): 7 Same data (4): 2 | • Mostly reported in interviews, sometimes in facilitator notes • Not reported in observations |
Time devoted (Ex: classes shorter than 120 min) | Mean score: 1.7 Counts Completely different (1): 13 Some similarities (2): 2 Mostly similar (3): 3 Same data (4): 2 | • Primarily mentioned only in observations, coming across as shorter sessions • Other sources sometimes showed differences in time devoted by administrative personnel |
Who is involved (Ex: class facilitator resigned and replaced) | Mean score: 2.8 Counts Completely different (1): 2 Some similarities (2): 4 Mostly similar (3): 11 Same data (4): 3 | • Some similarities between interviews and facilitator notes • Seemed to come in to play from all sources |
Other (Ex: practice staff began using instant messaging rather than meetings) | Mean score: 2.9 Counts Completely different (1): 6 Some similarities (2): 2 Mostly similar (3): 0 Same data (4): 4 | • Mostly not reported by any source, likely due to good characterization of data • Mostly came from one source for each practice, varied between observations and interviews • Did not occur in observations |
Overall | Mean score: 2.4 Counts Completely different (1): 58 Some similarities (2): 25 Mostly similar (3): 38 Same data (4): 39 | • All data sources had unique data present • Observation data was most relevant for timing of sessions • Facilitator note data had most instances of background for adaptations • Interviews and facilitator notes matched up a lot of the time; observation data was more likely to be independent • 36% of data was completely different between sources |
Identifying the adaptation components and clusters (steps 5 and 6)
Adaptation type (FRAME) | ||||
---|---|---|---|---|
Process/implementation | Classes/program | Total | ||
FRAME adaptation descriptor | Process/implementation | 123 | 0 (N/A) | 123 |
Classes/program | 0 (N/A) | 79 | 79 | |
What was adapted | Follow-up or tracking | 21 | 1 | 22 |
Program content | 0 | 49 | 49 | |
Recruitment | 43 | 1 | 44 | |
Resources | 3 | 5 | 8 | |
Scheduling | 17 | 3 | 20 | |
Time devoted | 7 | 8 | 15 | |
Who is involved | 26 | 12 | 38 | |
Other | 6 | 0 | 6 | |
Why was it adapted | Increase reach/engagement | 36 | 5 | 41 |
Increase retention | 5 | 1 | 6 | |
Improve feasibility | 41 | 8 | 49 | |
Improve fit with recipients | 2 | 13 | 15 | |
Improve outcomes | 6 | 21 | 27 | |
Reduce cost | 6 | 1 | 7 | |
Increase satisfaction | 7 | 10 | 17 | |
Other or N/A | 6 | 12 | 18 | |
Outside forces/ “just happened” | 14 | 8 | 22 | |
Adaptation level (planning) | Planned (proactive) | 13 | 18 | 31 |
Unplanned (reactive) | 110 | 61 | 171 | |
Adaptation level (fidelity) | Within protocol | 116 | 46 | 162 |
Outside of protocol | 5 | 24 | 29 | |
Outside of protocol (condition specific) | 0 | 5 | 5 | |
Became within protocol | 2 | 4 | 6 | |
Delivery level | Individual-patient | 2 | 0 | 2 |
Individual-practice member | 21 | 38 | 59 | |
Practice | 97 | 41 | 138 | |
Intervention group | 0 | 0 | 0 | |
All practices | 0 | 0 | 0 | |
Context modifier | Format | 8 | 46 | 54 |
Setting | 6 | 1 | 7 | |
Personnel | 38 | 12 | 50 | |
Population | 25 | 1 | 26 | |
N/A | 46 | 19 | 65 | |
When did it occur | Proactive — planning | 11 | 9 | 20 |
Reactive — during or after 1st class | 43 | 30 | 73 | |
Reactive — later into implementation (after 2nd) | 29 | 9 | 38 | |
Unclear | 40 | 31 | 71 | |
For how long did it occur | Temporary adaptation | 2 | 8 | 10 |
Permanent adaptation | 103 | 50 | 153 | |
Evolving adaptation | 5 | 1 | 6 | |
Unclear | 13 | 20 | 33 |
Why did the adaptation occur? | |||||||
---|---|---|---|---|---|---|---|
What was adapted? | Fidelity | Improve feasibility (During planning: after planning) | Increase reach/engagement (During planning: after planning) | Increase satisfaction (During planning: after planning) | Improve outcomes (During planning: after planning) | Improve fit (During planning: after planning) | Other/NA (During planning: after planning) |
Content (n = 47) | Consistent (n = 31) | 0:0 | 0:3 | 0:4 | 2:10 | 2:6 | 0:4 |
Inconsistent (n = 16) | 0:0 | 0:1 | 0:2 | 1:2 | 0:2 | 0:8 | |
Who is involved (n = 12) | Consistent (n = 5) | 1:2 | 0:0 | 0:0 | 0:0 | 1:0 | 0:1 |
Inconsistent (n = 7) | 0:1 | 0:0 | 0:1 | 1:2 | 0:0 | 0:2 | |
Time devoted (n = 8) | Consistent (n = 2) | 0:1 | 0:0 | 0:1 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 6) | 0:1 | 0:0 | 0:1 | 0:0 | 0:0 | 0:4 | |
Scheduling (n = 3) | Consistent (n = 3) | 0:1 | 0:1 | 0:1 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Resources (n = 2) | Consistent (n = 2) | 0:0 | 0:0 | 0:0 | 0:1 | 0:0 | 0:1 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Follow-up (n = 1) | Consistent (n = 1) | 0:1 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Recruitment (n = 1) | Consistent (n = 1) | 0:0 | 0:1 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Other/NA (n = 5) | Consistent (n = 5) | 0:1 | 0:0 | 0:0 | 1:1 | 0:2 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 |
Why did the adaptation occur? | |||||||
---|---|---|---|---|---|---|---|
What was adapted? | Fidelity | Improve feasibility (during planning: after planning) | Increase reach/engagement (during planning: after planning) | Increase satisfaction (during planning: after planning) | Improve outcomes (during planning: after planning) | Improve fit (during planning: after planning) | Other/NA (during planning: after planning) |
Recruitment (n = 43) | Consistent (n = 39) | 0:6 | 1:27 | 0:1 | 0:0 | 0:0 | 2:2 |
Inconsistent (n = 4) | 0:0 | 0:2 | 0:0 | 0:0 | 0:0 | 0:2 | |
Who is involved (n = 26) | Consistent (n = 26) | 2:12 | 0:0 | 0:0 | 1:0 | 0:1 | 1:9 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Follow-up (n = 21) | Consistent (n = 21) | 0:10 | 0:0 | 0:5 | 0:5 | 0:0 | 0:1 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Scheduling (n = 16) | Consistent (n = 16) | 1:5 | 0:10 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Time devoted (n = 6) | Consistent (n = 6) | 1:5 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Resources (n = 2) | Consistent (n = 2) | 0:1 | 0:1 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Content (n = 0) (n = 1) | Consistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 |
Inconsistent (n = 0) | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | 0:0 | |
Other/NA (n = 9) | Consistent (n = 8) | 1:3 | 0:0 | 0:0 | 0:0 | 0:1 | 1:2 |
Inconsistent (n = 1) | 0:0 | 0:0 | 0:1 | 0:0 | 0:0 | 0:0 |