Background
Aim
Methods
-
Consensus approach: We circulated the iCAT dimensions identified as being relevant to describing the complexity of interventions in single, primary evaluation studies [7] (Additional file 2) for feedback and comment to an expert panel of methodologists working in the areas of complex interventions and systematic reviews. Through a series of group and individual telephone calls, we obtained feedback on whether the iCAT tool could be adapted for application in systematic reviews of complex interventions; and how best to integrate the tool into the review process. A total of 18 people participated in the discussions and/or provided feedback on the tool. We then collated and circulated all comments for confirmation of feedback and also invited additional comments.
-
Drafting of first version of the iCAT_SR: Based on the feedback received, we revised the iCAT for single studies to include four additional dimensions seen to be useful in the context of systematic reviews. These were: (1) the degree of interaction between intervention components; (2) the degree to which the effects of the intervention are dependent on the context or setting in which it is implemented; (3) the degree to which the effects of the intervention are changed by recipient or provider factors; and (4) the nature of the causal pathway between the intervention and the outcome it is intended to effect.
-
Second round of feedback on the draft iCAT_SR: We circulated a draft version of the tool, incorporating the agreed amendments, among the expert panel as well as to a wider group of systematic reviewers with an expressed interest in complex interventions. We approached the latter through a ‘complex intervention’ e-mail list. We then invited all those who had contributed to join a meeting at the 2012 Cochrane Colloquium (which around 50 people attended), where the development work was presented and the tool demonstrated through application to two empirical examples. Group discussion following the presentation resulted in additional feedback which informed further refinement of the tool.
Results
Dimensions of the iCAT_SR version 1
Core dimension | Assessment levels and criteria for each dimension | |
---|---|---|
1. Active components included in the intervention, in relation to the comparison |
More than one component and delivered as a bundle
| The intervention includes more than one component and some or all of these components need to be delivered as a bundle. |
More than one component
| The intervention includes more than one component. These components may be integrated into a package. | |
One component
| The intervention includes one component only. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
2. Behaviour or actions of intervention recipients or participants to which the intervention is directed |
Multi-target
| Intervention directed at three or more behaviours or actions. |
Dual target
| Intervention directed at two behaviours or actions. | |
Single target
| Intervention directed at one behaviour or action only. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
3. Organisational levels and categories targeted by the intervention |
Multi-level
| Intervention directed at two or more levels. |
Multi-category
| Intervention directed at two or more categories of individuals within the individual level (e.g. primary care professionals and primary care patients). | |
Single category
| Intervention directed only at single category of individuals within the individual level (e.g. professionals or patients or policy makers). | |
4. The degree of tailoring intended or flexibility permitted across sites or individuals in applying or implementing the intervention |
Highly tailored/flexible
| High degree of variation in implementation from site to site permitted and/or intervention designed to tailor to individuals or specific implementation settings. |
Moderately tailored/flexible
| Some variation in implementation from site to site permitted (i.e. some components of the intervention are tailored/flexible while others are not). | |
Inflexible
| Intervention implementation highly standardised with minimal variation from site to site. | |
Varies
a
| Varies across interventions to be considered for/included in the review | |
5. The level of skill required by those delivering the intervention in order to meet the intervention objectives |
High level skills
| Extensive specialised skills required, i.e. new skills in addition to expected existing skills AND/OR the extension of existing skills to a highly specialised area AND/OR skills requiring extensive additional training. |
Intermediate level skills
| Some specialised skills required, i.e. a small extension to the expected existing skills of professionals, decision makers or consumers. | |
Basic skills
| No specialised skills required. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
6. The level of skill required for the targeted behaviour when entering the included studies by those receiving the intervention, in order to meet the intervention objectives |
High level skills
| Extensive specialised skills required. |
Intermediate level skills
| Some specialised skills required. | |
Basic skills
| No specialised skills required. | |
Varies
a
| Varies across interventions to be considered for/included in the review. |
Optional dimension | Assessment levels and criteria for each dimension | |
---|---|---|
7. The degree of interaction between intervention components, including the independence/interdependence of intervention components |
High level interaction
| There is substantial interaction or inter-dependency between intervention components or actions i.e. the delivery of one intervention component impacts on the delivery of another, resulting in a synergistic effect. |
Moderate interaction
| There is some degree of interaction but no evidence of synergistic effects or dysynergistic effects. | |
Independent
| The intervention has only one component or action, or the components act independently. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
Unclear or unable to assess
| ||
8. The degree to which the effects of the intervention are dependent on the context or setting in which it is implemented |
Highly context dependent
| The effects of the intervention are likely to be strongly dependent on the implementation setting. |
Moderately context dependent
| The effects of the intervention are likely to be transferrable across a limited range of settings only (e.g. only within a specific country or health system). | |
Independent of context
| The effects of the intervention do not appear to be strongly dependent on the implementation setting, i.e. it is anticipated that the effects of the intervention will be similar across a wide range of contexts or settings. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
Unclear or unable to assess
| ||
9. The degree to which the effects of the intervention are changed by recipient or provider factors |
Highly dependent on individual-level factors
| The effects of the intervention are modified by both recipient and provider factors. |
Moderately dependent on individual-level factors
| The effects of the intervention are modified by one of recipient or provider factors. | |
Largely independent of individual-level factors
| The effects of the intervention are not modified substantially by recipient or provider factors. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
Unclear or unable to assess
| ||
10. The nature of the causal pathway between the intervention and the outcome it is intended to effect |
Pathway variable, long
| The causal pathway includes three or more steps between intervention and outcome or occurs over a long time period; is not linear, or is variable; and/or more than one causal pathway has been proposed. |
Pathway linear, long
| The causal pathway is linear but there are three or more steps between intervention and outcome. | |
Pathway linear, short
| The causal pathway is clear, short (only one or two steps), direct, linear. | |
Varies
a
| Varies across interventions to be considered for/included in the review. | |
Unclear or unable to assess
|
Complex interventions can include components that interact synergistically or dysynergistically, as follows: • Synergistic: Intervention components interact in ways that the total effect is greater than the sum of the individual effects of the components. • Dysynergistic: Intervention components act in ways that the total effect is less than the sum of the individual effects of the components. Where intervention components do not interact in these ways, one would expect the effect of the intervention to be the sum of the individual effects of all of the components. Complex interventions can also include components that are interdependent. Where such interdependencies exist, they can be described as: • Contemporaneous: The effect of one intervention component depends on another intervention component being present at the same time. On their own, each component may be less effective, ineffective, or harmful. • Temporal: The effect of one intervention component depends on another component being present beforehand. On their own, each component may be less effective, ineffective, or harmful. Where intervention components do not show interdependency, one would expect these components to be effective regardless of the presence or absence of other components. |
Grading the complexity of interventions using iCAT_SR
Core dimension | Description of the intervention in the review | Judgement | Support for judgement |
---|---|---|---|
1. Active components included in the intervention, in relation to the comparison | ‘Any intervention delivered by LHWs [lay health workers] and intended to improve maternal or child health (MCH) or the management of infectious diseases.’ ([26] p7) |
One component
| The active component is the delivery by a LHW of a health intervention. Although the nature of the intervention delivered and the extent to which LHWs worked with other providers varied considerably across trials included in the review, all interventions were delivered by LHWs. |
2. Behaviour or actions of intervention recipients or participants to which the intervention is directed | ‘Any intervention delivered by LHWs and intended to improve maternal or child health (MCH) or the management of infectious diseases…[]…a MCH or infectious diseases intervention was defined as follows. • Child health: any interventions aimed at improving the health of children aged less than five years. • Maternal health: any interventions aimed at improving reproductive health, ensuring safe motherhood, or directed at women in their role as carers for children aged less than five years. • Infectious diseases: any interventions aimed at preventing, diagnosing, or treating communicable diseases…’ ([26] p7-8) |
Varied
| Included interventions varied from having a single target (e.g., initiation of breastfeeding) to having multiple targets (e.g., community-based interventions directed at hygiene practices, nutrition practices and child caring behaviours among recipients, and intended to reduce neonatal mortality). |
3. Organisational levels and categories targeted by the intervention | ‘There were no restrictions on the types of patients or recipients for whom data were extracted.’ ([26] p7) |
Single category
| The interventions delivered by LHWs were directed at individual patients or community members, or groups of patients or community members, within communities or primary care. |
4. The degree of tailoring intended or flexibility permitted across sites or individuals in applying or implementing the intervention | ‘Any intervention delivered by LHWs and intended to improve maternal or child health (MCH) or the management of infectious diseases.’ ([26] p7) |
Varied from inflexible to highly flexible
| Because the review included any intervention delivered by LHWs and intended to improve MCH or the management of infectious diseases, the range of included interventions was very wide. Some interventions were implemented in a highly standardised way (e.g., structured telephone support for pregnant women from high risk groups [42]) while others allowed variation from site to side or individual tailoring (e.g., provision of health and parenting education to inner city mothers [43]). |
5. The level of skill required by those delivering the intervention in order to meet the intervention objectives | ‘Any lay health worker (paid or voluntary) …[]…For the purposes of this review, we defined the term lay health worker as any health worker who: • performed functions related to healthcare delivery, • was trained in some way in the context of the intervention, but • had received no formal professional or paraprofessional certificate or tertiary education degree.’ ([26] p7) |
Mostly varied from basic to intermediate level skills
| In the studies included in the review, all of the participating LHWs would have received some level of training. In some studies, LHWs received additional training to extend their skills so that they could deliver a specific task or tasks. |
6. The level of skill required for the targeted behaviour when entering the included studies by those receiving the intervention, in order to meet the intervention’s objectives | ‘There were no restrictions on the types of patients or recipients for whom data were extracted.’ ([26] p7) |
Basic skills
| No specialised skills were required of the patients/consumers participating in the trials. |
Optional dimension | Description of the intervention in the review | Judgement | Support for judgement |
7. The degree of interaction between intervention components, including the independence/interdependence of intervention components | The degree of interaction between intervention components was not specified in the review inclusion criteria, described explicitly in the data extraction or analysed as part of the review. The intervention was considered to have only one component for the purpose of the review. |
Unclear or unable to assess
| Not described or analysed in the review. Likely to vary across the included studies. |
8. The degree to which the effects of the intervention are dependent on the context or setting in which it is implemented | ‘A substantial proportion of the included studies…were conducted in LMICs [low and middle income countries] or were directed at low income groups in high income countries. Based on the premise that low income groups across different countries share similar constraints in accessing health care, it may be concluded that these interventions could potentially be extrapolated to other settings, be effective in reaching low income groups, and contribute to reducing health inequalities. However, the degree to which the findings from studies in high income settings can be generalised to low income settings remains unclear and requires further empirical research.’ ([26] p49) ‘While we explored whether there were differences between high, middle and low income countries in the barriers and facilitators we identified, the differences we did find were perhaps surprisingly few…[]… Some differences between settings did emerge, however.’ ([25] p39) |
Moderately to highly context dependent
| The effectiveness review did not address this question but identified it as important to consider in future work. The qualitative evidence synthesis noted that descriptions of study context were limited. The broad categories of high, middle and low income country did not appear to be key in terms of context dependency, but the synthesis identified a wide range of other ways in which the effects of LHW programmes may be dependent on implementation context or setting. |
9. The degree to which the effects of the intervention are changed by recipient or provider factors | Not considered in detail in the reviews. |
Moderately to highly dependent on individual-level factors
| Many LHW interventions are intended to change the behaviour or recipients (e.g. to increase breastfeeding or promote adherence to a treatment). We would therefore expect these interventions to be dependent on recipients’ readiness for behaviour change, their self-efficacy and the social support that they receive. |
10. The nature of the causal pathway between the intervention and the outcome it is intended to effect | ‘…the findings of the qualitative review were organised into chains of events that we proposed could lead to the outcomes measured in the review of effectiveness…’ ([25] p35). |
Pathway variable, long
| More than one causal pathway was identified and each pathway included three or more steps between intervention and outcome. |
Using the iCAT_SR in systematic reviews of effectiveness
Stage in the review processa
| Utility of the iCAT_SR |
---|---|
Formulating the PICO review question and developing criteria for including studies
| Prompts review authors to identify the key components of the intervention/s and how these interact; the actions to which these components are directed; the organisational levels targeted; the anticipated causal pathway/s or logic model etc. Overall, this may help review authors to conceptualise the intervention and define the scope of the review. |
Searching for studies
| By prompting to review authors to identify the key components of intervention/s and the recipients and organisational levels targeted, the tool may aid in identifying appropriate search terms. This may help in identifying eligible studies where, for example, the interventions of interest are broadly similar in terms of their component parts but have widely varying names in the literature. |
Selecting studies for inclusion
| Makes explicit the key components of the intervention/s and the recipients and organisational levels targeted, and therefore helps to ensure that study inclusion decisions are easier and more consistent across the review author team. |
Extracting data
| Facilitates the organisation and standardisation of data relating to intervention description and intervention complexity. The dimensions of the tool can inform development of the data extraction form for the review. |
Analysing data and undertaking meta-analyses
| Enables classification or grouping of interventions for analysis based on their components and/or participants and levels targeted. The tool may also inform analyses and interpretation by helping to generate a priori hypotheses about explanatory factors that could potentially explain differences in results both across studies and across subgroups within studies. These explanatory factors can then be used to explore heterogeneity in subgroup analyses and meta-regressions. |
Presenting results and developing ‘Summary of findings’ tables
b
| Enables classification or grouping of interventions based on their components and/or participants and levels targeted, and thus facilitates clear and logical presentation of the review findings. The tool may also identify important research gaps, for example where the causal pathway of an intervention is not clear or where there are important questions regarding interactions between intervention components. |
Interpreting results and drawing conclusions
| Aids refining a logic model or causal pathway for the intervention/s that was developed at the protocol stage. |