Background
Review protocol level considerations
|
Specify aspects of review question: Population, Intervention, Comparator, Outcome (PICO). Outline how complexities in the PICO will be accommodated in the review. Confounders: the review protocol should pre-specify relevant confounders which are related to exposure to the intervention. Co-interventions: the review protocol should state interventions or exposures which are related to the intervention and prognostic for the outcome of interest.
Study level considerations (to be assessed for each study)
Target trial (TT): define key characteristics (eligibility of participants, intervention, comparator, outcome, effect of interest (see below)) of a hypothetical RCT (this should not be limited by pragmatic or ethical concerns). Effect of interest (EoI): per protocol (PP—starting and adhering to intervention as outlined in intervention protocol) OR intention to treat (ITT—assignment to receive intervention regardless of subsequent exposure or adherence)
Bias domains
Pre-intervention
Domain 1: confounding Domain 2: selection of participants into the study At intervention Domain 3: classification of interventions
Post-intervention
Domain 4: deviation from intended interventions* Domain 5: missing data* Domain 6: measurement of outcomes* Domain 7: selection of reported result* * Domain also part of RoB version 2.0 for RCTs Assessment options for each signalling question (SQ): Yes, Probably Yes, Probably No, No, No Information. Domain level RoB assessment options: Low, Moderate, Serious, Critical, No information. Overall assessment (by outcome): Low, Moderate, Serious, Critical. |
Methods
Results
Protocol level considerations
Protocol/name of included study | Design | Intervention | TT comparison | Effect of interest assessed in study (classification) | Co-interventions/additional co-interventions | Outcome | Participants | Confounders stated in review protocol/additional confounders |
---|---|---|---|---|---|---|---|---|
Actual comparison | ||||||||
Review protocol | – | Warmth and energy efficiency improvements | Eligible for intervention but not in receipt of it | ITT (Classification: baseline) | None | Respiratory health assessed 12 months since intervention (incl self/parent report) | General population eligible to receive intervention | Baseline health; housing quality; socio-economic status; eligibility for intervention |
No intervention/usual care | ||||||||
Braubach et al. [12] | CBA | Thermal insulation improvements | Eligible for intervention but not in receipt of it | No consensus (Classification: unclear) | Unclear: various additional improvements to communal areas, water, and power supply Not clear if balanced across groups. Mixed views on whether these were “important”. | Asthma attacks in past 3 months | Housing agency tenants | External temperature: contrast between baseline and follow-up despite 12 months later |
No intervention: area not selected for intervention | ||||||||
Hopton et al. [14] | CBA | Installation of “Heat with Rent” scheme | Eligible for intervention but not in receipt of it | No consensus (Classification: post hoc) | None | Wheeze in past 12 months (parent report) | Children (<16) in social housing | Length of time in house; reason for moving to house |
No intervention: non-participation in “Heat with Rent” scheme | ||||||||
Walker et al. [16] | CBA | Installation (and possibly repair) of heating system | Eligible for intervention but not in receipt of it | ITT (Classification: baseline) | None | Experience of wheeze in past year (self-report) | Social housing tenants and elderly (> 60 years) private sector households | Age; gender; recent life events; change in smoking exposure; housing type/tenure; central heating at baseline household composition |
No intervention: not eligible for programme (some contamination during study) | ||||||||
Shortt et al. [13] | CBA | Installation of heating system and other energy efficiency measures | Eligible for intervention but not in receipt of it | No consensus (Classification: post hoc) | Unclear: promotion of welfare uptake to intervention and comparison households. Not reported if balanced across groups. Assessed as “important”. | Asthma (self-report) | Vulnerable groups (>65 years, low income and “infirm”) | Housing tenure |
No intervention: not eligible for programme | ||||||||
Somerville et al. [15] | UBA | Installation of heating system | NA | No consensus (Classification: NA) | None | Wheeze by day (parent report) | Children (< 16) with asthma in damp social housing | Smokers in household; pets in household; house type; age |
Uncontrolled |
Study specific target trial (TT)
Effect of interest in target trial and post hoc classification of intervention status
Level of agreement and reasons for lack of agreement in RoB domain assessments
Domain 1: confounding | Domain 2: selection | Domain 3: classification of intervention | Domain 4: deviation from interventions | Domain 5: missing data | Domain 6: measurement of outcomes | Domain 7: selection of reported result | ROBINS-I overall | Cochrane risk of bias (version1) | EPHPP | ||
---|---|---|---|---|---|---|---|---|---|---|---|
Braubach [12] | 3–4 | (0) 3–4 Ŧ | 1 | 1–4 | 2–3 | 2–3 | 2 | 3–4 | Serious–Critical | Critical | Low |
Hopton [14] |
2–4
|
1–4
| 1–2 |
1–3
| 3 | 2–3 | 2 |
2–4
| Moderate–Critical | Critical | Moderate |
Shortt [13] | 3–4 | 3–4 | 1–2 | 2–3 | 2–3 | (0) 2–3Ŧ | 2–3 | 3–4 | Serious–Critical | Critical | Moderate |
Somerville [15] | 4 | 3–4 | 1 | 1–2 | 2–3 | 3 | 2 | 4 | Critical | Critical | Moderate |
Walker [16] | 2–3 |
1–4
| 1–2 | 1–2 | (0) 2–3Ŧ |
1–3
| 2–3 | 2–3 | Moderate–Serious | Critical | Low |
Domain 2: selection of participants into the study
|
• Further guidance on the distinction between SQ2.2 and SQ2.3, (“Were the post-intervention variables that influenced selection: likely to be associated with the intervention (SQ2.2); OR likely to be influenced by the outcome or a cause of the outcome (SQ2.3)”. For analysis relying on post hoc classification of intervention status, this may be difficult to assess but may be a critical source of bias. Also, for interventions that address socio-economic determinants of health, it is highly likely that selection to receive the intervention will be on variables such as income or other measures of socio-economic deprivation, which is also a determinant of the outcome, health. • Start of intervention coinciding with start of follow-up: clarification on how this should be assessed for studies where a baseline assessment of the outcome was made before the participants received the intervention and then at follow-up after the intervention. • Can SQ2.4 (Do start of follow-up and start of intervention coincide for most participants?) be applied to non-event type outcomes? • Clarification of whether variation in lengths of exposure to the intervention at follow-up could contribute to selection bias.
Domain 4: deviation from intended interventions
Effect of interest
• Can analysis of post hoc classification of intervention status be interpreted as per protocol? • Clarification of question “If your aim for this study is to assess the effect of initiating and adhering to the intervention (as in a per protocol analysis)”. Does this relate to the aim for the review as agreed in the review protocol characteristics or the aim of the analysis used in the study being assessed?
Implementation and adherence
• Clarification about what is meant by successful implementation (SQ 4.4) and how authors should decide the level at which implementation failure (SQ 4.4) and adherence (SQ4.5) is assessed (see Table 5). • How should interventions which are tailored to individual need be assessed? • Can it be assumed that there is no implementation failure where classification of intervention status is post hoc? By definition, all those in the intervention group may be assessed to have received the intervention, but this will be dependent on the level at which the intervention is being assessed (see Table 5).
Co-interventions
• Clarification about what constitutes a co-intervention (see Table 5). • Clarification about when a co-intervention should be considered to be “important” (SQ4.3). Should there be an established association with the outcome?
Contamination and switching
• Clarification about the meaning when “contamination” constitutes “switching”, especially in cases where classification of intervention status is post hoc. Guidance implies that contamination is inadvertent but this is difficult to determine in studies, see page 35 of ROBINS-I guidance. |
Domain 1: confounding
Domain 2: selection of participants into the study
Provided below are quotes from two included studies on study aim(s) and interventions
|
Braubach [12]* Study aim: “to assess potential health impacts of improved thermal insulation. Key objectives were to assess the impact of thermal insulation changes on indoor environments, and evaluate potential effects on residents’ health.” Intervention description (may include possible co-interventions): “thermal insulation of all building facades; thermal insulation of the roof/ceiling of highest dwelling; thermal insulation of basement/floor of lowest dwelling; installation of energy-efficient windows where replacements were necessary; installation of new heating systems in buildings with substandard systems. Additional renovation projects without significance for thermal comfort were painting of staircases, installation of intercom systems, new power and water supply systems, improvement of outside spaces/greenery and other repairs as required. However, these renovations are not part of the survey and their impact will not be looked at, although they may improve the general quality of the dwelling significantly.” Shortt [13]* Study aim: “The evaluation focussed on two elements of the process firstly, assessing the benefits to households in terms of indoor environment, health and wellbeing and household income…. This paper focuses primarily on the installation of central heating in selected households and the immediate effect on the dwellings and their occupants” Intervention description (may include possible co-interventions): “installation of central heating systems and other energy efficiency measures in homes…The overall aim was to develop an energy efficiency programme in partnership with key agencies and local communities and as a result to increase energy awareness, increase uptake in grants and reduce the adverse effects on health and well-being caused by cold homes.” The intervention also involved “encouraging higher uptake of social security benefits.” *In both the above examples, the intervention was tailored according to the need of the individual household, but details of this was not reported and was not controlled for in the analysis. In addition, subject to participants’ own resources, the interventions were available to participants regardless of participation in the study, raising the potential for contamination. Again, this was not reported on.
Possible levels of implementation: potential for implementation failure/variation in adherence**
Programme level: selective uptake by eligible households or by external factors e.g. changes in funds available to those delivering intervention Operational level: incomplete delivery or installation of intended intervention(s). Successful implementation may also require an educational component to ensure recipients know how to use intervention effectively and have aspects such as potential benefits and costs explained. Household level: householders in receipt of intervention do not use intervention as intended—heat more rooms but with same cost. Impacts are dependent on behavioural change. **External factors can also affect implementation and/or adherence such as changes in fuel costs. |