Skip to main content
main-content

01.12.2012 | Correspondence | Ausgabe 1/2012 Open Access

BMC Public Health 1/2012

External validity in healthy public policy: application of the RE-AIM tool to the field of housing improvement

Zeitschrift:
BMC Public Health > Ausgabe 1/2012
Autoren:
Hilary J Thomson, Sian Thomas
Wichtige Hinweise

Electronic supplementary material

The online version of this article (doi:10.​1186/​1471-2458-12-633) contains supplementary material, which is available to authorized users.

Competing interests

HT & ST are not aware of any competing interest in this work.

Authors’ contributions

HT & ST selected and assessed each study. HT prepared and analysed the data and wrote the paper with comments from ST. Both authors read and approved the final manuscript.

Background

Improving the use of research findings in policy and practice requires, among other things, clear reporting of external validity items [13]. External validity sometimes referred to as generalisability, means the extent to which causal inferences reported in one study can be applied to different populations, setting, treatments and outcomes [4, 5]. For interventions this requires clear reporting of population characteristics (including setting and reach of the intervention), details of the intervention (including implementation, and adaptation to local settings), outcomes, and sustainability of the intervention and impacts [6, 7]. Improved reporting of these items can assist the reader in judging the applicability and relevance of study findings to their own situation. It has been argued that improved reporting of external validity items may improve the usefulness and the appropriate use of research findings, as well as potentially contributing to improved quality of available evidence [2, 3].
Despite an emerging acknowledgement by publishers of the importance of external validity, there is little clear guidance on what should be reported to facilitate judgements about external validity of study findings [6]. Within the health field much has been devoted to the development of tools to assess internal validity, but far less to external validity. One tool which articulates the required external validity items has been developed by the members of the RE-AIM team [8]. This tool is informed by Cronbach et al’s work on generalisability theory and the related UTOS elements [9]; Units (e.g. individual patients, moderator variables, sub-populations), Treatments (variations in treatment delivery or modality), Occasions (e.g. patterns of maintenance or relapse over time in response to treatments), and Settings (e,g, medical clinics, worksites, schools in which the intervention is being implemented and evaluated). The RE-AIM checklist was developed for public health interventions, specifically within the field of health promotion. While acknowledging the limitations of developing a standard tool the authors of this checklist have called for piloting to test and refine available tools [6]. The tool has been applied to studies of health promotion interventions [10, 11], but to our knowledge, has not been tested in the field of healthy public policy, that is non-health sector interventions such as education, welfare, housing, transport etc.
We used a recent systematic review of the health impacts of housing improvements [12] to investigate levels of reporting of external validity items using the RE-AIM tool [8]. This brief report presents the level of reporting and reflects on how reporting of external validity might need to be adapted for use in the broader field of healthy public policy.

Methods

The Green & Glasgow tool was tailored to meet the characteristics of the studies being assessed. This required some rewording of the original questions to improve clarity or adapting some details of the original questions developed by Green & Glasgow to be more appropriate to a group of housing studies. A set of criteria to assess the extent to which each external validity item had been reported was developed by both authors, i.e. “large extent”, “some extent”, “unclear”, “not at all” or “not applicable (N/A)”. A summary of the items assessed is provided in Table 1. The full version of the tool with the criteria for assessment is available as Additional file 1.
Table 1
Reworded external validity items and extent of reporting by item (n = 39 studies) *
  
Large extent
Some extent
Unclear
Not at all
A Population: Representativeness of target population, setting & reach of intervention
1
Are data presented on variations in participation rate in improved housing interventions by a) setting b) delivery staff/organisations c) residents (for intervention among general target population not study area)
0
0
0
39
2
Is the intended target audience for adoption clearly described
11
18
8
2
3
Is the intended target setting for adoption clearly described?
4
27
5
3
4
Is there analysis of the baseline socio-demographic and ‘condition tested’ (health status) of evaluation participants versus non-participants? (relating to evaluation population only)
0
0
2
37
B Intervention: Implementation & adaptation
5
Are data presented on consistency of implementation of intervention & its different components?
0
2
2
35
6
Are data presented on the level of training of experience required to deliver the programme or quality of implementation by different types of staff?
0
1
1
37
7
Is information reported on whether/how the intervention is modified to individuals/households within the study?
5
6
0
11
8
Are data presented on mediating factors or processes (mechanisms) through which the intervention had an impact?
2
12
4
21
C Outcomes for decision making
9
Are the reported health (even if only one measure of health is comparable) outcomes comparable to wider policy/other studies?
23
14
0
2
10
Have additional outcomes of potential adverse impacts been reported? e.g. socio-economic impacts
4
21
1
13
11
Have authors demonstrated consideration of variation in reported health outcomes (key outcome of interest) by population sub-groups, or intervention setting/delivery staff?
2
4
1
32
12
Is there sensitivity analysis of dose–response/threshold level required to observe health effect (effect on key outcome of interest not proxies)?
3
4
1
31
13
Are data on costs presented? Are standard economic/accounting methods used?
2
19
0
18
D Maintenance and institutionalisation of intervention
14
Are long term effects reported? (12 months or longer since exposure to the intervention)
10
13
4
11
15
Are data reported on the sustainability (or reinvention or evolution) of programme implementation and intervention, at least 12 months after the formal evaluation?
0
0
0
29
16 a
Is the drop-out rate/attrition reported?
19 (Yes)
10 (N/A)
16 b
Are data on attrition by baseline health status of dropouts reported and are analyses conducted of the representativeness of remaining sample at time of final follow-up (or main follow-up time point- as appropriate)?
0
0
0
29 (10 N/A)
(adapted from Green LW, Glasgow RE. Evaluating the Relevance, Generalization, and Applicability of Research: Issues in External Validation and Translation Methodology. Eval Health Prof 2006;29(1):126–153.)
* see Additional file 1 for full details of external validity assessment tool.
Thirty nine intervention studies which had assessed the health impacts of housing improvement and were included in an earlier systematic review [12] were assessed independently by two reviewers for the extent of reporting of the external validity items detailed in the tool, i.e. “large extent”, “some extent”, “unclear”, “not at all” or “not applicable (N/A)”. Disagreements on the assessment between the two reviewers were resolved by discussion and where disagreements persisted the questions or assessment criteria (large extent, some extent etc) were further clarified. Only the studies and related papers included in the published review were included in this assessment, this included the key paper for each study (n = 39) plus a further 29 publications linked with the included studies. Authors were not contacted to obtain further information on external validity items.
Data were extracted and entered onto a Microsoft Access database ©. A summary score for the level of reporting was calculated for each domain (Reach; Implementation; Outcomes; Maintenance). The codes for each item were converted into a numeric value (“large extent” or “some extent” = 1 “unclear” or “not at all” or “N/A” = 0). A sub-total score for each domain and a total score was calculated for each study. The score indicating the level of reporting for each of the external validity items, and summary scores for the domains were tabulated along with key study characteristics identified in the original systematic review, these included intervention type, context, study design, and overall assessment of internal validity (for details of the internal validity assessment and definitions of the study intervention categories see the full systematic review [12]).

Results

Application of Green & Glasgow tool

There was considerable disagreement between the two reviewers requiring substantial iteration to clarify the meaning and purpose of some of the external validity items (Table 1 & Additional file 1). Three items were particularly difficult to clarify (2, 3 & 11) and were reworked to relate to descriptions of the study population or setting and eligibility for the intervention. Eight items were rephrased for clarification and/or inclusion of terms or issues relevant to the field. Item 16 was split into two. Five items (5, 6, 13, 14, & 15) remained unchanged from the original tool developed by Green & Glasgow [8]. Following these edits there was improved agreement between the reviewers but some disagreements persisted and were resolved by discussion. The two items with greatest disagreement were 2 & 3 where half or more of the assessments differed (50% and 68.8% respectively). Levels of agreement were highest for items 5, 6, 15 and 16.

Reporting of external validity in housing improvement studies

Reporting of external validity items was low across the studies (Table 2 & Additional file 2); overall 35.3 % of items were reported (mean 6, range 2–9, median 6). Within each external validity domain (Reach, Implementation, Outcomes, & Maintenance) few studies reported more than half the items either “to some extent” or “to a large extent”. The “outcomes” domain had a greater number of reported items among the studies (mean items reported 49.8%); the “intervention” domain was the most poorly reported (mean items reported 29.0%). No item was universally reported. Item 9 & 16a were most commonly reported. Three items were not reported in any study: items 1, 14, and 15 (Table 2). There was little variation in the number of reported items between intervention type, location or date of study. The better quality studies reported more external validity items (mean number of external validity items reported by Internal Validity Grade A/B/C 6.7/5.36/5.4).
Table 2
Number of external validity items reported in each study by domain
Author year
Study design
Internal validity grade
External validity domains (maximum possible score)
Reach & representat’n (4)
Implementat’n & adaptation (4)
Outcomes (5)
Maintenance & institutionalisat’n (4)
Total (17)
Intervention: Warmth & Energy Efficiency improvements (post 1980) (n = 19)
Heyman et al. 2010
RCT
A
1
2
3
2
8
Braubach et al. 2008
CBA
A
2
1
1
1
5
Howden-Chapman et al. 2008
RCT
A
2
1
1
1
5
Barton et al. 2007
RCT
A
2
1
3
2
8
Howden-Chapman et al. 2007
RCT
A
2
0
3
1
6
Platt et al. 2007
CBA
A
1
1
4
2
8
Lloyd et al. 2008
CBA
B
1
0
3
2
6
Shortt et al. 2007
CBA
B
2
2
2
2
8
Somerville et al. 2000
UBA
B
2
1
3
1
7
Hopton et al. 1996
CBA
B
2
0
2
1
5
Warm Front Study Group 2006
RC
C
1
1
5
0
7
Allen 2005 a
UBA
C
1
1
2
2
6
Allen 2005 b
UBA
C
1
2
1
1
5
Health Action Kirklees 2005
R
C
1
0
1
0
2
Eick et al. 2004
RCT
C
2
1
4
2
9
Winder et al. 2003
UBA
C
1
0
0
1
2
Caldwell et al. 2001
CBA
C
2
2
3
2
9
Green et al. 1999
RC
C
1
0
3
0
4
Iversen et al. 1986
CBA
C
1
0
1
1
3
Mean (range)
  
1.47 (1–2)
0.84 (0–2)
2.37 (0–4)
1.26 (0–2)
5.95 (2–9)
Intervention: Rehousing/retrofitting +/− neighbourhood renewal (post 1995) (n = 10)
Kearns et al. 2008
CBA
A
1
1
3
2
7
Thomson et al. 2007
CBA
A
2
0
2
2
6
Critchley et al. 2004
CBA
A
0
1
4
1
6
Thomas et al. 2005
CBA
B
2
1
3
1
7
Barnes et al. 2003
CBA
B
2
0
2
2
6
Evans et al. 2002
CBA
B
1
0
2
0
3
Blackman et al. 2001
UBA
C
2
0
2
2
6
Wells 2000
UBA
C
2
0
1
2
5
Ambrose 1999
UBA
C
2
1
3
2
8
Halpern 1995
XUBA
C
2
1
2
1
6
Mean (range)
  
1.6 (0–2)
0.50 (0–1)
2.40 (1–4)
1.50 (0–2)
6 (3–8)
Intervention: Provision of basic housing needs/developing country intervention (n = 6)
Cattaneo et al. 2006
RC
B
2
1
4
1
8
Choudhary et al. 2002
RC
B
1
1
3
1
6
Aga Khan Health Service 2001
XCBA
B
1
1
2
1
5
Spiegel et al. 2003
XCBA
C
2
0
1
1
4
Aiga et al. 2002
XCBA
C
1
1
3
0
5
Wolff et al. 2001
XCBA
C
2
0
2
1
5
Mean (range)
  
1.50 (1–2)
0.67 (0–1)
2.67 (1–4)
0.83 (0–1)
5.5 (4–8)
Intervention: Rehousing from slums (pre 1965) (n = 4)
Wilner et al. 1960
CBA
A
2
1
3
2
8
McGonigle et al. 1936
XCBA
B
2
1
4
1
8
Ferguson 1954
RC
C
1
0
1
1
3
Chapin 1938
UBA
C
2
1
4
2
9
Mean (range)
  
1.75 (1–2)
0.75 (0–1)
3.00 (2–4)
1.50 (1–2)
7.00 (3–9)
TOTAL (n = 39 studies) MEAN (range)
  
1.54 (0–2)
0.72 (0–2)
2.49 (0–4)
1.28 (0–2)
6.00 (2–9)
Study design: RCT: Randomised Controlled Trial; CBA: Controlled Before & After; UBA: Uncontrolled Before & After; XCBA: Cross-sectional Controlled Before & After; XUBA: Cross-sectional Uncontrolled Before & After; RC: Retrospective controlled; R: Retrospective uncontrolled.
List of references available in reference 4 or from the author.

Discussion

Following adaptation and development of detailed assessment criteria relevant to the studies being assessed, the external validity tool was successfully applied to studies of housing improvement drawing on the primary paper and associated papers available at the time of the original review [12]. Reporting of external validity items was low overall (median 35.6%) and across individual domains in the tool. This is comparable to the level of reporting in a group of studies of childhood obesity prevention (median 34.5%) [10].
The studies we assessed represented a broad range of study designs, contexts, and other aspects of study quality and interventions, as well as representing both published and unpublished studies. There was no suggestion of a link between study characteristics and reporting of external validity. The apparent link between internal and external validity reporting may be explained by the overlap in assessed items, specifically attrition and sample selection. It is possible that author contact and publications since the original searches in 2008 will have reported additional relevant data.
The domain relating to the intervention (items 5–8) was least often reported; Klesges et al. reported a similar issue [10]. While faithful replication of a novel intervention may depend on detailed reporting of intervention components and implementation [13], this may be less important for a well established intervention, such as housing improvements. Moreover, for complex social interventions, such as housing improvements, data confirming intervention function may be of more value than the details of intervention form [14]. Data on changes effected by the intervention, such as improved warmth, may be used to refine generalisable theories regarding tackling socio-economic determinants of health even where the specific intervention may not be widely generalisable. Where there is evidence to support the theory that changes in an intermediate outcome can lead to health, selection and implementation of appropriate and effective interventions to improve a named socio-economic outcome, such as warmth, may be made locally. This issue is of particular relevance to interventions where the hypothesised health impacts are dependent on the intervention affecting an intermediate variable, for example healthy public policy interventions tackling socio-economic determinants of health.
There is little doubt that reporting of external validity items needs improving. However, in agreement with Green et al, development of a standard tool may not be appropriate [6]. In our study there was poor agreement between the two assessors in the interpretation of the tool. In response it was necessary to amend and clarify meaning to allow appropriate application of the tool to this group of studies. Specifically, aspects of maintenance and reach require tailoring to the intervention, and reporting of differential effects using sub-group analysis will inevitably be limited where studies are typically small. Where a well established intervention like housing improvement is being evaluated items assessing reporting of population details (namely items 2 & 3) may require editing to clarify whether they relate to reporting of the target population and context for the study or the wider intervention. We chose to focus on the study population. In addition, criteria to indicate the extent of reporting were developed to reflect issues pertinent to our particular group of studies. The use of graded criteria improves the sensitivity and interpretation of the tool beyond the previous version which is restricted to a binary assessment [10, 11].

Conclusion

The RE-AIM checklist provides a useful framework to guide authors about which external validity items to report but, as indicated in earlier discussions, strict adherence to the checklist may not be appropriate. Despite this there is a need for authors to improve reporting of items to improve the transferability of the research findings. Within health public policy, the hypothesised health impacts of interventions are often dependent on an intermediate impact on a socio-economic determinant of health. Data confirming the function of the intervention may be as important as details of the intervention form and may help refine generalisable theories about the health impacts of tackling socio-economic determinants of health.

Funding

HT & ST are funded by the Chief Scientist Office at the Scottish Government Health Directorate as part of the Evaluating Social Interventions programme at the MRC Social and Public Health Science Unit, (U.130059812).

Acknowledgement

We acknowledge comments from Professor Mark Petticrew on an earlier abstract of this work.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Competing interests

HT & ST are not aware of any competing interest in this work.

Authors’ contributions

HT & ST selected and assessed each study. HT prepared and analysed the data and wrote the paper with comments from ST. Both authors read and approved the final manuscript.
Zusatzmaterial
Additional file 1: External validity reporting assessment tool: amended version and criteria developed by Thomson & Thomas.(DOC 72 KB)
12889_2011_4280_MOESM1_ESM.doc
Additional file 2: Assessment codes for individual external validity items with domain score sub-totals (ST) and total score by study.(DOCX 33 KB)
12889_2011_4280_MOESM2_ESM.docx
Literatur
Über diesen Artikel

Weitere Artikel der Ausgabe 1/2012

BMC Public Health 1/2012 Zur Ausgabe