Realist review of policy intervention studies aimed at reducing exposures to environmental hazards in the United States

This review relies on a realist perspective for data analysis [12‐14]. Realist reviews seek to answer the question: “What is it about this policy that works, for whom and in what circumstances?” [14] Realist reviews focus on explanation in addition to description, with the goal of identifying the factors that make particular policies effective. From this perspective, policy interventions occur within different environments, and these interactions create reactions (mechanisms) that cause particular outcomes. Such interactions are referred to as “context-mechanism-outcome configurations” [12].

Interpreting context-mechanism-outcome configurations implies an understanding of the nature of each parameter. The context is the circumstances or environment in which policy interventions are implemented. These may include organizational, socioeconomic, cultural, and political conditions, the perceptions of stakeholders, and the process of implementation. The mechanism is the reaction to the intervention, which may be created intentionally or unintentionally depending on the context. These interactions may or may not lead to the intended outcome.

Assessing evidence using realistic review addresses context, mechanism, and outcomes by explicitly stating the assumptions of policymakers regarding the expected outcomes and how they will be achieved, and researching the implemented intervention to determine whether these assumptions were borne out. An example of a context-mechanism-outcome configuration for one of the studies included in this review is provided in Fig. 1. This study, which assessed an economic intervention, sought to determine whether an environmental purchasing program could reduce mercury waste at health care facilities in the Great Lakes region [15]. The context was an environment in which health care facilities lacked information about the mercury levels in different potential products available for purchase. The researchers created an intervention, the Health Care Environmental Purchasing Tool, which asked the facilities’ suppliers to provide information about mercury levels in products. The researchers provided an incentive payment for health care facilities to pilot the tool in their purchasing departments. The mechanism, or the reaction triggered by the intervention, was positive interest in using the tool, particularly given that it was developed in concert with purchasing officials and in light of institutional sustainability goals and the incentive payments. Using the tool led purchasers to the outcome, which was change in purchasing patterns that resulted in both one-time and annual reductions in mercury waste.

The realist review approach is designed to address complex policy questions, and synthesize evidence from multiple settings and interventions. Research based on a realist perspective can use qualitative, quantitative, or mixed methods of data collection and analysis. This review, which relies on measures of chemical exposure, considers quantitative results. A realist review systematically identifies and assesses the results of existing studies. It focuses identifying the mechanisms that resulted from the interaction between the intervention studied and the context, which allows what is referred to as a generative approach to causality. This approach allows specific consideration of the possibility that particular policy interventions could be transferred to other settings [13].

In order to identify key elements critical to the success or failure of policy interventions, information gathered from studies included specifics on the nature of interventions, the context, and their implementation, or success of the intervention, as assessed by measurements of chemical exposures.

One author (NW) extracted background data on the included studies (author, title, journal) using a standardized form prior to coding for content. Two authors (DA, NW) independently extracted data on content and risk of bias for all included studies, double-coding 20 % of the included studies to assure consistency and objectivity. Double-coded articles were discussed until findings reached consensus. All information was drawn from studies using a tool developed and modified by two of the authors (DA, LB). Information drawn for analysis included [1]: methods, including the type of intervention, study site, study design, aims, inclusion and exclusion criteria, outcome measures, outcome verification (if any), and methods of analysis [2]; context, including the targeted population and sample size [3]; intervention and assessment details, including exposure, duration of the intervention, the time between intervention and assessment, comparisons, and effect modifiers [4]; outcomes, detailing the results of the intervention [5]; risks of bias, including selection bias, concealment of allocation, incomplete outcome data, and any other potential threats to validity.

Articles were read as transcripts of the research conducted and the coders extracted information regarding contextual factors, the reactions triggered by the intervention, and statements of how the intervention was intended to work. Only the reported findings were listed in the data extraction tool, rather than coder interpretations. A sample of the data extracted using the tool is provided in Appendix 1. This study measured air quality before and after a regulatory intervention [16].

The data analysis was performed iteratively and manually. Preliminary results were discussed between the two researchers extracting data (DA, NW), the report was drafted by one of the researchers (DA), and was reviewed and revised by the principal researcher (LB).

The studies included in this review covered three primary types of policy interventions. The summary of findings in Table 1 provides an overview of the taxonomy of interventions, the number of studies that covered each type of intervention, the nature of the outcomes included in the review, and whether the study measured exposures in humans or the environment. We categorized the studies by the type of intervention (regulatory, economic, or educational) in advance to reflect the different types of policy approaches taken to reducing environmental exposures.

The types of policy interventions designed to reduce exposures to environmental toxins fell into three categories. The first type, regulatory interventions, includes laws designed to eliminate exposures through bans or requirements for modifications on equipment producing toxins. The second type, economic intervention, includes changes that provide a financial incentive in service of a reduced environmental exposure. The third type, educational interventions, includes policies designed to inform at-risk populations how to reduce their own exposure.

The studies included in this review did not always describe the nature of the interventions assessed, making it different to compare studies. Moreover, only a minority of studies provided baseline data to compare with the post-implementation measures intended to identify the effects of interventions. Table 1 shows the different types of research designs that were used. Two studies used controlled trial designs, one randomized, with control and intervention groups to allow comparisons (Rich 2002 randomized, Aschengrau 1998 nonrandomized). One study used a retrospective cohort design (Brown 2001), which included both a control group and baseline data. Of the 25 total studies, the majority, 14 studies relied on an interrupted time-series design, where measures were taken before and after an intervention was implemented (see Table 1). These studies all lacked control groups. The remaining studies relied on cross-sectional or time-series cross-sectional data, where measures were taken at a single point in time in a single sample (Thomas 2013, Postma 2011) or at multiple points in time from new samples (6 studies, see Table 1). All of the included studies provided at least one quantitative measure of outcomes, in the form of data detailing exposure levels or their proxies.

All interventions had a logic that underlay the design of the policy. The restriction of this review to studies that explicitly measured exposure data meant that all of the studies included in the analysis stated the intended outcome of the intervention. The reliance on existing datasets did not always make it possible rule out alternative explanations. For example, one study of the impact of the EPA (Environmental Protection Agency) NOx (mono-nitrogen oxides) Budget Trading Program reviewed New York state hospital discharge records for respiratory diagnoses after the implementation of the program (Lin 2013), but could not control for the simultaneous implementation of the state’s clean indoor air law, although implementation of this law would also be expected to affect levels of respiratory disease [18]. In addition, the use of existing datasets to measure exposures meant that the policy intervention may have had additional effects that could not be measured with the data that were available to researchers.

A majority of the included studies (15 of 25) described problems with incomplete outcome data; these issues are described further in results. In all of these cases, the method used to address missing data was exclusion of cases. Selection bias was also noted as a concern; five of the 25 included studies noted that the sample data were non-representative in some way, due to the use of either a convenience sample or a volunteer population. Only one study used a randomized controlled trial design (Rich 2002); the methods used for allocation and concealment (blinding) were not described. The risks of bias identified here may underestimate the actual risks of bias, as these issues were typically not discussed in the included studies.

Although measures of exposure identified by the inclusion criteria could be assessed in either the environment or in humans, the overwhelming majority of studies (18 of 25) reported only environmental measures.

The types of environmental exposures studied fell into five categories: (a) airborne pollutants, (b) hazardous waste, (c) lead paint, (d) pesticides, and (e) water pollution. A total of 12 regulatory and economic studies reviewed policy interventions and outcomes aimed at reducing airborne pollutants (see Table 1). All of these studies relied on environmental measures. Only one study of an economic intervention sought to assess reductions in hazardous waste; relying on environmental measures. Similarly, a single study addressed pesticide exposure using measures in humans. An additional five studies considered regulatory and educational interventions directed at reducing exposure to lead paint. Of these, two studies relied on the environmental measure of hard surfaces testing, and three studies used measures of both environmental and human exposures. A total of six studies assessed regulatory and educational interventions directed at water pollution; of these, five relied on measures of environmental exposures and one on human exposure data.

The next sections review the data analysis in each category of intervention, describing first the characteristics of the interventions and their outcomes, then the context and mechanisms. The extraction tool and sample article in Appendix 1 details the kinds of data drawn from each article.

In total there were 21 regulatory interventions covering four types of exposures: airborne pollutants, lead paint, water pollution, and pesticides (see Table 1).

Outcomes: Regulations of vehicle fuels and emissions standards were the most commonly studied form of policy intervention addressing airborne pollutants, making up over half of the 11 included articles. Bahadur 2010 studied increasing restrictions on diesel fuel in California using data collected by the California Air Resource Board to show a statewide decline in black carbon levels [19]. Davis 2012 focused on reductions over time in allowed vehicle carbon monoxide levels to show that multiple pollutants measured by the California Air Resource Board declined with increasingly stringent regulation, even when controlling for reduced driving during recessions [25]. Two studies specifically considered changes in vehicle emissions controls standards [21, 26]. Both collected data on particulate matter and other pollutants using road sensors, one in three different cities (Pokharel 2013) and one at multiple sites around greater Los Angeles (Bishop 2013). Both studies showed decreasing levels of pollutants over time with the implementation of increasingly stringent air quality standards. Two other studies reviewed requirements to retrofit or replace vehicles that had been identified as key sources of pollutant exposure. Mott 2002 focused on the requirement to add catalytic converters to U.S. vehicles and showed steady reductions on carbon monoxide-related deaths using national death certificate data over more than 30 years [24]. Dallman 2011 assessed air quality by measuring multiple pollutants released into the air above diesel trucks entering the Port of Oakland before and after the implementation of a requirement to retrofit truck filters or replace older trucks with new vehicles. It found a significant decrease in levels of measured pollutants [16].

The remaining studies reviewed broader air quality interventions. Three studies specifically considered policies that required reductions in nitrogen oxides emissions. Gego 2007 used data drawn from federal monitoring sites in the eastern United States and found reductions in ozone levels despite incomplete implementation of the law [22]. Aleksic 2013 specifically assessed ozone levels in New York before and after implementation of the standards using data from federal monitoring sites and also found significant declines [23]. Lin 2013 specifically assessed a nitrogen oxides emissions reduction Budget Trading Program, which was correlated with a decline in hospitalizations based on a respiratory diagnosis after implementation [18]. Cheung 2005 sought to assess the effects of a range of air quality regulations implemented over time. This California-specific study in the Los Angeles Basin found reductions in particulate matter over time [20]. Thomas 2013 measured sulfur levels in juniper trees after the implementation of the Clean Air Act, and found both reduced emissions and a recovery in the health of the trees [27].

Regarding context, all of the interventions took place in the United States, and were measuring outcomes that could be affected both by the specific interventions studied and by additional regulatory changes, which typically occurred at the federal level. Many of the studies did not attempt to describe the effects of specific interventions, assuming instead that the combined effect of additional regulations was causally related to the outcomes of reduced pollutant levels. This assumption was not always plausible. For example, Lin 2013, which focused on hospital admissions for respiratory diseases in the wake of nitrogen oxides regulation, noted that the simultaneous passage of a New York state clean air law targeted at tobacco use was also likely to have affected the outcome measure [18].

The specific mechanisms triggered by the interventions were rarely described. For some of the studies, those in compliance or out of compliance with regulations may not have been aware of the implications of their behavior at all.

Nevertheless, some studies specifically assessed changes in measured pollutants before and after the implementation of individual interventions. Studies at the local and regional level that reviewed air quality measures before and after the implementation of specific interventions suggested that two kinds of interventions reduced exposures: (a) those requiring retrofit or replacement of known producers of airborne pollutants (e.g., fuels, filters, vehicles), and (b) interventions that required reductions in the levels of certain pollutants as independently verified by federal monitoring, but that allowed state and local governments to choose the mechanism by which to achieve these reductions.

These studies suggest that the findings of the larger national studies, although they lack controls for effect modifiers, may be valid as well. Interventions that eliminated known sources of airborne pollutants reduced environmental exposures. Where federal monitoring was in place to measure air quality outcomes, interventions that established a maximum standard and allowed states and localities to develop their own plans to meet it also appeared to be successful. All of the regulatory studies of airborne pollutants showed that regulatory interventions at least partially reduced exposures.

Outcomes: The four studies considered multiple interventions that could reduce lead exposure. Rich 2002 tested cleaning techniques, comparing standard practices to more or less intensive methods. While all of the interventions reviewed were successful in reducing lead dust measured on hard surfaces, the combination of cleaning with TSP (trisodium phosphate) and vacuuming with a HEPA (high-efficiency particulate air) filter was most effective [28]. Brown 2001 compared strict enforcement of lead testing and remediation housing policies in one state to less-strict enforcement in a second state, and determined that the stricter enforcement was associated with lower blood levels of lead as well as higher property values [30]. Galke 2001 used multiple sites to review a range of HUD (US Department of Housing and Urban Development) interventions developed by local governments; these interventions were not described in detail. The study included an assessment component and determined that all measures reduced exposures, and that lower-intensity interventions that did not include abatement could also do so [31]. Breysse 2007 reviewed lead treatments required in the city of Baltimore by law, including an assessment of whether visual inspection was sufficient to identify risk. The study suggested that visual inspection was significantly less effective than testing of hard surfaces, and that although lead dust levels declined after remediation they remained above recommended levels [29].

Contextual factors and mechanisms: Context and mechanism were not described in detail in these studies. None of the three studies that discussed local interventions (Brown 2001, Galke 2001, Breysse 2007) specifically detailed the interventions that were used to reduce lead exposure. Although Brown 2001 proposed that stricter enforcement could reduce lead exposure, the authors noted that the direction of causality for the finding of association was unclear. It is possible that areas with higher property values had more resources available for enforcement as well as for remediation of lead risks [30]. The Rich 2002 study of cleaning techniques noted that HEPA-filter vacuums, although they reduced exposure, were both more expensive and more difficult to purchase in poor, urban areas where the risks of lead exposure were higher. As an alternative, the authors noted that more frequent cleaning could compensate to some extent for less effective equipment [28]. The Breysse 2007 study of lead remediation interventions in Baltimore suggested that objectively measuring of outcomes through testing was more effective than visual assessment by inspectors [29]. Consistent with this finding, the Galke 2001 multi-site study of local interventions that showed a range of interventions reduced exposure also included an assessment component [31].

The mechanisms, or reactions triggered by the interventions, were not described in any of the four studies, although the authors of the Rich 2002 study of cleaning methods speculated about possible public responses to a change in recommendations based on their findings. The authors of the Brown 2001 enforcement study advised directing more resources toward stricter enforcement of lead abatement policies, but competing priorities for those resources were not described, as this proposal was hypothetical. The mechanisms in the other two studies could not be described because the interventions themselves were not fully described.

Outcomes: Regulatory interventions that sought to address water pollution were divided in terms of scope, with two studies (Lakind 2010, Hundal 2014) reviewing national-level outcomes and the rest reviewing regional data. The national studies included one study of environmental exposure in humans. After the implementation of a new drinking water disinfection rule that requiring the reduction of pollutant levels post-treatment, Lakind 2010 used national blood sample data drawn from NHANES (The National Health and Nutrition Examination Survey), found that one of the targeted pollutants decreased in the blood samples but the other targeted pollutants did not [36]. Hundal 2014’s study of the Clean Water Act assessed trace metal levels in sludge after implementation using EPA data and found both that trace metal concentrations declined significantly and that the post-implementation sludge could be classified as “exceptional quality” [34]. Kauffman 2011 assessed multiple interventions, including not only federal water quality standards but the establishment of sewage treatment plants and phosphate detergent bans. Using data collected by USGS monitoring stations, it found that overall water quality along the Delaware River significantly improved, and measured levels of multiple pollutants declined [33].

The other studies specifically considered state-level interventions to reduce water pollution. Dorsey 2010’s study of the California Clean Beach Initiative measured water quality at multiple sites that had successfully bid for grant funding to develop programs to improve local water quality. Fecal indicator bacteria were measured before and after the program implementation and the interventions with the greatest reductions were assessed as evidencing the best intervention practices [32]. Daberkow 2001’s study of Nebraska’s efforts to meet federal drinking water standards through a fertilizer management program showed that providing advice on appropriate usage levels led to a decline in fertilizer use and associated pollutants, but that both remained above recommended levels [35].

The studies of regulatory interventions targeting water pollution were evenly split between assessments of national policies and state policies. The efforts to assess context are hampered by the fact that only one of the studies of national policies, Lakind 2010, could describe the effects of specific interventions. For two studies (Kauffman 2011, Hundal 2014) the period of analysis included implementations of multiple interventions. Studies of state-specific policies (Dorsey 2010, Daberkow 2001) were better able to assess the effects of specific interventions. Studies of both national and state interventions were more successful in measuring outcomes when data that could measure compliance were available or when data collection had been built into the intervention. Lakind 2010’s study of changes in blood pollutant levels after the establishment of new drinking water disinfection standards was sponsored by the industry that developed the new disinfection process; the effect of this potential conflict of interest was not discussed by the authors [36].

The mechanisms triggered by the interventions were not fully described. The studies that included multiple interventions (Kauffman 2011, Hundal 2014) could not disentangle mechanisms, even if there had been an effort to describe them. Studies that use anonymized national data on human exposure such as NHANES, like Lakind 2010, cannot assess the reaction even to specific interventions. Studies of single interventions (Dorsey 2010, Daberkow 2001) may also find that the reactions triggered by the intervention were not those that had been anticipated. Daberkow 2001 relied on providing advice to farmers as a means to reduce fertilizer use, but noted that this advice was undercut by the farmers’ overly optimistic assessments of expected crop yields, which led to continued over-fertilization and as a result, to unexpectedly high levels of water pollution [35]. Studies of interventions that provided a strict standard, measurable outcome data, or a clear enforcement mechanism showed reductions in pollutants, possibly by making the implications of meeting or not meeting the regulatory standard apparent.

The studies of specific state-level interventions suggested that monitoring of outcomes may identify policies that reduce measured pollutants. Dorsey 2010’s study of the California Clean Beach Initiative monitored five types of interventions proposed by local agencies for funding and was able to determine which intervention was associated with the greatest decrease in fecal indicator bacteria (diversion and a combined approach). Daberkow 2001 found that efforts to decrease fertilizer use in Nebraska by providing advice to farmers, which did not require changes to behavior or link reductions in fertilizer use to incentives or monitoring, were less effective.

Outcomes: Clune 2012’s study, which was national in scope, found that measures of pesticide concentrations in urine declined significantly after implementation of the new standard [37].

The review included two studies that considered education interventions, which addressed two types of exposures: lead paint (Aschengrau 1998) and water pollution (Postma 2011).

Study characteristics: The two educational studies assessed efforts to teach families how to reduce their environmental exposures. Aschengrau 1998 was a controlled trial that sent outreach workers to the intervention group to teach caregivers how to clean housing units where children were exposed to lead paint [38]. Postma 2011, a cross-sectional survey, encouraged families to test well water for contaminants [39]. The Aschengrau 1998 study of lead remediation instruction reviewed measures of environmental exposure (hard surfaces testing) as well as human exposure (blood levels) before and after the intervention. The Postma 2011 study of well water contaminants assessed measures of environmental contamination after the intervention.

Outcomes: The Aschengrau 1998 educational intervention that sought to address lead paint exposure was assessed at the local level, in the city of Boston. Outreach workers made a home visit to participants in the intervention groups to educate caregivers how to remove lead dust through cleaning in order to reduce exposure, but after the intervention, researchers found that caregivers did not complete all of the suggested cleaning, particularly in window wells. The intervention was only effective for children that had severely high lead levels, and showed no effect or a negative effect in less severe cases [38]. The Postma 2011 study of well water contaminants after an educational outreach effort in Montana and Washington found that 27 % of households tested positive for at least one contaminant post-intervention, and that 89 % of wells contaminated with coliform in the first test were still contaminated with coliform at retest. Higher socioeconomic status was correlated with higher testing rates before the intervention, and with a lower risk of exposure to pollutants [39].

The mechanism, or the reaction to the intervention, suggests that the value of educational interventions may be limited. Caregivers targeted by the lead paint educational intervention studied by Aschengrau 1998 did not complete the suggested cleaning strategies, leaving children at risk of continued exposure to lead. The studies of lead paint regulatory interventions, discussed above, suggest that these caregivers may not have had the resources available to clean as advised. Similarly, the efforts by Postma 2011 to provide information about well water contaminants did not necessarily lead to remediation, given that nearly 90 % of households with well water contaminated by coliform on the first assessment still had contaminated water on the second assessment. In both cases, there was inadequate response to the intervention, suggesting a broken mechanism. This may have been caused by an incomplete understanding of the context in which interventions were undertaken.

The review included two studies that considered economic interventions, which addressed two types of exposures: airborne pollution (Lu 2012) and hazardous waste (Eagan 2002).

Study characteristics: Lu 2012 considered the effects of changing natural gas prices relative to regulation on national sulfur emissions levels, after passage of the Acid Rain Program and cap-and-trade regulation [40]. Eagan 2002 assessed a purchasing tool that provided information on mercury levels in products for reducing waste mercury production by health care facilities in the Great Lakes region [15]. Both studies reviewed measures of environmental exposure, specifically sulfur emissions tracked by the EPA (Lu 2012), and the extent of mercury waste produced by health care facilities in the study sample (Eagan 2002).

Outcomes: The economic intervention study by Lu 2012 assessed the effect of changing natural gas prices relative to national regulation on sulfur emissions and was conducted at the national level. Lu 2012 found that the drop in sulfur emissions was attributable to the regulatory interventions rather than the economic effects of changing natural gas prices [40]. Eagan 2002’s study of hazardous waste focused on developing a tool for health care facilities in the Great Lakes region to voluntarily reduce mercury waste by purchasing products that contained lower levels of mercury. The intervention developed was a purchasing tool sent to suppliers which provided information on the mercury levels in each product, combined with an incentive payment that encouraged participating health care facilities to pilot use of the tool. After the intervention, new purchases reduced mercury waste both in durable equipment (a permanent reduction) and in annual purchases (an annual and ongoing reduction). Five of the nine participating health care facilities stated that they would continue to use the purchasing tool in the future [15].

The mechanism, or reaction to the intervention, reflected the different populations included in the studies. The population that voluntarily enrolled in the Eagan 2002 study was responsive to the intervention, and changed behavior to reflect the goal of reducing waste mercury. Even so, nearly half of the participants stated they would no longer use the intervention after the end of the study and its incentive payments. The population subject to a natural experiment in Lu 2012—changing gas prices—was not responsive to the economic intervention. The contemporaneous regulatory interventions appeared to have more effect in reducing exposures.