To what extent can the activities of the South Australian Health in All Policies initiative be linked to population health outcomes using a program theory-based evaluation?

There have been many international calls for effective action on the social determinants of health (SDH) involving a range of government sectors [1‐6]. Health in All Policies (HiAP) was developed by the European Union, and the WHO defined it as: ‘an approach to public policies across sectors that systematically takes into account the health implications of decisions, seeks synergies, and avoids harmful health impacts in order to improve population health and health equity’ [7]. HiAP initiatives have been adopted in many cities, regions and countries (see for example [1‐3]). Establishing the health impact of HiAP is methodologically challenging. There are multiple problems with applying randomised control trials to complex entities such as cities, regional and national governments [4, 5, 8]. Foremost among the problems are the difficulties of establishing control units that are really identical, the complications of ‘controlling’ the activities of experimental or control administrative units, then making realistic claims about attribution of observed differences in health status in complex systems where many factors affect health and outcomes may take years or decades to become apparent. In response, evaluators have advocated the use of logic models based on chains of causality to assess impact [6, 9, 10]. Such models are particularly useful when combined with insights from the critical realist evaluation approaches. A central aspect of this form of evaluation is a concern with causality and the identification of causal mechanisms in social phenomena like a HiAP initative in a manner quite unlike the positivist search for causal generalizations [11]. Emphasis is placed on the contribution that a policy intervention makes to an outcome and views this contribution within its social, economic and political context [12]. These models map context, detail mechanisms that influence pathways to outcomes, and measure outputs and outcomes. Thus logic models using a critical realist approach hold most promise for untangling the question of whether HiAP is effective in promoting health [13, 14]. Our study is the first application of such models to HiAP to assess whether it has contributed to health.

We collected mixed-methods data (described below) within a theory-based program logic framework [12] over five years (2012–2016). This paper reports on the population health outcomes that, using the theory-based program logic framework, can be claimed to be likely from the SA Health in All Policies intiative. It draws on new data analysis as well as prior data analyses from the evaluation that have been reported in other papers [20‐23]. The other papers consider HiAP processes and show how we have developed and applied the theory-based logic model to understand HiAP’s influence on population health outcomes. These papers and their key findings are summarised in Table 1. Drawing on these prior papers in addition to new analyses undertaken for the current paper is vital because the evaluation has produced considerable amounts of data that cannot be adequately presented and discussed within the contraints of a single overview paper.

The methods are summarised in Table 2 and elaborated on below and have been described previously [20].

Our findings are shaped by the program logic model (see Fig. 1) and describe how HiAP encouraged action on social determinants, and the likelihood that these actions have led or will lead to improved health and equity.

Our main finding is that, through using a pragmatic and theory-based program logic approach to causal analysis, SA HiAP can be judged to have made a modest contribution to actions likely to have improved population health in South Australia. It did this using the activities and policy processes shown in Fig. 1. The intiative was small: the SA Health budget for 2015–16 was $5.8 billion. HiAP totalled 0.00948% of that budget. Judged against this modest input, impact in terms of changing the culture of the public service and encouraging health to be understood as an important consideration in other policy areas, especially in urban planning and the environment, was significant. We argue that most actions listed in Table 3 are likely to lead to longer term health benefits given that the evidence links these interventions to health outcomes. Of course whether those health outcomes eventuate will also depend on the South Australian and global context. Contextual factors reducing the support from other departments accounted for one initiative not proceeding. In another instance we observed lifestyle drift from an initially wide-ranging identification of social wellbeing concerns for overseas students to an information pamphlet.

The activities and impacts shown in green in Fig. 1 were found to have occurred and collectively to have built an authorising environment for a range of policy processes that support health and wellbeing. Our evidence indicates that HiAP helped public servants to appreciate how their sectors affected health [20]. We predict this will be positive for the state’s future but also caution that the government’s response to the economic climate by reducing the public service results in staff turnover, loss of capacity and knowledge. This may have a negative impact, outweighing the positive HiAP effect. Elsewhere we have noted that the pursuit of health equity was more prominent in the early days of HiAP where some rhetorical support existed for progressing an equity agenda [21]. Subsequently, however, economic pressures resulted in the government narrowing its priorities to economic goals, and reflecting HiAP’s dependency on its political context, little was done to address inequities in our study period.

A major strength of our study is that it is the only application of a theory-based program logic to assess the likely contribution of a complex, longer term public health initiative. While we can only make limited attributions of causality of HiAP to improved health and wellbeing in South Australia, we can do this within a predictive model which sets the likely outcomes and the processes by which they are achieved [12]. The use of program logic to support a narrative argument concerning the likelihood of long term health impact is novel and has not been applied longitudinally to HiAP in any other setting. This study demonstrated that producing the evidence for HiAP’s impact called for in Leppo et al. [37] is possible with a longitudinal design and partnership between public servants and academics. Further, Guglielmin et al. [38] conducted a scoping review of local implementation of HiAP and concluded that contribution to theory development on HiAP implementation requires further research that ‘specifically investigates the facilitators and barriers of HiAP locally within their political and policy context’. This is precisely what our research has done by highlighting the ways in which the South Australian economic and political contexts had a significant impact on implementation and so influenced the likelihood of health outcomes.

Limitations to our research are that the attribution of causality is necessarily tentative and relies on imputed evidence. In the actions we observed, HiAP was a facilitator of change with many partners. This makes determining its contribution difficult. While the program logic shows a smooth flow through of action emphasising the contribution of the HiAP initiative, the actual picture is messier, with contributions, and in some cases the major contribution, coming from other sectors. However, interview respondents consistently informed us that in every case HiAP played a facilitating role and we were able to document that over time. There are also limits to how much account we could take of context. Labonté [39] noted in a commentary responding to our theory-based program logic that our work did not include ‘considerations of political and economic shifts at the national or global scales (eg, hyperglobalization, capital mobility, economic financialization, and tax competition)’ and that it ‘assumes that the policy actors at the South Australian state level have the capacities to implement HiAP in ways that can institutionalize sustained improvements in health equity’. Our program logic model was able to take into account some globalised context and did this, for example, in terms of the closure of an automotive manufacturing plant and the general impact of de-industrialisation in South Australia. It is, however, difficult to measure and account for the kind of global changes Labonté refers to in research which focuses on local activity. Institutional and political constraints meant SA HiAP focused on improving daily living conditions and was not able to address the underlying drivers of health inequities.

We draw five key lessons from our evaluation of SA HiAP to assist other jurisdictions using or considering using the approach. These are:

The SA HiAP approach had a small core budget. The approach was constrained in the extent to which it could address the underlying economic and power distribution impact of health, but can be judged to have had a minor impact on longer term population health. Many drivers of health are outside the remit of a state government. HiAP could do more to address health inequities and a promising option may be to partner with citizen groups and others to create a strong constituency for action on more structural determinants of health and a political will for intersectoral policy action for health and equity.