Proximal impacts
All three EFHIAs were described as informing the thinking about the proposals assessed and informing subsequent decisions, though the extent and nature of that change varied a lot. The extent to which they informed decisions seemed to be associated with the level of involvement of those responsible for implementing the plans in the EFHIA process.
If [the EFHIA] had been built in earlier, I would have had more ownership of it. And certainly if anyone above me had built it in [to the planning process], I would have felt a greater sense of responsibility to act… So I think making sure the people at the right level are involved at the right, at an early stage.
NSW STI Strategy EFHIA interviewee
Although almost all interviewees described the EFHIAs as informing subsequent decision-making to some extent, this was not necessarily described as leading to changes to decisions and implementation.
Changing decisions and implementation
The extent to which the three EFHIAs in this study influenced subsequent decision-making and implementation varied markedly, even when described by interviewees involved in the same EFHIA. Only one of the documents available to be analysed had been formally revised following the EFHIA (The Good for Kids, Good for Life implementation plan). This document showed that all the recommendations in the report were clearly incorporated into the implementation plan. This process was described in interviews as involving a degree of modification and negotiation but also emphasised that once recommendations were contained in the implementation plan they would be monitored for progress and reported against.
But the, the beauty of it was that [the EFHIA] wasn’t my responsibility any more. It was sort of becoming embedded across [the program]. Yeah, yeah and once the recommendations were sort of offered and strategies presented back and negotiation around them to give them what we wanted. But they became a part of the program plan.
Good for Kids, Good for Life EFHIA interviewee
It is more difficult to point to concrete changes arising from the other two EFHIAs in this study. Interviewees disagreed about the extent of change that could be attributed to the EFHIAs. Both plans undertook substantial changes in response to broader changes to the NSW health system following the EFHIAs, which limited the extent to which subsequent changes can be attributed to the EFHIAs.
The positive thing that came out of it for me was that ah we heard some things had been changed. The difficulty was, and um, was that we had no idea what had been changed and we had no access to the documentation. And we had no access to the decision making around it.
ABHI Implementation Plan EFHIA interviewee
To be honest, I’m not sure that much else came out of it. I think, you know, given how difficult it was, I think just the fact that maybe some people might consider using health impact assessment and that we may have influenced the Strategy are not bad outcomes.
NSW STI Strategy EFHIA interviewee
Changes in health determinants
Three interviewees described addressing the determinants of health as an important intent underpinning the use of EFHIA, though they were not able to identify any changes to specific determinants arising from the EFHIAs they participated in. Two of the documents analysed made explicit mention of the determinants of health.
Predictive efficacy and achieving goals
These impacts were included in the original conceptual framework (see Figure
1) but were not found in either the interviews or document analysis. Predictive efficacy refers to the extent to which predicted impacts eventuated and achieving goals refers to the extent to which the stated goals of the assessment were met. Both these factors seem to have been of limited relevance in the EFHIAs in this study, though this may be due to the study’s setting, i.e. rapid EFHIAs being conducted voluntarily rather than to meet a regulatory requirement.
Distal impacts
Understanding
The EFHIAs were all described as leading to better understandings of how other agencies worked, and the pressures and concerns that informed health service planning. They also led to understanding of ways of working in partnership with other stakeholders.
[The EFHIA] made them think about and what our [Aboriginal communities’] way of doing business is. Don’t like this approach, the major consultation processes that needed to be undertaken before it actually was, before it was to be done. And that’s my recollection. I think I actually thought [the proposal] had some good points to it. I think it was a valuable process but it would be more valuable if it had been thinking about this stuff when they planned it.
Good for Kids, Good for Life EFHIA interviewee
The EFHIAs were also described as leading to better understandings of planning processes and how the plans were originally developed, though this view was contested in some cases.
Yeah, I think in hindsight, I would want to know more about why [we would] would want to do one, and what they hoped to get out of it, and I would want [people undertaking the EFHIA] to know more about what we would hope to get out of it, so that those misunderstandings or miscommunications didn’t happen in the process.
ABHI Implementation Plan EFHIA interviewee
Understanding of health equity specifically
Understanding of health equity and the determinants of health inequalities was highlighted as a major impact of all three EFHIAs. This was described as better understanding of the (i) potential health inequities that could arise or be exacerbated as a result of the type of proposal being assessed, and (ii) the distribution of potential impacts amongst population sub-groups based on different approaches to disaggregation (age, gender, socioeconomic status, location, etc.).
This change was likely to be due to the explicit focus on health equity in all EFHIAs. The extent to which understandings of equity changed as a result of the EFHIA varied between the three case studies, and even between interviewees within each one. The level of involvement in the EFHIA process (being the person responsible for undertaking the EFHIA, participating in the assessment/appraisal step, etc.) seemed to be closely associated with the extent of improved understandings of health equity, though this was not universal amongst the interviewees.
Understanding of health equity in the context of health service planning was also recognised by interviewees as not being straightforward:
I think there is something conceptually difficult about saying, “Okay, well you’ve identified gay men and drug users but then, who among those groups and more, you know that sort of… how do you prioritise… I mean, you know, how do you, and clearly with gay men you could, you could prioritise young gay men or do you could prioritise homeless young gay men… It really adds a layer of complexity and it makes it quite hard to conceptualise what you’re trying to achieve.
NSW STI Strategy EFHIA interviewee
I think from my own learning, one of the things we learned, I learnt, was that we overlook gender as one of the dimensions or differential impacts that, throughout the document, particularly things referring to adults, they really treated men and women as if they’re the same thing, and we know that their participation and their engagement’s very different, but we don’t necessarily articulate that… That was an unexpected finding for us, is how easy it is to overlook gender.
ABHI Implementation Plan EFHIA interviewee
This item was not in the original conceptual framework but arose consistently in the interviews and documents reviewed. It was described separately and using different language than was used for other forms of understanding, such as understandings of the determinants of health or understanding how other agencies worked.
Learning
The rapid nature of the EFHIAs was recognised by interviewees as responding to the decision-making context but that this may also have impaired the extent to which learning could take place. The nature of learning that was desired and anticipated from the EFHIA also seemed to be varied, with some participants talking about how they hoped the EFHIA would provide technical insights whereas others hoped it would enable people to think about the proposals, and health service planning in general, in a different way. In particular there were differing expectations about the nature and extent of alternatives that might be considered. The EFHIAs were described by four participants as involving a learning new concepts or approaches to addressing health equity concerns.
We were able to enter into some discussions with them about what might be alternatives, so I think that in these sorts of environments, we’ve got an opportunity to influence the implementation. It’s actually really important to have debate, and that’s what I think the EFHIA allowed.
ABHI Implementation Plan EFHIA interviewee
It hasn’t obstructed anyone, in getting them to reflect on their work, really, even if they weren’t, you know, up-skilling in the process of HIA, they probably could have learnt a few things about equity considerations, and how to incorporate that, so I think that might have been a missed opportunity to engage people in the process, probably the rapid nature makes that a little difficult.
ABHI Implementation Plan EFHIA interviewee
Influencing other activities
The EFHIAs were described as having impacts on a range of other activities, principally in terms of related planning and implementation issues that crossed over with other parts of health services. This influence on activities could be regarded as both positive and negative. In the ABHI Implementation Plan EFHIA this influence was described as impairing or undermining relationships and potentially limiting future collaboration.
[EFHIAs] can be used to change the way other sectors think about health and equity, like land use plans and that sort of thing, and I don’t think this is something that is going to work with health plans which are already pretty good at health equity. This will probably make me think about how I can use this with local government more though.
ABHI Implementation Plan EFHIA interviewee
Ideally I would like to say that what came out of it was a better relationships I don’t think that happened, but that would have been, in terms of my original thought at the beginning, that was one of the outcomes I had hoped would come out of it.
ABHI Implementation Plan EFHIA interviewee
In the case of the Good for Kids, Good for Life EFHIA it was described by all interviewees as opening up lines of communication within the program and clarifying decision-making and resourcing processes for those involved.
[The EFHIA] suited our purposes for making the programme culturally appropriate, but to do it on its own wouldn’t have done that. We sort of had a sort of a line to three other areas, sort of. So having the consultation or a more comprehensive consultation [that was] being done at the same time. Having, um, Aboriginal people working on the program, so identifying staffing and, also having some sort of resourcing agreement that what came out of it was actually going to be resourced, and like where we can go and do it.
Good for Kids, Good for Life EFHIA interviewee
Engagement
The EFHIAs were described by five interviewees as offering more avenues for engagement and participation than would usually be possible in health service planning. This was seen as closely linked to the structured EFHIA process and the degree of collaboration it involved.
Lots of the strategy documents are about, you know, let’s get a bunch of people together and we’ll build a shared understanding and we’ll make a commitment together to move forward with any existing funds, and that can be, be limited.
NSW STI Strategy EFHIA interviewee
Perception of HIA
Twelve of the interviewees described the EFHIA process changed their perception and understanding of HIA, and in particular EFHIA, and where it might usefully fit within future planning activities. Even in cases where the EFHIA was described as less successful this change in the perception of HIA was reported.
Individual responses
The second coding pass of sparsely coded or uncoded parts of the interviews during the analysis highlighted a number of sections in the interviews where people described how the EFHIA process had changed their perceptions, understandings and relationships at an individual level rather than an organisational one. The language used to describe this was distinct from how the interviewees described organisational responses or how they regarded the EFHIA process. It is important to note however that this individual response as a result of the EFHIA was only reported by six of the interviewees.
I don’t I’ve already said this but in my head that many of them the areas that I probably overlooked the most would [have been] equity related.
Good for Kids, Good for Life EFHIA interviewee
It made me think about some of my kind of thinking.
ABHI Implementation Plan EFHIA interviewee
I wanted to understand the process because it was new to me , but it was hard and it involved a lot of these new ways of thinking about it, and I am an epidemiologist and I just wouldn’t analyse it that way naturally, so I think it changed my sense of how I should think about these problems.
NSW STI Strategy EFHIA participant
This item was not in the original conceptual framework but arose across the three EFHIAs and seems to be related to several other factors in the conceptual framework and is described in greater detail below.
Other factors influencing the impact of EFHIAs
The other factors that emerged in the analysis as important factors influencing the extent to which EFHIAs appear to have an impact on decision-making and implementation were (i) timing and timeliness and (ii) the interplay between values, agency and learning.The case studies highlighted the need to undertake the EFHIA at the right stage in broader decision-making processes, i.e. early enough to ensure they could usefully inform decision-making. The other aspect of this is timeliness, which was the ability to conduct the EFHIA within the timeframe required or imposed by broader decision-making and implementation processes. There was variation between the case studies in terms of both timing and timeliness and the interviewees did not always describe that timing and timeliness had been well addressed within the EFHIAs. Whilst these factors weren’t the sole predictors of subsequent proximal changes (see Figure
1) they were important ones. This also suggests that timing and timeliness are factors that need to be addressed during the screening and scoping steps for both EFHIAs and HIAs.
The case studies also highlighted the interplay between values, agency and learning as related factors that may facilitate or limit the extent of changes that can occur as a result of EFHIAs. The EFHIAs in this study all involved some examination of potential health inequalities and looking at their distribution, whether these inequalities could be mitigated, and whether they were unfair. In all three EFHIA cases this involved some degree of re-examining organisational and personal values in order to inform whether potential inequalities were unfair and unjust, as well as which potential impacts should be prioritised for action. This necessarily involved revisiting and articulating the values that informed the development of the proposals as well as which values would inform implementation. In this way values played an important role in mediating the potential impacts of the EFHIAs on subsequent decision-making and recommendations.
This examination of values was not necessarily welcomed by all interviewees, particularly in cases where they were not closely involved with the assessment process or in the decision to initiate the EFHIA. They described the EFHIAs as focusing on issues that were not relevant to the decision-making context or not understanding the broader context for the proposal being assessed. The extent to which interviewees were able to express individual agency by initiating the EFHIA or participating in the EFHIA process was also related to whether they saw the EFHIA as successful or not. In every case where the interviewee described the EFHIA as not being a success they were either (i) not involved in collecting and appraising evidence in the assessment process, or (ii) did not play a role in initiating or agreeing to the EFHIA being undertaken.
Individual agency and participation in the EFHIA was linked to values but also appeared to be linked to the nature of learning sought from the EFHIA. Those interviewees who reported being less involved in the process or that it was someone else’s idea often described the EFHIA as inappropriately looking at options and implementation recommendations, whereas they had expected the EFHIA would focus on technical assessment, rather than focusing on implementation, or act as a “learning activity” (a phase used by four interviewees).
I do remember getting it back and going hang on a minute, we gave you really clear parameters about what you’re allowed, or whatever, for want of a better word, ‘to look at’, and it came back saying that. I really believe that it did misrepresent our intention behind it, and why we’d given these parameters around what was fixed and what wasn’t fixed… I think it does misrepresent, and it was quite antagonistic
ABHI Implementation Plan EFHIA interviewee
The HIA was successful, but really just marginally so. The proposal was too developed and worked up to change much, and the equity, the equity issues were not glaringly obvious ones. It was hard for novices, I guess that’s really what we were, hard for us to assess when it was a learning activity.
NSW STI Strategy EFHIA
Conversely those who were actively involved in the EFHIA process through their own choice described gaining new ideas about how to approach the issue the proposal was designed to address and a new appreciation of equity, particularly in relation to the proposal area being assessed.
I think there’s real value in an equity-focused HIA, because I think it does try and make people understand what equity is about. But I do think it’s a very hard concept to grasp, and people look at it, and I think that really happened with this policy, people look at it and they see that you’ve created these priority populations, so therefore you must have considered equity. And trying to get people to dig underneath that, even really quite, you know, educated and intelligent people, can be quite difficult. Because, it’s complicated.
NSW STI Strategy EFHIA
This suggested that there were different understandings about the nature of learning sought from conducting the EFHIAs, ranging from technical to conceptual and even social learning [
106,
107]. A shared understanding about the learning desired from an EFHIA, or lack thereof, may have affected its subsequent impact on decision-making and implementation, or even have lead to conflict. This shared understanding about learning also appeared to be linked to the interplay between values, individual agency and learning in these cases. It is important to note that while this interplay affected how the EFHIA was perceived, the effect was not uniform. While most people who had either not been directly involved in the EFHIA or not initiated it described the EFHIA as having fewer impacts, not all did. Even those who were most critical identified a number of positive impacts arising from the EFHIAs, in particular in terms of understandings of equity.