Process evaluation findings
The data from the process evaluation identified four factors that assisted the EFHIA. These were the support, commitment and openness of the Department of Health to having their plan assessed, the clarity of the instructions from the Department of Health (included in Additional File
1), the structured process the EFHIA followed, and the composition and experience of the expert panel coupled with the ease with which they were able to work together. Because the Centre for Health Equity Training, Research and Evaluation (CHETRE) is recognised as having expertise in this area and had worked with many of the managers on previous projects [
71], there was a level of trust which facilitated the conduct of the EFHIA. The structured process made it transparent what commitment was required of the participants and the expert panel in terms of time and scope of activities to be undertaken.
Three major constraints to undertaking the EFHIA were identified. These were the timeframe required, reliance on expert opinion and a limited range of literature rather than a broader range of evidence, and the difficulty in being objective concerning negative or unanticipated consequences that individual members of the EFHIA working group or expert panel strongly supported.
For policy officers and managers the EFHIA provided an opportunity for reflection on how issues of equity had been addressed in the draft implementation plan and how these and other issues could be improved. For example what balance was needed between innovation, which often had high political and professional appeal, and expanding and sustaining existing programs for which there was evidence of effectiveness. In other words, managers had to decide if they should inject more funding and support into a new suit of programs and to neglect existing programs.
Impact evaluation findings
The interviews identified a number of direct and indirect changes to the NSW ABHI implementation plan that occurred as a result of the EFHIA. Though there was a high degree of concordance about the process of the EFHIA (how it was conducted, who was involved, what the major events were, etc.) there was disagreement between the interviewees about the extent of change that occurred as a result of the rapid EFHIA. This disagreement was not fundamental: all those interviewed felt that the EFHIA had some impact on further planning and decision-making. Rather the disagreement was about the extent of change that could be attributed to the EFHIA. This ranged from a very small amount of influence according to some of those interviewed, to what was regarded as a moderate amount of influence by others.
The impact evaluation identified five major themes in relation to the impact of the rapid EFHIA:
1. Changing implementation planning
There were a number of changes to the NSW ABHI implementation plan that were attributed to the EFHIA. The most obvious of these were that the managers responsible for the development of aspects of the plan were asked to re-draft their sections to take the EFHIA into account.
"What we did after we got the equity focused HIA, we gave it to all the managers, and then for each one of their little, sort of, almost section in the plan, we said 'we want you to write a proper plan about how you're going to do it, and we want in your plan, to specifically say how you're going to address the recommendations of this.'
Interviewee 1
Other changes were attributed to the EFIA, though there was greater disagreement between the EFHIA working group and the Department of Health staff about the nature, extent and reasons for the changes. An example of this were the recommended changes to the proposed resource allocation split between urban and rural Area Health Services for a specific activity within the implementation plan, to favour more resources going to rural health services.
I think it actually did have at least one impact that I know of, which was that we had identified that not enough money was being invested in rural areas, although the resources were going to be allocated, the rural and the urban areas were going to get the same resources. And so I understood that... the problem is, for some of the rural areas, what they would have been getting wasn't enough to actually employ someone, so sixty thousand in the thick of an urban area's quite a lot, but in a rural area, it actually doesn't give you capacity. So I understood that what happened was each of the rural area health services was given a larger amount of money than the urban areas, and that they then wrote up their proposals... So it did have that impact.
Interviewee 2
Several interviewees identified this as a change attributable to the EFHIA but others discounted it, as the measure was not implemented in the form originally outlined in the draft implementation plan due to reallocation of funding. This mirrors some of the difficulties that have been found in other HIA impact evaluations in trying to attribute changes solely to a specific HIA [
72‐
74].
2. Consolidating understandings of equity
There was a broad agreement between the interviewees that the rapid EFHIA had brought the potential health equity impacts to the fore of the development of the implementation plan and that this was unlikely to have been as clearly addressed without the EFHIA. The interviewees from the Department of Health regarded this focus on health equity issues as a consolidation and focusing of existing knowledge, rather than being transformative or revelatory in nature - it was regarded as possibly under-considered information rather than unknown.
It probably provided a useful tool to make sure that people considered equity issues. Had we not had support for those type of [equity] issues being considered up the line, it probably would have been used as an internal advocacy tool...
...I'm not convinced that [the EFHIA] made people do things differently, because I think that they probably, should've, would've, hopefully would've, done those things anyway. It was nice that it was explicit, rather than left to being implicit.
Interviewee 3
So for me, personally, if you see the change [in] the acceptance of equity as a value determining and influencing people's thinking and work, it's good.
Interviewee 5
HIA's usefulness in consolidating knowledge and understanding of health issues and potential health impacts has been noted in the literature [
72,
75], and approaches that explicitly examine health equity impacts seem likely to enhance understandings of health equity as well.
A challenge that was identified by the interviewees from both the EFHIA working group and the Department of Health was that many of the potential health equity issues that could have arisen did not relate to the overall structure or nature of the ABHI initiatives, but to the way the initiatives would be implemented.
...when I was going through the [EFHIA] recommendations, that some of them appeared to have gone beyond just saying what would happen. If you're just trying to provide equity focused [recommendations], they're often about good planning, which I think was probably very apparent to the people during the equity focused HIA going 'what is the good planning in this?'
Interviewee 1
There were also some things in [the EFHIA report] that, I guess, implied, that we wouldn't consider, some issues that I think can be dealt with in careful planning, and careful implementation, and the intention, as I said before, if the [ABHI implementation plan] was really about 'this is the flavour of where we're going with this' we're going to have to obviously have greater implementation plans around each of these strategies, we've only got sixty pages to do it in.
Interviewee 3
This highlights some of the challenges in undertaking assessments of implementation plans. Policies, such as the ABHI as a whole, are necessarily aspirational, setting out areas of activity in broad terms. EFHIAs, and HIAs in general, need to consider implementation as it is at that stage where many unintended and previously unidentified impacts are likely to arise. The NSW ABHI implementation plan included details of how the initiatives would be implemented, but many of the very detailed planning and implementation activities were appropriately determined by operational and service managers. This identifies a tension relevant to all HIAs, but to EFHIA in particular; to what extent should an HIA focus on making recommendations to assist implementation?
Two preconditions seem to be important enablers of EFHIAs of implementation plans: a high degree of understanding of the policy context and processes being assessed; and trust and constructive engagement between the assessors and those responsible for the development and implementation of the implementation plan. This is similar to the findings of studies that have looked at the impacts of HIAs in the Netherlands [
76] and impact assessments more broadly [
77].
3. Enabling discussion of alternatives
Several interviewees stated that the EFHIA enabled consideration of different ways of achieving the implementation plan's objectives.
...even during the time we were doing it... we were able to enter into some discussions about what might be alternatives. So I think that in these sorts of environments, we've got an opportunity to influence the implementation. It's actually really important to have debate and that's what I think the EFHIA allowed.
Interviewee 2
The identification and assessment of alternatives is an important and under-emphasised part of HIA and impact assessment practice [
13,
78,
79]. The development of more formal procedures for generating alternatives that address health inequities, which may then be assessed using EFHIA, would be of use in ensuring health equity is considered earlier in the formulation of policy options.
4. Missed opportunities
There was a degree of ambivalence towards the rapid EFHIA on the part of several of the interviewees, amongst both people from the Department of Health and the EFHIA working group. The terms "lost opportunity" or "missed opportunity" came up several times during the interviews. Whilst all five interviewees acknowledged that the EFHIA had some degree of direct and indirect impact on subsequent activities, three interviewees expressed disappointment that more didn't come from doing the EFHIA, in the form of either more robust consideration of equity in health policy in general or ongoing collaboration with the expert panel.
No, it was really a lost opportunity, I think, to get people engaged, and not only engaged in HIA, but in equity...
Interviewee 1
I think there might have been a missed opportunity... The EFHIA focused too much on issues that would have been addressed at later stages in the planning anyway.
Interviewee 4
This can be attributed, at least in part, to feelings that the Department was not able to be fully involved in the EFHIA due to the competing pressures involved in finalising the implementation plan. The EFHIA was viewed by two of the interviewees as being overly critical of the development of the implementation plan and failing to recognise the time-pressured and politicised context it was being developed in.
...by doing an HIA, if you start then telling people how to do good planning, it's almost like it's a little bit insulting to those who believe they are good planners, rightly or wrongly... So I think there's a fine line between telling people how to suck eggs, when they already know how to suck eggs, but doing it in a different way.
Interviewee 3
I think people felt when recommendations came in, that they saw as a critique, or not that they were a critique, because different... They were like 'Oh, but it wasn't a proper plan anyway, it was just, you know, we were just trying to get the money, and that was our goal at that time, just get the money, and we said we'd do this, but not sure if we really will'.
Interviewee 1
This suggests that the involvement of stakeholders and decision-makers in the process of EFHIA is more than an ideological commitment to participation and representation; it is critical in enabling it to have an impact on decision-making and implementation [
80,
81]. There are of course significant, and possibly insurmountable, tensions between the rapid processes required if EFHIA is to inform policy development and implementation in a timely fashion and the need to engage stakeholders fully in the process of conducting the EFHIA.
5. Differing conceptualisations of the purpose of the EFHIA
Although all participants described changes that were attributable to the EFHIA, in general the Department of Health staff described the EFHIA as making a more modest contribution to the development and implementation of the implementation plan than the EFHIA working group did. This difference may be attributed to greater involvement and familiarity with the ongoing development and roll-out of the implementation plan or differing understandings about the role and purpose of the EFHIA.
We didn't have a shared understanding of why we were undertaking it. Our purposes were probably different from CHETRE's purposes, and maybe that's where they don't work, but if you have two differing purposes, unless you can fully appreciate what those two different purposes are, maybe it doesn't work out as well as it could...
...I think there was a feeling that, well, we could get something out of [the EFHIA]. There were probably two rationales for why it would be useful. One is that we could get some, a critique if you like, or some feedback about, through an equity lens, on the strategies that we had proposed. And the second one was that it would perhaps serve a process of helping people who are more engaged in the consultation process.
Interviewee 3
In a way, it was about improving the quality of the document, it was actually quite important to be able to debate some of the issues.
Interviewee 2
These differences may be partly due to variations in the way people involved in the EFHIA understood the purpose of the HIA in general. Those from the Department of Health tended to describe HIA as a process for using evidence to informing decision-making, whereas those from the EFHIA working group tended to describe HIA as a process for quality enhancement and examining unanticipated impacts.
There is an increasing consensus internationally that impact assessment should be understood as a learning activity [
16,
82‐
85]. Glasbergen [
86] describes three types of learning that can occur through impact assessment:
-
Technical learning, which involves searching for technical solutions to fixed policy objectives;
-
Conceptual learning, which involves redefining policy goals, problem definitions and strategies; and
-
Social learning, which emphasises dialogue and increased interaction between policy actors (this is distinct from the concept of social learning described in the psychology literature [
87]).
Some of the differences in this case may be understood as different attitudes to desired learning goals of the EFHIA. Many of those from the Department of Health described the EFHIA as a technical learning activity; those involved in the working group described the EFHIA in terms more consistent with conceptual learning. There was very little discussion of impacts that might be classed as social learning within the impact evaluation interviews. Understanding the different types of learning that may come from an EFHIA, and which one is desired within a specific context, is important as different expectations may serve to create confusion and tension amongst those involved. In our work we would describe HIA as both a technical tool and a process and it is this process that provides the opportunities for conceptual and social learning to build ongoing relationships with other stakeholders.