Methods
Ethics clearance was obtained from the University of Melbourne Human Research Ethics Committee (ID 1750086.1).
The project was overseen by a project reference group (PRG), whose members included representation from evaluation end-users including the ACCS, the Lowitja Institute, the Department of Health, the Department of the Prime Minister and Cabinet, and the Productivity Commission.
Here we conduct an evaluation of current evaluation practice across Aboriginal and Torres Strait Islander health and wellbeing policy, programs and services 2007–2017. We do so by using a 2-step approach. Firstly, by determining what principles governments state as being important to the way they work in Aboriginal and Torres Strait Islander contexts, and then secondly by assessing whether these principles are embedded in evaluation tendering and evaluation practice. We do so as a means of holding governments to account and measuring transparency of current practice. These 2-steps are described below.
Review of government planning documents: identification of principles
A review of National, and State and Territory health planning documents was conducted in March 2017 to identify the principles that governments state as important for working in Aboriginal and Torres Strait Islander health contexts. An online search of Commonwealth and each State and Territory health department website was carried out to locate current Aboriginal and Torres Strait health planning policy. Thematic analysis was used to review these policy documents to identify the principles that governments state as important. Principles appearing across two or more documents were included for review. Content analysis was then used to explore how these principles have been articulated across each of the government planning documents. Documents were reviewed by two researchers (MK and JL) and where there was any variance, discussion was had to reach consensus. Once the principles were agreed upon, the wording used within reports was analysed to draw out how governments articulated these principles.
Identification of government tenders and evaluation reports
Evaluation tenders are issued by commissioners of evaluations and detail the requirements of an evaluation and the obligations and responsibilities of the evaluator. Two sites were searched for tenders to evaluate programs in health and wellbeing:
AusTenders.com and
Tenders.net. The timeframe for the review was January 2007 to January 2017.
While
AusTenders.com can be searched directly,
Tenders.net does not list expired tenders. A special request was made for an offline search to be conducted and the results sent by email. In searching for relevant tenders, a broad definition was given to ‘health’ and ‘wellbeing’ in order to include evaluation in related fields, such as education, justice and sport. The search resulted in a large number of hits, with more than 12,000 hits returned from a search of the AusTender site. Search results were truncated to include only minimal information, so it is not possible to see the full material of the tender that is being searched. However, with the number of hits returned, it is possible that the words ‘Indigenous’ and/or ‘Aboriginal’ are included in a standard phrase in every Australian Government tender, which led to all tenders appearing as search results. The initial advice from
Tenders.net was that a preliminary search showed 1864 matches; however, the final spreadsheet provided had 3441 results. While the representative from
Tenders.net advised that the dataset included all public tenders listed on the AusTender site, as well as other sites, the AusTender search returned many results that did not appear in the
Tenders.net spreadsheet. All search results from
Tenders.net and AusTender were examined. After elimination of duplications and results that did not fit the criteria, 381 individual records were included. A further nine evaluations were identified from the website of the Australian Indigenous Health
InfoNet, bringing the total records included to 390.
Despite the fact that all tenders are publicly listed initially, none of the tendering organisations nor the sites for publicly listing tenders keep a repository of tender information once it has been let. Tender documents were therefore located by contacting the person listed as being responsible for the tender. If the person was no longer available, the department responsible for the tender was contacted.
Review of government tenders and evaluation reports
We reviewed tenders as a means of examining commissioning practice and reviewed evaluation reports as a means of examining evaluation practice.
Evaluation tenders and reports were reviewed to see if they included health planning principles that governments state as important. We then deductively identified how these were articulated by governments. For tenders, whole documents were reviewed with a focus on the selection criteria as these sections detailed how the evaluation should be conducted. For evaluation reports, whole documents were also reviewed with focus on the methodology, evaluation questions, outcome measures and program logics, as these provided information on how the evaluation was done and what was evaluated.
For reporting, we present the percent of tenders and reports underpinned by the principles identified in government planning documents.
Discussion
In this paper we reviewed Australian Commonwealth and State and Territory government health planning policy to find out what principles governments state as being important to the work they do in Aboriginal and Torres Strait Islander contexts. We then assessed whether these principles are reflected in evaluation commissioning and evaluation practice. We found that despite a strong rhetoric that emphasised the importance of partnership and holism and to a lesser degree cultural competency and capacity building, these principles were not widely reflected in evaluation practice for the period 2007–2017.
For Commonwealth and State and Territory governments, establishing a solid evidence base that centres Aboriginal and Torres Strait Islander perspectives and understandings across the policy cycle (planning, implementation, evaluation) were stated as goals of the work they do. We found that principles such as evidence-based, accountability and equity underpinned most of the evaluations. Despite all seven health plans including the principle of partnership approaches, there was imperfect evidence of this in practice. The mechanisms to centre Aboriginal and Torres Strait Islander perspectives and understandings to fulfil the principle of partnership were less frequently embedded in evaluation practice. We note that although evaluations often engaged Aboriginal people or communities (i.e., consulted with key people or had an Aboriginal reference group), engagement as a mechanism for including Aboriginal perspectives is not always as far reaching as partnerships where the ACCS is at least theoretically provided ownership of decision making. However, in saying this we remain mindful that the process of colonisation impacts the power dynamic between Aboriginal partner/s and any dominant institution, where economic, resourcing and political power is largely held by the later [
28]. In health planning, the meaning of engagement encompassed concepts from consulting through to community control, but rarely did we see Aboriginal community controlled organisations leading or enacting the principle of self-determination over evaluation. Although not represented in government health planning documents, many have asserted the importance of embedding the principle of self-determination in evaluation so that Aboriginal perspectives can be centred [
29,
30]. Examples of evaluations that centre self-determination principles in utilising the skills, strengths and knowledge of the community-controlled sector are rare in contemporary practice [
29‐
33]. Indeed, NACCHO (2019) raises concerns that evaluations are largely dominated by a private sector that is spatially and culturally removed from Aboriginal people and organisations, and question how such practice can centre Aboriginal perspectives, knowledge and experiences [
29].
Our finding that the principles of equity, accountability and evidence-based most frequently underpin evaluations is best understood in terms of wider Commonwealth policy. The 2007–2017 period was marked by the Closing the Gap policy agenda and its overarching objective to build a robust evidence base, addressing disparities and achieving health equity for Aboriginal and Torres Strait Islander people. In recent years there have been increasing criticisms that Closing the Gap policy lacks the perspective of Aboriginal people through the decade-long exclusion of key Aboriginal bodies such as NACCHO from leading policy formation [
34]. Policy without the strong voice of the ACCS has meant that diverse aspirations of all Aboriginal people have been excluded in favour of policy whose objectives are only equitable for Aboriginal people who aspire to live under socio-cultural standards set by the dominant culture [
35,
36]. The full breadth of aspirations of Aboriginal people can only be responded to when these voices formulate policy. The recent 2019 Closing the Gap Partnership Agreements signed by the Commonwealth Government, State and Territory Governments and the Coalition of Aboriginal and Torres Strait Islander Peak Bodies has been a recent development to increase ACCS participation and involvement in Closing the Gap policy [
37]. While peak bodies have long advocated for Aboriginal leadership in defining policy, health bodies such as NACCHO have also called for Aboriginal definitions of successful programs and policy to be given weight, alongside traditional evidence-based approaches [
29].
In addition to lacking Aboriginal perspectives, evaluations infrequently centred Aboriginal understandings of health, despite health planning documents stating that a holistic understanding of health should underpin the work being done. Rather than drawing on NACCHOs definition of health adapted from the National Aboriginal Health Strategy (1989) (see Table
2) [
27], evaluations instead focused on individual and biological outcomes (i.e., change in disease rates, behaviours or knowledge). We note that even when a holistic concept of health was present it was not well interpreted. It was not uncommon for evaluators to alter the concept of holistic health and present dominant social determinants, or individualised mental health measures. These findings echo a review by Lutschini (2005) that reported that despite the holistic concept of health being centrally placed in policy and strategies, policy makers often lacked a coherent articulation of the concept, were often uncritical and unreflective in their use and interpretation of it and often altered the concept and constitutional element without justification [
38]. In saying this, we recognise that in the absence of widely accepted and valid quantitative measures for holistic health and wellbeing, evaluating for the holistic concept of health is difficult [
39]. But we also know of the increasing scholarship in recent years to capture concepts of holistic health using narrative methods and Indigenous methodologies [
40].
Across the Australian health planning documents there were not always consistent articulations of what was meant by the principles of shared responsibility, cultural competence, engagement, partnership, capacity building, equity, a holistic concept of health, accountability, or evidence-based. There were also varying degrees to which these articulations engaged with Aboriginal understandings and preferred articulations of these concepts.
A limitation of our review is that the principles we are evaluating are those that governments state as important to the work they do. Although such principles would have been developed through consultation with the ACCS, they may not capture those principles that are most important to Aboriginal people and the ACCS. We are cognisant that in neo-colonial contexts, health policy, including evaluation, cannot be separated from the control and regulation of Aboriginal bodies [
6]. There is, without a doubt, an imbalanced power dynamic in current policy that sees dominant cultures policing Aboriginal populations and controlling the health agenda. As such, improving evaluation practice requires more than just governments doing what they say they will. As recognised by the Productivity Commission, there is a real need for transformative reform. We highlight that some of the principles that the ACCS have highlighted are important to evaluation and policy more broadly. These include: self-determination, community control, rights based approached, Aboriginal-led, investment in Aboriginal capacity, strengthen-based, do no harm, ethical, effective, transparent, cultural continuity, recognition of systems inequalities, recognition of past colonising and culturally safe evaluations, community benefit, transformative/decolonising orientations, social justice, Aboriginal cultural and intellectual property rights, empowering, and recognising diversity. It is obvious that the ACCS want Aboriginal perspectives, experiences and understandings to be central to evaluation policy pertaining to Aboriginal and Torres Strait Islander people [
29‐
33,
41]. In addition, the reference group for this project that included members of the ACCS emphasised the importance of capitalising on Aboriginal strengths and data governance and sovereignty [
13].
We also recognise that a limitation of our evaluation is that we only provide a count of the principles that have been reported within publicly available evaluations put out for advertised tenders by Australian governments. It is not clear what level of activity we have failed to capture. Our method of identifying public tenders, has also meant that we do not capture the full breadth of evaluations tendered by Aboriginal community controlled and other organisations. We realise that we are not capturing how such principles relate to internal evaluations, including those done by Aboriginal organisations as part of continuous quality improvement, monitoring and safety activity. In addition, the focus on external independent evaluations means that the evaluators are less likely to be subject to disincentives to reporting poor or adverse outcomes than might be the case in other forms of evaluation.
For evaluations to centre Aboriginal perspectives and understandings, there needs to be greater involvement of Aboriginal and Torres Strait communities and organisations in the planning and implementation of programs, policies and services, not just evaluation [
2]. This recognises that many of the terms for evaluation including the outcome measures are decided early in the process of planning a policy, program or service [
35]. As rightly identified by Altman (2019), for any principle to be meaningfully embedded in an evaluation there needs to be recognition of it across the policy cycle and importantly at the planning stage, where the parameters of the evaluation are largely set [
35]. For example, principles such as self-determination, social justice or anti-racism cannot be fully realised in evaluation if measures of programmatic success are pre-defined by government. To centre Aboriginal perspectives there needs to be mechanisms for Aboriginal and Torres Strait Islander leadership and ownership at all phases of the program planning and evaluation cycle. With recent development of the ‘Ngaa-bi-nya Aboriginal and Torres Strait Islander framework’ and the ‘Lowitja Evaluation framework to improve Aboriginal and Torres Strait Islander health’ there are now frameworks that talk to the ethical responsibilities of doing evaluation in Aboriginal contexts [
11,
13]. Use of ethical frameworks that delineate the responsibilities of all parties in evaluation (including commissioners, evaluators, implementors, Aboriginal community, Aboriginal participants) have potential to improve evaluation practice.
The findings we report here also highlight problems of transparency in current tendering and evaluation practice. We found that despite tenders being publicly listed none of the tendering organisations had mechanism for tracking once it was let. We were only able to access 5% of tenders. It is a concern that there is no publicly available repository for tenders as this would provide a means to conduct quality assurance practices to improve practice. We are also cautious that our findings only relate to 5% of all tenders; as such we do not know the true extent to which tendering practice is reflective of health planning. It should also be recognised that the tendering process is one way to change practice. One straightforward mechanism to include Aboriginal perspectives in current practice is for all tenders to explicitly state that a partnership and engagement approach is a requirement for prospective evaluators. This would also be beneficial for capacity building and cultural competency criteria to be included if commissioners are to align their practice with their health planning. Government principles for working with Aboriginal and Torres Strait Islander people should underpin evaluation tendering selection and reporting.
In Aboriginal health, it is widely accepted that evaluation reports remain ‘on the shelf’ and do not inform the next iteration in the health policy cycle. However, we found here that many reports did not even make it ‘to the shelf’ as we found only 1 in 4 reports was publicly accessible [
13]. This raises important questions regarding to what extent evidence from evaluation contributes to the next phase of planning in the policy cycle, when it is not easily available, especially to the ACCS. Given that governments place such strong emphasis on evidence-based policy and programs in Aboriginal and Torres Strait Islander health contexts, the evidence must be made available in order to close the policy-evaluation cycle. We propose that transparency and accountability across evaluation can be improved by ensuring public access to tender documents, evaluation reports and documentation of responses to evaluations.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.