Background
Australia has joined a growing number of countries to report performance data of health service providers into the public domain [
1‐
4]. Underpinning this trend are goals of increasing healthcare service-provider accountability and transparency, enabling consumers to have greater information and choice when making decisions about their healthcare, encouraging improved quality of care, and improving provider performance and productivity [
2,
5‐
8]. Research on the impact of public reporting of hospital performance data (hereafter referred to as PR) is growing; systematic reviews of such research suggest that PR stimulates change at the hospital level [
9,
10]. The evidence is uncertain, however, on whether PR improves patient health outcomes or changes consumer behaviour [
6,
9,
11‐
16]. Limited evidence on the effects of PR does not necessarily imply lack of effect, rather, need for future research [
12]. The vast majority of research on PR stems from the USA, culturally and systemically binding those results to the US health system and its people.
Unintended, negative or dysfunctional consequences of PR relate to the peculiarities of specific systems of PR and can include provider avoidance of high risk patients [
14,
17,
18], and focus on measures and targets to the detriment of quality of care (particularly when PR is linked to pay-for-performance schemes) [
19]. A number of “dysfunctional consequences” and abuses of performance data have been suggested, related to poor measurement, misplaced incentives and sanctions, breach of trust, gaming, tunnel vision, and politicisation of performance systems [
20,
21].
Australian mechanisms for PR
Australia is made up of a federation of six states and two territories – all of which are self-governing. It has a three-tiered system of government comprising the Australian government (national), individual state and territory governments, and local level municipal governments (councils). All Australian citizens are eligible for free healthcare through the publicly funded healthcare system. The private sector is also robust with private health insurers sharing the cost of private hospital and specialist care with the government and healthcare consumers. Unless entering hospital through an emergency department, access to hospitals and specialist doctors is via referral by general practitioners (family physicians).
The Australian Institute of Health and Welfare (AIHW), an agency of the Australian Government, annually release around 150 reports on aspects of health and welfare [
22] – while not necessarily written for a lay audience the reports are in the public domain. In 2011, the Council of Australian Governments (COAG) agreed to improve PR through the establishment of the National Health Performance Authority (NHPA - under the National Health Reform Act 2011) – an independent agency to monitor and report on the performance of hospitals and primary healthcare organisations across Australia. At that time, PR became mandatory for public hospitals. While it remains voluntary for private hospitals, some private health insurers require provider participation [
23]. The mission of the NHPA was “to monitor and report on the comparable performance of health care organisations to stimulate and inform improvements in the Australian health system, to increase transparency and accountability and to inform consumers” [
24]. One of the main ways PR occurred through the NHPA was via the national
MyHospitals website [
23]. The purpose of
MyHospitals was “to ensure the entire Australian community has easy access to nationally consistent and comparable performance information for public and private hospitals” [
23]. In addition, state and territory governments maintain their own PR websites. However, there is little consistency between them on the metrics reported; some provide hospital comparisons, and some report in real time, daily, weekly, monthly or quarterly blocks.
In 2014, the Australian government stated its intention to merge the AIHW and NHPA for more streamlined and efficient collecting and reporting of data [
2]. In 2016,
MyHospitals was transferred to AIHW and the NHPA was closed. Having Australia’s national system of PR moved to the AIHW, provides opportunity for a new phase for PR, one that could benefit from reflection on what has and has not worked for the existing system of PR. Since its move, few changes have occurred to the information available through
MyHospitals. Its intended audience is unchanged, being: “members of the public, clinicians including doctors and nurses, academics and researchers, hospital and health service managers, journalists and others” [
25]; and its reporting framework is also unchanged. Australia’s national health Performance and Accountability Framework [
26] forms the basis for the indicators – which in their entirety aim to report on 17 aspects of equity, effectiveness and efficiency. Currently, however, just seven of 17 proposed indicators are reported. Other indicators are “under development” or require “extensive methodological development to create accurate, nationally comparable information at the local level” [
25]. Metrics reported include hand washing rates,
Staphylococcus aureus bloodstream infections, waiting times for elective surgery, time spent in emergency departments and financial performance in terms of ‘national weighted activity units’. Indicators still pending include those for patient experience and access to services compared to need.
In this article, we examine healthcare consumer, provider and purchaser perspectives on the implementation of PR in Australia. We specifically focus on perceptions of the objectives of PR, its strengths, and barriers to greater effectiveness. In doing so, we seek to lay the foundation for better understanding how and what strategies might improve systems of PR.
Methods
Research design
Great expectations: Achieving the promise of public reporting of health service performance in Australia was a three-year (2014–2017), mixed methods study funded by Medibank, a private health insurance company. The aim of this multi-phased, mixed methods study was to identify potential strategies to improve the impact of PR on quality of care in private and public hospitals. This component of the study used thematic analysis of semi-structured interviews from expert informants (healthcare consumers, providers and purchasers) towards meeting this aim. A reference group comprising representatives from public and private sector health providers, purchasers and consumer organisations provided guidance on methodology, interview questions and identification of organisations and potential expert informants for contact for interviews. Ethics approval was granted by the Population and Global Health Human Ethics Advisory Group (HEAG), The University of Melbourne.
Purposive sampling was used to identify individuals and organisations to provide healthcare consumer, provider and purchaser opinions on PR across the public and private sectors, and across all Australian jurisdictions (all state and territory governments, and the Australian Government). Anonymity was assured for participants, many of whom provided personal as well as organisational perspectives – which were not always in harmony. The interview questions were shaped by: the aims of the project (i.e. examining the effectiveness of current PR strategies and identifying potential strategies to improve the impact of Australian PRs); reference group input (for broader industry perspectives); and the researchers’ expertise and knowledge of issues arising in the literature (e.g. unintended consequences, theories of how PR works). The draft questions were reviewed by the reference group to test their appropriateness for the indented audiences, and modified based on their input. The same questions were asked during each interview (see Additional file
1 for interview questions). Informants were provided with information about the research and researchers, and had opportunity to review the interview questions prior to their interview.
Data collection
Thirty-five organisations or individuals were approached by emailed letter with follow up phone call, to contribute to the research; one declined participation with no reason given. In total, 34 face-to-face or telephone interviews including 41 informants (i.e. two informants were present during seven interviews), were undertaken by three researchers between 17 February 2015 and 30 April 2015. One interview per organisation was conducted at a place and time convenient to the informants, normally their place of work, with no non-participants present. The interviewers did not have a relationship with the informants prior to the study. Field notes were not made as it was specifically the opinions of the informants that were required. The interviews varied in length from 17 to 51 min (average length 36 min). All interviews were audio recorded and transcribed verbatim. Interview participants had the opportunity to review and correct the transcript if they wished (one did so).
Seven consumer, 15 provider, and 19 purchaser representatives were interviewed. Informants mainly held senior positions in their organisations including chief or other executive, director/assistant director, President/Vice President, and senior or national manager. An independent patient advocate associated with a number of consumer organisations also contributed. ‘Consumer’ representative participants were from consumer health forums and peak bodies; ‘providers’ were from professional and provider associations and colleges, public and private providers of hospital services; and ‘purchasers’ were from government health departments, private health insurers, and independent government agencies (see Table
1).
Table 1
Categories of interview participants (expert informants)
1 | Consumer | Consumer | Consumer | Consumer advocacy organisations with national or state focus, and one independent advocate | 6 | 7 |
2 | Provider | Public | PrPub | National and state based health providers and provider associations | 3 | 4 |
3 | | Private | PrPriv | National and state based health providers and provider associations | 3 | 3 |
4 | | Mixed | PrMix | National medical practitioner professional colleges, associations and councils | 6 | 8 |
5 | Purchaser | Government | PurGov | Government health departments from states, territories and Commonwealth | 9 | 12 |
6 | | Private | PurPriv | National private health insurance funders | 4 | 4 |
7 | | Independent | PurIndept | National independent government agencies (relevant Authorities and Commissions) | 3 | 3 |
| Total (N) | 34 | 41 |
Questions in addition to those in the guideline were asked to some interviewees where clarification was needed. Authors MK and DD undertook n = 9 and n = 4 interviews respectively. The remaining interviews (n = 21) were undertaken by SM (see acknowledgements). Data collection ceased on reaching national, state and territory representation and a cross section of input across the informant types, as considered appropriate by the reference group. Reaching data saturation was not the basis of cessation of data collection, although analysis of data suggested a very high level of saturation, with the repetition of many themes.
Data analysis
Qualitative data analysis software NVivo10 [
27] facilitated thematic analysis of interview data [
28]. Initially, RC and MB (authors) independently undertook thematic coding of four interviews (two from providers, one consumer and one purchaser). Themes were derived from the data and coding discrepancies were discussed and resolved, leading to the development of an agreed outline of high-level themes shaped by the aims of the research and emergent issues. RC coded all remaining interviews. All authors read transcripts and participated in discussion about themes arising.
In this paper we focus on the major themes raised relating to the current systems of PR. Data were coded to high-level themes related to PR, the following themes are reported in this paper: perceived objective or purpose of PR; who uses the information; strengths; weaknesses; unintended effects; and perceived barriers to success. The last three categories contributed to the ‘Barriers to effective implementation of PR’ section in this paper. During a second round of analysis, data were re-coded within the high-level themes listed above to generate more specific sub-themes or issues. It was at this stage that the barriers were categorised as conceptual, systems-level, technical and resource related, or socio-cultural. These categorisations arose from the data, not from an imported theoretical framework. Within NVivo10 we cross-tabulated the main themes and associated sub-issue by the seven informant types (see Table
1); however, given the small numbers in each informant subgroup, such disaggregation added little to the results. Consequently, for reporting we kept disaggregation to consumer, provider, and purchaser groups. In reporting below, we table the most repeated issues relating to the strengths of the current system of PR and barriers to its more effective implementation. To contextualise the issues relating to the conceptual barriers, we include information on the perceived objectives/purpose and audience of PR. Participants did not provide feedback on the findings.
The seven two-person interviews were checked for compatibility of opinions voiced; none contained contradictory opinions. During those interviews, the two informants were not asked to independently respond to each question; rather, they tended to work together letting the person with greater knowledge answer a particular question, or one person dominated with details added by the other. Results reported in the tables refer to representative organisations (
N = 34) rather than to individual interviewees (
N = 41). Informant quotes are labelled by informant type and sector (see Table
1). The use of numbers in the results tables, indicating the number of interviews where a particular issue was mentioned, is to provide an indication of the potency of particular issues compared to others. It does not follow that informants who did not offer a certain opinion would not share that opinion; rather, they might not have discussed that particular issue. Results are indicative of opinion and are not generalizable.
Discussion
The informants who contributed to this research represented a broad cross-section of experts who, in their daily work, are in direct contact with the healthcare system in Australia, representing healthcare consumers, providers, professional associations, government departments and agencies. These results will combine with other elements of the research project that aims to identify promising strategies to improve the impact of PR in public and private hospitals. The conceptual, systems-level, technical/resource and socio-cultural barriers to PR raised by informants point to fundamental issues in PR development and implementation in Australia. Greater understanding of these issues can lead to refinement of PR systems in Australia and potentially in other countries.
Informants expressed variable notions of what constitutes PR and who should be its audience. Tensions were expressed related to framing PR so there is balance between what is best for consumers versus, best for hospitals, versus best for government departments of health. The tensions related to lack of clear purpose and target audience for PR (and were perhaps indication of the lack of entrenchment of current PR systems). This insight is not new, but it suggests progress in this area is slow. For example, in 2003, Marshall et al. stated that advocates of PR “are often unclear about the objectives of reporting initiatives and how they expect the various stakeholders to respond” [
29]. Further, in a review of PR across seven countries, it was observed that “system objectives are not always well defined and documented”, that they “typically address a number of audiences”, and that some systems expressly for patient audiences are “ill-suited” for their needs “as the nature of information collected and their presentation might require specialist knowledge in order to be usable and useful” [
3].
Amongst our informants, the primary objective of PR moved, depending on their perspective, between informing consumer decision-making, driving quality and service improvements among providers, increasing provider accountability, and increasing transparency. The mission of Australia’s NHPA had included meeting those same four objectives [
24], although the only objective made explicit relating to Australia’s national PR system (the
MyHosptials website) is to ensure “easy access to nationally consistent and comparable performance information for public and private hospitals” for consumer, provider and other audiences [
25]. Currently in Australia, the utility of
MyHospitals is hampered by the: limited array of performance indicators reported (7 of 17); lack of mandate for private hospital reporting; and datedness of the information (e.g. 2013–2014 was the most recent year reported in August 2016 for most indicators). Until remedied, Australia’s national PR system will be limited in its effectiveness and cannot be fully assessed on whether, how or what institutional performance improvements might be associated with it. While reporting of ‘hospital standardised mortality ratios’ and other mortality measures are among the planned 17 indicators, it is unclear what sort of measures of patient experience might eventually be reported. Patient reported outcome measures were suggested by informants to be of great value to consumers and clinicians and necessary to increase the relevancy of PR [
30].
Clarity of design to strengthen PR frameworks
It is apparent that alignment between a defined primary objective and the needs of a specific target audience is required when designing PR frameworks, including making explicit the data requirements and frames of understanding of the various audiences. Other important considerations are the use of healthcare quality and performance indicators that are relevant to their audience, and having that information presented appropriately. Rather than a single website for providing data to multiple types of audiences (such as MyHospitals), a national PR website aimed only at the general public might be preferable, with information for professional audiences (e.g. healthcare managers, clinicians, academics, bureaucrats) pitched differently, elsewhere.
Although there is an absence of evidence in Australia on the actual use of PR data by healthcare consumers (or by non-consumer audiences), PR is ostensibly for the ‘public’, so it follows that models of PR should aim to have greater direct impact on healthcare consumers. However, as our informants outlined, there is belief in some quarters that the public ought not be the main target audience for PR. This was cited to be due to poor health literacy, structural barriers (geographic factors, lack of choice), and lack of an engaged-patient culture (referred to by informants as a “consumerist culture”) which contributes to consumers lacking the knowledge, confidence, and practical ability to use such information. Other informants, however, considered that tax-payers (public citizens) should have ultimate ownership of the public healthcare system, and should therefore have available a transparent information platform (i.e. PR) to ensure healthcare system transparency and accountability. These two viewpoints suggest different epistemological underpinnings for PR that impact on the implementation of PR frameworks. The first suggests that the instrumental value of PR is the most important, whereas the second prioritises its intrinsic value. These align with the “government priorities for action” and “health systems professional” views elaborated earlier. When instrumental value dominates, PR should drive quality, safety and performance improvements (including emphasis on strengthening data feedback to clinicians). Such PR might have only an indirect impact on consumers through institutional improvements in quality and safety. When intrinsic value dominates, steps to strengthen PR include publishing more rather than less data for greatest transparency and accountability. Within the ‘intrinsic value’ framework it does not matter if consumers understand or use the data, they remain the primary stakeholders.
In strengthening systems of PR, lessons can be learned from countries with long established system of PR [
1,
3,
4]. For example, the USA have best practice guidelines for presenting healthcare performance data to consumers, maximising consumer understanding and awareness of PR information [
31], and prioritising “the public” in public reporting [
32]. The UK experience supports incorporating anecdotal and other consumer experience data in PR systems [
1,
33]; and both have offered information on unintended consequences of PR [
17,
19‐
21]. However, the Australian experience, as outlined in this article, also provides lessons for other countries looking to implement or strengthen systems of PR: i.e. being cognisant of and addressing the multi-levelled barriers that can prevent systems of PR from achieving greatest impact. Addressing the conceptual, systems-level, technical/resource and socio-cultural barriers identified through this research should assist in strengthening the impact of PR at consumer, provider and purchaser levels. In particular, ensuring alignment between the objective and needs of the primary audience for PR. Informants’ opinions on what a strengthened, more effective system of PR in Australia should ‘look like’ is to be the topic of another paper.
Strengths and limitations
This research comes at an important time in the development and implementation of PR in Australia. That is, in consideration of the short-lived nature of the NHPA, the transfer of administration of MyHospitals by the AIHW, the ongoing methodological development relating to indicators not currently reported on, and the interest of private health insurers keen to better understanding the impacts of PR (this study was undertaken with funding from a private health insurer).
Although ‘public performance reporting’ was described in participant information documents, it was not explicitly defined at the commencement of each interview. On analysing the interview transcripts it became apparent that informants referred to a number of different reporting systems and mechanisms including where data are not made publicly available – for example, non-public internal and external hospital performance reporting. Due to this, extra care was taken during analysis to ensure that comparisons made were appropriate and in context. Rather than being a limitation, the lack of a clear definition of PR proved an opportunity to highlight the diversity of understanding among stakeholders, and perhaps confusion for some informants, on what constitutes PR – itself a significant finding of this study.
Limited time afforded by some informants for interviews was a barrier to greater probing on some issues; this affected the depth and breadth of some data. As a result, it is possible that in some instances the number of organisations noted in the results tables under-represents the true number sharing an opinion. The results tables, however, are provided as a guide only to compare similarities and differences between the informant groups; they are a means of highlighting issues and their content. The same goes for the accompanying explanatory narrative, these exploratory data are not intended to be generalizable. Despite these limitations, the experts who contributed to this study offered a breadth of perspectives that had not been previously canvased, thus providing a significant and rich narrative of experience and perspectives related to the relatively short time that PR has been a priority within Australia’s national health reform agenda. Further research is required to objectively assess the potency of the barriers to PR that have been suggested by our informants.
Conclusions
The Australian experience highlights important issues in the implementation of PR that should be considered when implementing such systems anywhere. Clarity on the ‘primary’ objective and primary audience for PR is needed to ensure that PR systems deliver appropriate information that can lead to the greatest impacts and gains. A system of PR that tries to achieve too many objectives for a mixed audience of consumers, providers and purchasers, might fall short of fully achieving any of the intended health system improvements. More than one system of PR might be needed to meet the different data needs of various audiences. Although further research is required to objectively assess the potency of the barriers to PR suggested by informants, addressing, where possible, the conceptual, systems-level, technical/resource and socio-cultural barriers to PR, and drawing from shared lessons from international experience, should strengthen future systems of PR to better deliver information to meet audience (stakeholder) needs. This, in turn, should generate greater potential for generating continuous quality improvement in transparent, accountable healthcare sectors.
Acknowledgements
Funding for this research was provided by Medibank Health Research Fund. Dr. Stuart McLennan conducted 21 of the interviews and contributed to the design of the interview questions. The Project Reference Group consisted of representatives from Medibank Private, Victorian Healthcare Association, GMHBA Health Insurance, NSW Bureau of Health Information, Australian Commission on Safety and Quality in Health Care, National Health Performance Authority, Victorian Department of Health and Human Services and Consumers Health Forum of Australia. They provided significant contribution in identification of potential interview participants and the conceptualisation of the interview question guideline. We would also like to thank participating informants for their time and valuable insights.