Health systems performance assessment in low-income countries: learning from international experiences

Over the last three decades, efforts have been made to develop performance assessment frameworks that take into consideration the peculiarities of health systems, including the various determinants of health, the consideration of health (by some) as a public good, and the multiplicity of stakeholders in health with different perspectives on health systems performance [1‐3]. These frameworks have been developed largely in the context of high-income countries (HICs) [4‐6].

Many LICs struggle with questions like the following: How can health system stakeholders determine if the health system is (not) performing as it should? How can the reasons for this be established? What tool(s) can help governments carry out their stewardship role? An appropriate health systems performance assessment framework can be such a tool, help answer these questions, and support evidence-based decision-making. Such tools are important in all circumstances but are particularly crucial in LICs given the markedly limited resources versus the huge needs.

Health system performance assessment (HSPA) frameworks are determined primarily by issues high on the agenda in a health system. The key issues in health systems in HICs tend to revolve around containing costs while maintaining high-quality services, in an environment of advancing technology and high expectations from the society [5, 7, 8]. In low-income countries (LICs), however, the priority is increasing geographical and package coverage of basic services in the face of marked needs and minimal funding [9‐12]. These differences in health system agenda issues reflect distinct differences in socioeconomic development and demographic and epidemiological profiles [13, 14].

A few HSPA frameworks have been developed in LICs in the recent past, but most of the research on HSPA has been carried out in HICs; experiences in LICs tend not to be explicitly documented, and few have been studied [15‐18]. An example of a HSPA framework in a LIC is the Uganda District League Table (UDLT), which was introduced in 2003 to compare performance among districts and determine ‘good’ and ‘poor’ performers, and the reasons why. This was in the context of devolution, with the mandate for overall stewardship, resource mobilization and allocation at the national level, and the management for service delivery at the district level. The table includes a number of input, process, and output indicators, some of which are used in a composite index for ranking the districts from the ‘best’ to the ‘worst’ performer [19]. The UDLT has been in use now for 10 years, and though it has been noted to have achieved some of the intended objectives, a number of challenges have been noted [20, 21].

Most of the experiences and research in HSPA have taken place in HICs. However given the widely differing contexts, it is not desirable for a LIC country like Uganda to just copy these experiences. This paper is part of broader research aimed at studying the Uganda District League Table with the purpose of updating/adjusting it to provide an appropriate HSPA framework for the district level in Uganda today. This specific study has two objectives: to develop a set of attributes for a ‘good’ HSPA framework from literature and experiences in HSPA in different contexts; and to utilize the attributes for a structured approach to learning lessons from international experiences in HSPA. The attributes and lessons learnt will subsequently be utilized for updating the district health system performance assessment in Uganda. Other LICs seeking to develop/improve their HSPA frameworks can adopt the approach for their contexts.

The study was carried out in three stages. The first stage, a literature review, served to generate an initial list of attributes for the HSPA framework. The second stage involved the validation/contextualization of the attributes to LICs using a Uganda-based expert group (EG); and the third stage used the attributes to review a selection of current HSPA frameworks for the purpose of learning lessons for LICs seeking to develop/adjust frameworks.

There is a growing body of literature on HSPA, especially from the last three decades. The publishing of the World Health Report 2000 is one of the landmarks, and has stimulated debate, research and experimenting in HSPA. Most of the research and experiences are from HICs and international and regional agencies like the WHO and the Organization for Economic Cooperation and Development (OECD). In the more recent past, mostly since 2000, some low- and middle-income countries have developed frameworks and set up institutions for HSPA. However, there are few system-wide frameworks in LICs, and further still, research and publications on HSPAs in LICs are limited. Most of the material in this article on the Ghana and South Africa HSPAs has come from national/agency websites, with minimal information from peer-reviewed journals.

Some authors have proposed that rather than just copying models/approaches a structured approach to learning from experiences in health systems with differing contexts can be of benefit [90, 91]. In HSPA it has been noted that with due recognition of cultural, economic, demographic, organizational and political differences, it is possible to learn from one another [17, 62]. A structured approach has been taken in this research, whereby a combination of literature review for derivation of attributes of HSPA frameworks and validation of these attributes by a Ugandan-based EG were combined with the application of the attributes to review the experiences of six HSPA frameworks. This process has highlighted a range of experiences and provided an opportunity for learning lessons by LICs seeking to develop/adjust their HSPA frameworks. This section documents some of the lessons teased out by this exercise and a summary is provided in Table 4.

The review showed that the most common mode is to involve policy makers and technical experts at the national level in the development of the HSPA, with the government leading the process. Some countries/agencies, however, have shown that there are benefits from doing it differently. Canada’s extensive involvement of various stakeholders across several levels and constituencies, including civil society agencies, is seen as having contributed to a broader ownership of the framework [15]. In the Netherlands, the explicit involvement of researchers is said to have made the framework more evidence-based and has resulted in several publications on HSPA that provide information in the public domain about these experiences [75]. The development of the SA DHB by a private entity has shown that this approach is possible and has facilitated wide availability of the resulting information [87]. The appropriate mix of stakeholders will depend on the particular context. The limited participation of other stakeholders, including consumers of health system services, is a gap in many of the frameworks.

Different countries/agencies have different approaches to conceptualizing a health system. The elaboration of a health system conceptual model before the development of the HSPA framework by WHO, was ground breaking [14, 23, 24]. The use of the Lalonde model at the health system level and the BSC at the healthcare management level as in the Netherlands is interesting because it provides the possibility of highlighting accountability relationships and a way to introduce market orientation into the assessment [17]. On the whole, though, the contribution of healthcare to the broader health system and specifically improvements in health remains difficult to quantify [92]. The responsibility for assessment of and action on aspects of the health system beyond healthcare management is often ambiguous. This ambiguity is likely to be a bigger issue in LICs where major challenges in sanitation, education, social infrastructure, and access to safe water exist which are known to have a big impact on population health [53].

Reviewing the different HSPA frameworks in regard to the policy context of the various countries and values of the different societies revealed marked differences, as expected. A number of lessons can be learned by countries seeking to develop their frameworks, including the basic fact that contexts differ, which should be taken into consideration in the development of a HSPA framework. Some countries have taken particular efforts to make the frameworks relevant to their situations. The Canadian Health Indicator Framework (CHIF), provides a number of lessons: the consideration of the decentralized nature of the country; explicit consideration of minorities, including special efforts to collect data from these communities; and the emphasis on efficiency given cost containment as an issue in the Canadian health system agenda [15, 74]. The WHO framework, on the other hand, sought to be generic because of the mandate of the organization and attempted to provide an approach that many countries could use. This however was the cause for controversy around the WHO HSPA Framework as it was deemed to imply universal values and/or their application [50, 51, 93, 94].

Some of the differences between contexts, including between HICs and low- or middle-income countries and how they relate to HSPA frameworks, may be subtle although no less important. Examples include differences relating to levels of governance, literacy, various forms of empowerment, and expectations of the population. The involvement of civil society agencies in the development and use of the CHIF could be taken as an indication of the relatively well-developed state of governance in Canada whereas a private entity being responsible for the DHB may indicate a younger and possibly dynamic governance environment in South Africa [76, 79].

The elaboration of the different HSPAs showed substantial similarities among the HICs and differences between them and the low- and middle-income countries. The HIC frameworks reflected a focus in performance assessment of the healthcare system and reflected the epidemiology, demography, and health financing arrangements in these countries. This focus was evident at the dimension, sub-dimension, and indicator levels, and especially with input and process indicators [5, 15]. The frameworks from low- and middle-income countries’ on the other hand, maintained more population/health system level assessments in addition to reflecting health system organization, demography, and epidemiology. The preponderance of population level indicators may reflect health system agenda issues – the high morbidity and mortality levels, consideration of non-healthcare determinants of health, and concurrence with international aspirations like the Millennium Development Goals. However, it could also be the result of ambiguity in health system and HSPA conceptualization and challenges in data availability for the more specific service/provider level information [9, 73]. Relating the different data pieces to create a story and explain inadequacies in performance was often not done in the different frameworks.

Experiences vary regarding the institutional set-up for HSPA. Possible lessons include that substantial and long-term investment in technology, methods, and networks is beneficial in building a robust HSPA set-up as in Canada and that champions are useful in putting and keeping HSPA on the agenda, as noted in Australia, Canada (ministers of health and heads of government) and the Netherlands (policy makers and researchers). However, some of the experiences show that it is possible to start modestly using routinely available data of less than stellar quality, as was done in Ghana and South Africa. Collaborations with other institutions/approaches have been useful in such cases as seen with the SA DHB and the South Africa Health Review [72, 77]. Hyder AA (2003) proposed a similar approach for Pakistan [95]. This is likely to provide the stimulus needed to make the necessary investments in HSPA. Even the HICs frameworks considered in this study did not have all the required resources in place when they started, and substantial investment had to be made along the way [14, 74]. It is important, though, to monitor for data availability and quality and have pragmatic plans for scaling up, including setting up systems for data quality assessment. A key question to ponder particularly in LICs is how much investment should be made in HSPA, relatively, given the many needs in the health system and beyond?

HSPA frameworks are useful if they affect decision-making, and experiences with the frameworks show that countries studied differ in this regard. Possible lessons include use of complementary data sources and analytical approaches; mixing internal and external mechanisms for eliciting change; and encouraging collaboration among different entities in the health system and beyond. The Canadian experience using a consortium including the Ministry of Health, Statistics Canada, and professional bodies like the Canadian Medical Association and Canadian Council for Health Services Accreditation, the use of technology and innovation, and provision of information to the public is of interest to note. The mechanisms employed for eliciting change are a combination of continuing quality improvement and public accountability [15, 96]. In Australia, the emphasis is on a generic approach to management with the ministries of health at the different levels working together with management/accountability bodies to develop HCAs and financial incentives [15].

The above experiences show that it is important to think about how the framework will elicit change at the time of development and not later. Some of the frameworks though were not strong on this. Some authors argue that more effort is required in further analysis of the ‘black box’ between outcomes and activities from which they resulted. This is related to the nature of a health system as a complex adaptive system – multiple actors, non-linearity, unpredictability [27, 29, 30]. The use of evidence-based studies and consultation of stakeholders, and being explicit at what needs to be done, including the utilization of systematic outcome mapping – tracing of all the steps that lead to the outcome of interest - has been proposed. This has been favorably contrasted with just disseminating information and assuming it will lead to decision-making [71, 97, 98]. In the reviewed experiences, information was limited about the unintended and often negative effects of HSPA. Yet these have been documented as a major challenge by some authors from both the theoretical and empirical perspectives and are said to include tunnel vision, gaming, myopia, and misrepresentation of data [39, 56, 99].

A number of experiences with the development and implementation of HSPA frameworks was reviewed here. Some, like the CHIF and the WHO HSPA framework, have been in place for more than a decade; others, like the Ghana Holistic Assessment and South Africa District Health Barometer, are much more recent. Different processes have taken place in the different countries. It has been noted that countries are more likely to adapt frameworks from certain contexts, as shown by the example of the Netherlands and Australia adapting from the Canadian framework; Ghana and South Africa, however, were seen to have adapted from the generic example of the United Nations agencies, including WHO.

In addition to viewing the HSPA frameworks against individual attributes, it is important to see them as a whole. This perspective is necessary because it is good to have the comprehensive picture of the HSPA framework, and not relate with it in a piecemeal manner which might happen with a focus on individual attributes. Indeed this is essential given our working definition of a HSPA framework, and the ‘dynamic complexity’ of health systems as argued by some authors [27, 30, 39]. The attributes are obviously interrelated; for example, the process of development is clearly related to how the framework relates to the context and in turn, this is linked to the purpose, dimensions, and the indicators. The attributes thus bear a complex (non-linear) and context specific relationship to one another, in terms of content, timing, and the stakeholders to be involved. The relationship is likely to be influenced by amongst other things, whether it is new development or review of a HSPA framework; whether there is an articulated health system conceptual framework; the level of governance; and the level of development of systems including information management systems. Viewed comprehensively, the Canadian and WHO HSPA frameworks, though different, were responsive to a number of attributes and provided the widest range of lessons for LICs.

LICs seeking to develop/adjust HSPA frameworks need not reinvent the wheel. A structured approach to learning from the experiences of other countries/agencies including HICs can be useful. In this study literature review and input from a group of Ugandan experts provided a set of attributes for a HSPA framework. The attributes covered process of development, clarity of the health system model, relationship with policy/organizational context and social values, elaboration/content, institutional set-up for performance assessment, mechanism for eliciting change in the health system, and adaptability.

The attributes have been used to learn lessons from six currently in use HSPA frameworks. A number of lessons were noted from the experiences of other countries and agencies, and include (a) the recognition that it is possible to involve a range of stakeholders during the process of development and review of a framework and such inclusion is beneficial to the process; (b) explicit health system and HSPA conceptual models facilitate clarity of relationships between different entities and supports attribution; (c) policy and organizational context can be appropriately reflected in the framework, making it more relevant and usable; (d) there are marked differences between the structure and content of frameworks in HICs and low- and middle-income countries; (e) champions can make a positive difference for HSPA; you can start small, but substantial and sustained investment facilitates optimal functionality of HSPA; and (f) mechanisms for eliciting change in the health system should be developed alongside the framework. Some gaps in HSPA have been identified including: limited participation of those who benefit from the health system in development and use of HSPA frameworks; ambiguity in regard to health system conceptual models in some frameworks; inadequate distinction between the health system and the healthcare system; and a limited understanding of what works in terms of eliciting change in the health system.

The attributes developed in this study can form the basis of guidelines for a country/entity like Uganda seeking to develop or review its HSPA framework. More research on HSPA development and application, especially in LICs, is required to build the evidence-base to enable learning from different experiences.

CKT qualifications include: MB Ch B; MA Demography; Msc Health Policy Planning and Financing; She is currently undertaking a Ph D at Institute of Tropical Medicine Antwerp and Catholic University of Louvain. She is employed by the Ministry of Health, Uganda where she has worked in the health sector in Uganda for the last 22 years mostly in health policy formulation, sector monitoring and supervision;

VCdS qualifications include: MD; MPH; She lectures at the Institute of Tropical Medicine Antwerp, Belgium; She has worked in the health sector in Brazil and various African countries.

FS qualifications include: MB Ch B; MPH; PH D. He is a lecturer and Makerere University College of Health Sciences’ School of Public Health and has carried out extensive research and publication in health systems in developing countries.

JNO qualifications include: M B Ch B; Msc Health Economics; She is currently undertaking a Ph D at Institute of Tropical Medicine Antwerp and Catholic University of Louvain. She works for the WHO Regional Office for Africa in the health system cluster.

JM qualifications include: MD, MPH, Ph D. He is a Professor at Catholic University of Louvain, Belgium.

BC qualifications include: MD, MPH, Ph D. he is a Professor at Institute of Tropical Medicine Antwerp.