Background
Efforts in assessing performance of health systems can be traced back almost three centuries, although most of the theoretical and empirical work in health system performance assessment (HSPA) has taken place in the last three decades [
1‐
3]. One of the approaches that has been used for HSPA is the league table [
4,
5]. The ultimate purpose of HSPA is to improve the quality of decisions by stakeholders in the health system, thereby contributing to health system improvements. The design, process of development and implementation of the HSPA frameworks should facilitate the achievement of this purpose [
6,
7].
Uganda is a low income country (LIC) in sub-Sahara Africa with a Gross National Income (GNI) per capita of current US $ 670 (2014) and a high burden of disease (both communicable and non-communicable) [
8]. Although some improvements have been registered over the last three decades, the country still has poor health indices with infant mortality rate at 43 deaths per 1000 live births (2016) and maternal mortality ratio at 336 deaths per 100,000 live births (2016) [
9]. Total health expenditure at US $ 53 per capita is very low; recent estimates indicate the following mix: public 15.3%; private 38.4% and development partners 46.3% [
10]. The model of governance practiced in the country is the devolution form of decentralization, with political, administrative and technical authority at the national, district and sub-county levels [
11]. The central level is responsible for legislation, policy formulation and strategic planning, resource mobilization and monitoring and evaluation. The district is responsible for operational planning and management of health services, and carries the responsibility for inter-sectoral coordination of activities designed to improve population health [
12].
In 2003 the Ministry of Health (MoH) introduced the Uganda District League Table (DLT) to track district performance given decentralized service delivery and the need to know the range of performance across the country [
13]. The objective of this study was to carry out a comprehensive critique of the Uganda DLT using a normative HSPA framework. The review was intended to provide recommendations for improving Uganda’s district HSPA, and to provide lessons to other low and middle income countries (LMICs) with similar context like Uganda’s, as well as organizations seeking to develop or modify their HSPA frameworks.
A model HSPA framework
Many of the HSPA experiences that have been studied have been developed in high income countries (HICs) [
3,
14]. Although there are marked differences in contexts between HICs and LMICs, theoretical models and experiences of HSPA developed in one context can be used to inform the study and practice of HSPA in other contexts [
14,
15]. A broad research programme on HSPA sought to learn from theoretical and empirical work on HSPA in different contexts to inform the development of new or review of existing HSPA frameworks in LMICs. The research programme was constituted by researchers based in Uganda, Belgium and the World Health Organization (WHO). The first author and four of the co-authors had been involved in the development and/or implementation of the DLT. The experience had stimulated an interest in learning about HSPA frameworks and what makes them appropriate (or not) for their purpose.
In the first stage of the research programme a model HSPA framework was developed for the purpose of reviewing HSPA frameworks for their appropriateness [
16]. A structured literature review was carried out for the purpose of identifying characteristics of a ‘good’ HSPA framework. The review was initiated with a search of the PubMed database using the search term ‘health system performance assessment’. A total of 2522 articles published in English between 1995 and 2013 were identified. A review of titles, abstracts and eventually the full articles led to the identification of 16 relevant articles, 28 more articles were identified from the bibliography, making a total of 44 relevant articles [
16]. A number of characteristics for a ‘good’ HSPA framework were extracted from the articles, which were summarized into 6 attributes by the researchers. The six attributes of a ‘good’ HSPA framework covered: the process of development; the relationship with the health systems framework; the relationship with the policy organizational and societal context; the elaboration of the framework; the institutional set up for HSPA; and the mechanisms for eliciting change in the health system. The attributes were presented to a group of Ugandan based experts for the purpose of providing broader input into the process, increasing objectivity, validating the findings and improving uptake of findings in the Ugandan decision-making processes. The individuals selected for the expert group were those with a minimum of postgraduate qualifications in public health/health economics/social sciences, and at least 10 years’ experience in health system management [
16]. The expert group validated the six attribute model HSPA framework, provided some fresh perspectives, and introduced a seventh attribute covering the adaptability of a framework in different contexts and over time. The seven attributes are presented in Table
1. The resulting set of seven attributes was used to review a number of HSPA frameworks selected from high, middle and low income countries, with the objective of determining their responsiveness, and facilitating lesson learning for LMICs seeking to develop and/or review their HSPA frameworks. This process also served to determine the appropriateness of the model for critiquing HSPA frameworks [
16]. The model for a HSPA framework thus developed and validated through these processes was utilized to review the DLT in this paper.
Table 1
Attributes of a normative HSPA Framework
Process of development (and review) of the framework should be inclusive, with the participation of key stakeholders, and involve the explicit use of evidence to indicate causal links. Embedded in an explicit health system conceptual model, including the determinants of health, system goals, constitutive elements, and actors. Relate to the national policy, organizational set-up and societal context including consideration of the level of development, epidemiological and demographic patterns, mode of government, levels and sources of health financing, governance, principles and values of society.
Well developed with
a conceptual model, a clear purpose, dimensions and sub-dimensions, and with appropriate indicators
.
Supported by an institutional set-up for performance assessment with appropriate resources and networks, including champions for performance assessment and an enabling environment. Explicitly provide mechanisms for eliciting change in the health system – indicating how the measurement of performance is linked to changes in policy, management, and delivery of services by various levels and players in the health system. Adaptable to different contexts- with history of use and or adaptation in different contexts, the length of time it has been in use and changes made to improve or adjust the framework in view of major reforms in the health system or elsewhere.
Source: Tashobya
et al.
, 2014
|
Methods
The study documented in this paper is a component of a broader research programme on the appropriateness of HSPA frameworks organized in three stages. The first stage focused on the development of a model HSPA framework as reported in the previous section [
16]. The second stage of the research programme was a critique of a HSPA framework, which utilized the Uganda DLT as a case study. In the third stage of the research programme, the findings of the first and second stages will be used to inform the design of an adjusted district HSPA framework for Uganda, and to provide lessons for policy makers and researchers in other LMICs seeking to review or develop HSPA frameworks.
In the second stage of the programme, qualitative and quantitative research approaches were utilized to provide a comprehensive critique of the Uganda DLT. Qualitative data was sought from Key informant Interviews (KIIs) and grey and published literature. The model HSPA framework together with findings from literature, and the field knowledge of the Uganda-based researchers were used to develop an open-ended interview guide. Individuals to be interviewed were purposively selected from among health sector stakeholders given experience with the development, implementation, and/or use of information from the DLT. Interviewees were individuals working with the government at the national or local government levels, international agencies, researchers, and public and private sector players. The documents selected for review provided information on the Ugandan health system context over the last 20 years, and on the development and use of the Uganda DLT. The first author and four of the co-authors worked at, or closely, with the Uganda MoH over the last two decades, which facilitated the identification of Key Informants and the location of relevant documents, especially those not in the public domain.
The interview guide sought perspectives of respondents regarding their experiences with the DLT development and implementation, assessing the DLT along the attributes of a model HSPA framework, and whether Key Informants considered the DLT successful in achieving intended objectives. All the interviews were carried out by the first author in English, between June and August 2012. At the point of 30 interviews spread over the key constituencies, descriptive saturation was achieved (see Table
2). The interviews were audio recorded, transcribed, coded, and analyzed by the first author. The outputs were reviewed by two other members of the research team. Key Informant responses were analyzed together with information from grey and published literature to inform the critique of the DLT from multiple perspectives. In one approach, inductive analysis was used, and the findings were utilized to relate the story of the development and implementation of the Uganda DLT [
17]. In the study reported on here, deductive analysis, using the attributes of the normative HSPA framework, was applied to primary KIIs’ data and grey and published literature to provide another perspective to the critique of the DLT. In addition, a quantitative aspect of the critique was carried out, whereby quantitative data from the DLT database was analyzed using hierarchical cluster analysis [
18].
Table 2
Key Informants Affiliation and Responsibility
National Level | Ministry of Health | MOH 1 |
MoH 2 |
MoH 3 |
MoH 4 |
MoH 5 |
MoH 6 |
MoH 7 |
International Agency | IA1 |
IA2 |
IA3 |
Academia | ACAD1 |
ACAD2 |
Local Governments | Political Leaders | DPOL1 |
DPOL2 |
Administrative Managers | DADM1 |
Technical Managers | DTECH1 |
DTECH2 |
DTECH3 |
DTECH4 |
DTECH5 |
DTECH6 |
DTECH7 |
DTECH8 |
DTECH9 |
DTECH10 |
DTECH11 |
DTECH12 |
Civil Society Organisation | CSO 1 |
CSO 2 |
CSO 3 |
Discussion
In this section we consider the findings from the review of the Uganda DLT using a normative HSPA framework in light of available literature and experiences, for the purpose of supporting the development of recommendations for updating the Uganda district HSPA framework, and to tease out lessons for LMIC and global researchers and policy-makers.
The study noted that during the
processes of development and adjustment of the DLT, the stakeholders that were involved were mostly from the national level especially the MoH and development partner representatives most of whom were biomedical or public health/epidemiological professionals. Experiences from other countries have shown that bringing on board a wide range of stakeholders including a mix of policy makers, data collectors and data users; a range of professions and sectors, contributes to the appreciation of the multi-sectoral and multi-faceted nature of HSPA and improves the likelihood of using resulting data for decision-making [
7,
45]. The restricted involvement of stakeholders in the development of the DLT is likely to have contributed to the limited understanding and ownership of the DLT. It is the recommendation of this study that in future processes of development and/or adjustment of the Uganda district HSPA framework should involve a wider group of stakeholders with particular emphasis on district technical, political and administrative managers; researchers; representatives of various entities that collect and use data; and include individuals with different professional backgrounds including the biomedical, public health/statistics, and social science/organizational management.
The use of evidence, and explicit models indicating causal links at the time of development of the framework, it has been shown, improves stakeholder confidence and buy-in [
4,
5]. In the case of the Uganda DLT there was no evidence of the use of data and/or modeling during the development and adjustments, which omission may have contributed to the criticism of the model by some of the stakeholders. It is recommended that in future efforts to develop/adjust the Uganda district HSPA framework data is used to justify models that are being used.
A HSPA framework should reflect the understanding that health system managers are directly responsible for the management of the health care system, and indirectly for other aspects of the health system as health system stewards, and thus facilitate performance assessment across the health system [
1]. This study has shown that there is a gap in sub-national performance assessment in Uganda as embodied in the DLT, between the
health system and the
health care system. This is not an isolated finding; the lack of clarity on how the health care system relates to the other aspects of the health systems has been noted globally, and especially in regard to health system performance assessment [
3,
14]. However, lessons can be learnt from how some HSPA frameworks have been structured to address some of these challenges. The Canadian Health Indicator Framework for example was based on the Lalonde model of the health system which highlights the non-healthcare determinants of health. The Dutch HSPA framework builds on both the Lalonde model and the Balanced Score Card to support the explicit indication of how the healthcare system relates to the broader health system [
3]. It is the recommendation of this study that the Uganda district HSPA framework should be clearly situated in the wider national health system and HSPA framework that recognizes SDH. The HSPA framework should explicitly lay out the expectations from the health care system and other sectors, and thus facilitate the MoH and DHO as they seek to hold the different sectors accountable for actions in their domains. The framework should highlight the aspects of the HSPA framework where the MoH and the DHO are directly responsible and the aspects where they track progress as stewards.
A HSPA framework should be responsive to the context in which it is situated [
3,
7]. The Uganda policy, organizational and social context is very complex, and dynamic. The DLT assumes a coherent organization in terms of priority setting, management of resources and performance assessment. To a large extent this was the prevailing situation in Uganda in the early 2000s, with the early implementation of decentralization and SWAp. However over the last decade there have been a number of changes in sector coordination and funding that negate this scenario. Additionally, the proliferation of districts, stretched health system management capacity and retention and/or recentralization of some of the functions that should have been at the sub-national levels have contributed in practice to limited decision-space at the district level.
With the DLT, data on district performance is submitted to the national level, whereby it is analysed and presented, with the purpose of supporting decision making at the national, district and lower levels. Particular emphasis is put on comparison of performance between districts across the country. The DLT takes an (upward) accountability approach, which assumes good information on resources available to the districts at the national level and much more leverage than is the case today. It is the recommendation of this study that a different approach to district HSPA should be taken. This new approach should emphasize the collection, analysis and use of data for decision-making at the district level. This is in recognition that the district level has better access to information on resources available for service delivery including human and financial resources, and the detail of operational information. Such an approach we argue will provide for a more conducive environment for inter-sectoral collaboration at the district level and ownership of health system performance by the political, administrative and the technical managers – beyond the DHO. Quantitative and qualitative indicators should be built into such a tool. Such a tool would facilitate learning at the district level. Despite the observation that decision-space at the district level in Uganda is limited, opportunities do arise which empowered managers can take advantage of to improve district health system performance. Appropriate district HSPA tools can facilitate such managers. The role of the national level would be to develop appropriate models for district HSPA, to support districts in applying these models and to build capacity for HSPA.
Uganda has put in place the legal and institutional framework for decentralization and multiparty democracy. However the promises of these reforms in regard to participation of the community in decision-making and enhancing accountability are yet to be achieved. The low capacity of the population to demand for accountability has been related to the poor levels of socioeconomic development and to the history of conflict [
46,
47]. The DLT did not facilitate the link between the health system and the community it serves for purposes of HSPA. There are some promising experiences of civil society organizations working with communities to improve their capacity to engage with the government in regard to demanding for accountability and pushing for improvements in service delivery [
48,
49]. It is the recommendation of this study that efforts are taken to learn from such examples and to develop mechanisms for providing accountability to the communities that are served (downward accountability). Civil Society Organizations have been noted to be better at such innovations and can utilize both formal and informal structures for the purpose. The new/adjusted model of Uganda district HSPA should be set up to link with such innovations.
Theoretical and empirical studies have highlighted the importance of having a well-documented HSPA framework, with a conceptual reference model, clear purpose, dimensions and sub-dimensions and performance indicators [
3,
50]. The lack of a conceptual model and designation of dimensions and sub dimensions made it difficult to appreciate how the DLT interfaced with the wider health system, and how the different aspects of the DLT interfaced with one another. Over the last 14 years attempts were made to improve the quality of the DLT indicators in line with what various scholars have indicated as desirable characteristics for performance indicators [
7,
51]. However, gaps and challenges still exist, particularly concerning the lack and/or inadequacy of indicators related to non-health care determinants of health, input and process indicators, and indicators pertaining to the management of non-communicable diseases. Lessons can be learnt from a number of countries that have implemented HSPA frameworks over time, whereby indicator lists evolve depending on the health system information requirements and the capacity for data management [
16]. This study recommends that a conceptual model should be elaborated for Uganda district HSPA clearly linking it with the broader health system, and highlighting dimensions and sub-dimensions. The objectives of the district HSPA should be reviewed with the view to make sure this relates appropriately with the current context and the data that is being collected. A strategic approach should be taken towards the evolution of the performance indicator list, starting with those that are most needed to support decision making at the district level. Our recommendation is that emphasis in the short term should be on developing/adapting good indicators for inputs, processes and for non-communicable diseases.
A number of researchers highlight the importance of the institutional set-up for the implementation of any HSPA framework [
3,
52]. Uganda has made headway in developing an institutional set-up for HSPA including articulating a Monitoring and Evaluation Framework and steady progress towards a functional HMIS. However major challenges exist in regard to the institutional set-up for HSPA. The absence of an explicit unit for data analysis and packaging at the MoH is likely to have had an influence on the evolution and poor ownership of HSPA and the DLT. Lessons can be learnt from the development and implementation of national HSPA frameworks across the world. In Canada and Netherlands the explicit investment in networks for HSPA led to learning among the stakeholders and highlighted the comparative advantage of the different entities. In Australia senior political and generic administrative managers were utilized as champions for HSPA which helped to emphasize the multi-sectoral approach [
3]. In South Africa a private company the Health Systems Trust has for several years been responsible for the analysis and presentation of district HSPA in the form of the District Health Barometer [
16]. This study recommends that an explicit unit should be indicated in the Ugandan health system and appropriately facilitated, to support HSPA across the country. Such a unit would focus on carrying out data analysis and presentation; providing models for data analysis and presentation at sub-national levels including the district level; and facilitating the development and functionality of champions and networks for HSPA.
Questions have been raised about the validity of the conclusions of the DLT given the quality of data [
17,
41]. The strategic approach to improvements in district HSPA earlier highlighted should be extended to improvements in the quality and range of data. It is the recommendation of this study that key stakeholders should agree on data requirements for district HSPA in the short, medium and long term and plans made on how to get the data, taking into consideration available government and development partners technical and financial resources. System-wide data quality assessments should be held regularly – at a minimum in selected districts annually, and across the country every 2 to 3 years.
The analysis of data and the presentation of the information produced in HSPA affects its use for decision-making [
7,
50]. The main approach to data analysis and presentation of the Uganda DLT is the league table approach. The capacity of the DLT to present many data points relating to the different districts and indicators, including ranking using a composite index, makes it a convenient tool. However there have been a number of concerns raised with the use of the DLT in Uganda which include: comparing entities which are not comparable; the use of a summary rank that is difficult to interpret; and the difficulty to relate provided information to decision-making. The MoH has acknowledged the concern of comparing districts with marked differences, and since 2011 provides categories within the DLT. However the extent of application of this approach is to list the different districts under these categories with no explicit effort to carry out any further analysis. This study recommends that alternatives/complementary approaches for the analysis and presentation of district HSPA data be sought in addition to the league table. In the quantitative component of the broader research programme this study is situated in, it was demonstrated that hierarchical cluster analysis (HCA) can be used to group districts with similarities, and provide a compromise position between the overly summarized DLT rank and detailed data on all districts and several indicators. HCA also provides an opportunity to look beyond the rank of a district, and to start asking questions about why certain districts’ performance is as observed [
18].
The study noted that current dissemination of the DLT in the AHSPR and discussion at the Joint Review Mission and National Health Assembly as appropriate, but inadequate. It is the recommendation of this study that more fora, with emphasis on regional and district level fora, should be sought for discussion of district HSPA findings. Additional opportunities for sharing district HSPA information include quarterly MoH senior management meetings and the various fora at which MoH teams meet with local governments.
Another major gap noted is the absence of an explicit mechanism through which the DLT approach was expected to influence decision-making. Studies in other countries have shown that explicit indication of the mechanisms through which the HSPA framework is expected to cause change is important as it helps to communicate this, manage expectations and evaluate implementation [
7]. This study recommends that a theory of change should be explicitly articulated in line with the HSPA framework objectives and other aspects of the framework. Building on previous analyses and recommendations in this paper we propose a district HSPA framework that has as its primary objective the provision of information for decision-making for improvements in health services delivery, and as a secondary objective, comparison of performance across districts. In line with these objectives we would recommend the following theory of change. At the district level the expected mechanism of change should be using district HSPA information for peer learning and implementation of QIIs. At the national level the mechanisms of change should be benchmarking which will inform resource allocation and development of policies and guidelines. At the community level the mechanism of change envisaged is public disclosure of HSPA information. A number of innovative approaches by civil society organizations to support generic and health system performance assessment at the community level have been noted in Uganda [
48,
49]. These approaches and opportunities should be further explored and built on.
A HSPA framework should be aligned to its particular context, but at the same time should be adaptable. The DLT lacked dynamism and flexibility across the country. It is recommended that in future development of HSPA provisions should be made to encourage adaptations of the framework across the country, while retaining a core approach that is practiced across the country. Provision should also be made for regular reviews of the framework, say every 5 years, with the view to strategic adjustments as necessary.
Limitations of the study
Some of the authors were involved in the development and implementation of the DLT. The first author was among the MoH officials that developed the Uganda DLT and were responsible for its early implementation. Four of the co-authors were involved to various extents in the implementation of the DLT. Efforts were made to minimise any bias this may have introduced into the study through a team approach at the various stages of the study including conceptualisation, development of tools, data analysis and report writing. Particular effort was made to involve the researchers that had not been involved in the development and implementation of the DLT at all the stages of the study.
CKT qualifications include: MB Ch B; MA Demography; MSc Health Policy Planning and Financing; Ph D. She is employed by Makerere School of Public Health, Uganda, as a post-doctoral fellow. She has worked in the areas of health policy formulation, health financing, health sysytem performance assessment and district support supervision.
FS qualifications include: MB Ch B; MPH; Ph D. He is an Associate Professor at Makerere School of Public Health, Uganda and has carried out extensive research and publication in the areas of health policy, health financing and human resources for health.
JNO qualifications include: M B Ch B; Msc Health Economics; Ph D; She works for the WHO Inter-Country Support Team for Eastern and Southern Africa in the Health Systems and Services Cluster. Her areas of interest include understanding what influences uptake of research findings into policy, health financing and performance assessment.
JB qualifications include: MB Ch B, Msc Health Policy Planning and Financing; M Sc Epidemiology. She is employed by the World Health Organisation in the Rwanda Country Office in the Health Systems and Services Cluster. Her areas of interest are public private partneship for health and health systems performance assesment.
BM qualifications include: MD, MPH, Ph D. He is a Professor at Institute of Tropical Medicine, Antwerp. His areas of interest are theory driven evaluation, local health sysytems organisation and performance management.
JM qualifications include: MD, MPH, Ph D. He is a Professor at Catholic University of Louvain, Belgium. His areas of interest are health financing, and the intersection of health sysytems and disease control programmes.
BC qualifications include: MD, MPH, Ph D. he is a Professor at Institute of Tropical Medicine Antwerp. His areas of interest are commnuity health fiancing, local health sysytems organisation and patient-centred care.
TM qualifications include: MB CH B and MSc Health Policy Planning and Financing. He is employed by the MoH Uganda in the Health Planning Department. His areas of interest are public private partnership for health and health financing.