Background
Driven by the international health agenda that supports the United Nations Millennium Development Goals, efforts are underway across Africa to improve access to health care and reduce barriers to service uptake [
1,
2]. Critical to the success of these efforts is the capacity of national governments to monitor effectively patterns of service use through time so that the impacts of changes in health policy or improvements in service delivery can be evaluated. Such capacity, however, is rare in Africa and other resource-constrained regions [
3‐
5]. Existing efforts to monitor service use are often driven by international actors rather than national governments and are generally focused on specific interventions and age groups [
6‐
8], limited to small geographical regions [
9‐
11], or based on infrequently repeated national studies such as demographic and health surveys [
12] that are insensitive to changes occurring over shorter timescales [
13‐
15].
The importance of generating reliable statistics on variables such as service use is recognized by African governments, as demonstrated by the widespread uptake of assistance in statistical capacity-building offered by initiatives such as the Health Metrics Network [
16]. Most Ministries of Health across Africa operate some form of health management information system (HMIS) as their primary instrument for generating health system statistics, often representing the majority of national expenditure on health data. Amongst other functions, HMISs generally coordinate the routine acquisition of monthly treatment and attendance records from health facilities nationwide and their compilation into a single national database. In principle, then, HMISs are the most appropriate and widely available tool for monitoring levels of service use within countries [
17].
Despite the apparent suitability of an HMIS for monitoring a wide range of important health system metrics, and the substantial resources invested in their development and operation, the extent to which data from HMISs are used to generate statistics of use to decision makers is extremely limited [
18]. The endemic under-use of hugely expensive HMIS data represents an unacceptable inefficiency in already resource-constrained health systems and can be attributed largely to the perceived unreliability of these data, due primarily to poor data coverage [
19‐
21]. Typically, many health facilities never report monthly records to the HMIS, or do so only intermittently, leading to substantial gaps in national data [
22,
23]. Whilst international initiatives [
16,
24] to improve health information infrastructures in resource-constrained nations are to be welcomed, several decades of previous efforts by international donors have rarely yielded substantial improvements in HMIS data coverage [
17,
25].
In contrast to the widespread perception that existing HMIS data are inadequate for quantifying important health system metrics, recent studies have illustrated that such data can be used to answer certain questions reliably by employing appropriate statistical approaches [
26‐
28]. In this study, we illustrate how levels of service use can be tracked reliably from incomplete HMIS data. We use a space-time geostatistical framework to account for the potential biases introduced by missing data and focus on the example of temporal changes in the use of clinics across Kenya between 1996 and 2004, a time of major changes in national health policy. In doing so, we aim to present a tool that can enhance capacity for evidence-based decision making using existing data and, more broadly, strengthen the case for renewed focus on the use of HMISs for health system planning and monitoring in resource constrained countries.
Discussion
We have used a robust geostatistical method to monitor changes in clinic use in Kenya over a 9-year time series reported by an imperfect national health information system dataset. By applying space-time geostatistical methods to minimize any statistical bias introduced by missing monthly data we were able to present, for the first time, reliable monthly and annual time series of the mean level of service use at the national and provincial level. Such output has immediate potential to enhance the capacity of decision makers in monitoring nationwide patterns of service use and assessing the impact of changes in health policy and service delivery.
By developing our approach for the case of Kenya during the 1996–2004 period, we have been able to reconstruct national service use patterns during a time of major changes in health policy and the resulting time series are able to reveal some striking features that are likely to be of direct interest to decision makers. Interpretation of these features serves to illustrate the potential of incomplete HMIS data, when handled appropriately, to detect important and policy-relevant changes in health service use. Of particular interest is the pattern of nationwide decline in service use between 1996 and 2002, followed by a sharp rise in the government-run sector beginning sometime during 2002. By 2004, annual service use in this sector had increased by approximately 45% compared to the nadir of 2002. The patterns observed at the national level were replicated sub-nationally, between different levels of the health service provision and whether malaria or non-malaria attendances were considered. Furthermore, the absence of any similar patterns among the faith-based sector in outpatient numbers during the same period suggests that the factors that stimulated the changes in the government-run clinics were specific to that sector.
The observed gradual decline in utilisation between 1996 and 2002 could be attributed to a general deterioration in the quality of government-run health services in terms of personnel, drugs and infrastructure, increases in user charges at government facilities, and a parallel growth in private commercial providers [
49]. There could be several possible explanations for the sharp reversal of government clinic use in 2003 and 2004. It seems extremely unlikely that within such a short time period overall disease incidence would have increased by over 45% among the general population. Furthermore, the similarities between patterns for diagnoses of malaria, a vector-borne disease susceptible to short-term inter-annual variations, and the remaining non-malaria diagnoses suggest that the rise in service use in 2003–2004 was an indication of general health service use rather than a disease-specific change.
There have been several important changes in national health policy and services since 2002 that may have resulted in nationwide changes to service use behaviour. In mid-2004 there was a major change in user fee policy in Kenya with the introduction of the '10/20' initiative [
50] that replaced an inconsistent system of widely varying fees with a standard fee of 10 Kenyan Shillings (KShs) at dispensaries and 20 KShs at health centres (equivalent to approximately 0.14 and 0.28 USD, respectively). This policy was widely adhered to in the early stages of its implementation and resulted in a significant net reduction in fees charged [
50]. Increases in utilisation have been associated with the reduction or abolition of user fees in Uganda [
51], South Africa [
11], and Madagascar [
52], and the abnormally high utilisation seen in our time series in July 2004, and higher average monthly utilisation in the last 6 months of 2004 compared with the first half of the year (Figure
2) seem consistent with this explanation. The inflexion point in utilisation occurs during 2002 – some time before the formal introduction of the 10/20 policy in mid 2004. The observed increases during this period correspond temporally to a series of political and health system changes: the arrival of a new government in December 2002, a substantial increase in Ministry of Health funding for essential drugs during 2003 [
53], and the widespread media coverage in early 2003 of the Minister for Health's announcements that the government was committed to the provision of free malaria care treatment and a general abolition of user fees for vulnerable groups.
It is not the intention of this study to test formally different explanations for the various features revealed in our reconstructed time series. The results we present demonstrate that, contrary to the widely held perception, imperfect HMIS data can be used to monitor reliably a fundamental health-system metric: the extent to which a population is using health facilities. This monitoring can be implemented effectively at both national and provincial levels, and has sufficient sensitivity to detect both month-to-month variation and longer term trends. The use of a previously-developed geostatistical procedure that accounts for missing data allows the minimisation of bias in adjusted time series and the representation of uncertainty without the requirement of constructing detailed covariate datasets that are currently unavailable at the facility level.
There are at least three important caveats associated with the approach we present in this paper. Firstly, we address the critical problem of missing data and the confidence intervals we present account for the uncertainty introduced by the need to predict these missing data. We do not, however, address the inherent uncertainty of the data itself. We have assumed that, where a monthly record is present, the tally of outpatient visits is correct. The quality of HMIS data is known to vary widely, and the reliability of individual records cannot be quantified without substantial further studies or programmes to audit HMIS data quality. A second caveat arises from the need to limit our analysis to the cohort of facilities that were known to be operational at the beginning of the study (1996). Inevitably, the opening of new facilities may affect patient numbers at existing facilities and information to quantify the magnitude of this effect was not available. The most plausible influence of new facilities, if any exists, is the reduction of patient loads at existing ones. Such an effect would have exaggerated the observed decline in mean attendance levels between 1996 and 2002, but then mitigated the observed post-2002 increase. A third caveat is that the approach we present relies upon the availability of georeferencing information (latitude and longitude coordinates) for each facility [
29], and such spatially referenced databases remain the exception in Africa. The need for such databases is becoming more widely recognised, however, and it is hoped that initiatives such as the World Health Organisation's Service Availability Mapping project [
54] will increase their availability in the future.
Conclusion
Many resource-constrained countries lack the evidence base for timely and effective health system decision-making, and this is exemplified by the scarcity of reliable data on health service use. Despite massive investments in HMIS across Africa, the poor data coverage of these systems has led to their gross under-use as an evidence base for decision-making at the national level. In this paper, we demonstrated an approach that enables incomplete HMIS data to be used to generate reliable information on changes in health systems, using the example of service use in Kenya. Specifically, the approach provided robust time series of mean levels of outpatient utilisation, with associated measures of uncertainty, that reveal for the first time substantial changes in service use over the last decade. Such information is of obvious utility to decision makers in monitoring nationwide patterns of service use and assessing the impact of changes in health policy and service delivery. The approaches presented in this paper will continue to provide a robust monitoring mechanism in Kenya, and serve as a template for other countries in the region with imperfect national data.
Acknowledgements
The authors are grateful to Dr Ester Ogara, the head of the HMIS department, Kenyan Ministry of Health, for her department's support during this work. We are grateful to Joanna Greenfield for her comments on the analysis and manuscript and to Briony Tatem for her dedicated assistance in formatting the dataset. Professor David Rogers is thanked for helping with a Quick Basic programme to ordinate digital HMIS records for import into Access. RWS is a Wellcome Trust Principal Research Fellow (#079080); SIH is a Wellcome Trust Senior Research fellow (#790091); and AMN a Wellcome Trust Research Training Fellow (#081829). This paper is published with the permission of the director, KEMRI. This paper forms part of the output of the Malaria Atlas Project (MAP [
55]), principally funded by the Wellcome Trust, UK. This study received financial support from the School of Geography, University of Southampton, the Wellcome Trust (grants 058992 and 056642) and the Kenya Medical Research Institute (KEMRI). CAG is a member of the Consortium for Research on Equitable Health Systems (CREHS) which is funded by the UK Department for International Development. The funding sources had no role in study design; analysis, and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.
Competing interests
The author(s) declare that they have no competing interests.
Authors' contributions
PWG coordinated the study design, was responsible for the analysis and drafted/completed the manuscript. AMN, PMA, SIH and PG participated in the data assembly, analysis and drafting of the manuscript. SSK, CAG, SIH, PMA and RWS participated in the study design, analysis, interpretation of the findings and finalisation of the manuscript.