Background
Sustainability is a crucial aspect of a program’s life cycle: do activities and benefits continue after original support ends, and what aspects of a program’s design and activities help ensure such longevity? In the realm of global health, donor financing has driven the development and expansion of many initiatives, particularly in the fight against HIV/AIDS [
1] – for example, the President’s Emergency Plan for AIDS Relief (PEPFAR), which is “the largest [development fund] by any nation to combat a single disease internationally” [
2] and has committed over $65 billion to the HIV/AIDS pandemic since its inception in 2003 [
3]. There are numerous other significant sources of funding for HIV/AIDS programs, including other bilateral agencies, the Global Fund to Fight AIDS, Tuberculosis and Malaria, and private donors such as the Bill and Melinda Gates Foundation. However, recent evidence indicates that such donor funding has stagnated [
4].
The term health information system(s) (HIS) refers to the collection, storage, management, processing and transmission of information within the health sector [
5], and includes things such as: district-level routine information systems, disease surveillance systems, laboratory systems, and/or human resource management information. Electronic HIS can allow for improved timeliness, use, legibility, and data quality, as well as for easier transmission between facilities. Health information systems are critical for a strong health system. They are used to improve disease surveillance, facilitate the strategic use of information, manage patients and programs, and increase service quality through more efficient and efficacious care [
6,
7]. Internationally, there are increasing demands for accountability and transparency, so accurate and timely data are important for resource allocation and to monitor and evaluate initiatives’ effectiveness [
5,
8].
Sustainability
Sustainability is the capacity to maintain program services after the end of financial, managerial, and technical assistance from external donors [
9]. Table
1 presents factors hypothesized to affect sustainability, as described in the literature, and this was the conceptual framework for this study [
10‐
13]. Program-specific factors include the degree to which project goals are clearly specified and able to show results, the perceived effectiveness of the program in achieving these results, the availability of financing for the program, and the emphasis on training within the program. Relevant organizational factors include the flexibility to modify implementation to meet local needs and conditions, the participatory nature of donor-client and donor-community interactions, the existence of an effective project champion, the extent to which the program is integrated into the host institution and activities, and the institutional strength of the implementing agency. Contextual factors include the presence or absence of concurrent donor projects (either those that compete with the program or those that complement it), the receptivity of the community to participation, and the political, economic and cultural characteristics surrounding the project. Similar determinants have emerged as important for EHIS – for example, political commitment to the system, human resource and infrastructural constraints, physical and socioeconomic environment, alongside global determinants such as donor role, the technological environment, and institutional issues such as the project environment and knowledge management practices [
14‐
17].
Table 1
Sustainability framework: determinants of sustainability
Program/project-specific factors |
Type/goal(s) | Programs/projects that are better able to demonstrate results, often by being more narrowly focused |
Perceived effectiveness | Higher degree |
Financing | Ability to secure multiple sources of non-donor financing, particularly from national sources (during or by end of program/project) |
Training | Greater emphasis |
Organizational factors |
Local-level modifiability | Greater local-level ability to modify implementation to local needs and conditions |
Donor-client interactions | Characterized by joint participation/consensus-building |
Donor-community interactions | Characterized by joint participation/consensus-building |
Project champion | Existing and effective |
Integration | Higher degree of integration within host institution, national health authority institution or activities, and/or recipient community needs/priorities |
Institutional strength/capacities | Stronger |
Contextual factors |
Concurrent projects/donor-supported activities | Fewer similar other programs/projects and/or minimization of competing health problems |
Community characteristics | Higher receptivity to participation |
Political, economic and cultural characteristics | Socio-political stability, economic stability/growth, higher governmental institutional capacity |
This study uses the above-described sustainability framework to inform a case study about the potential sustainability of electronic health information systems (EHIS). The study focuses on work in three sub-Saharan African countries supported by the Centers for Disease Control and Prevention (CDC) under PEPFAR. This article complements a broader emerging literature about the implementation, effectiveness and impact of EHIS in low-resource settings [
18‐
20] including in Latin America [
21‐
23], Asia [
24‐
26] and sub-Saharan Africa [
27,
28]. Few studies have emphasized the sustainability of EHIS, particularly with a multi-country comparison and in the context of declining donor support, as is presented here (with some noteworthy exceptions [
21,
29]).
Methods
This was a qualitative study based on interviews with major stakeholders involved in ongoing donor-funded projects for strengthening HIV care through EHIS (as identified through collaborative discussions with in-country experts familiar with the local EHIS) in Malawi, Zambia and Zimbabwe. The interview guide (attached as an Additional file
1) was developed specifically for this study, and used open-ended questions to probe for details about the constructs shown in Table
1. The interview guide was informed by the sustainability framework, and questions were developed to probe different aspects of these constructs.
Sample selection
Study systems were selected among EHIS projects funded by PEPFAR via CDC, and were chosen to represent systems that had successfully achieved broad national or near-national implementation, and had sufficient tenure of operation to have generated multiple years of sustained exposure among stakeholders. The Zambian and Malawian EHIS are electronic medical record systems, designed and implemented primarily to manage clinical encounters, but also to inform a national data reporting system. The Zimbabwean human resources information system was designed and implemented at the national and provincial levels. A description of the study sites can be found in Table
2.
Table 2
Description of study sites and systems
Baobab Health Trust (Baobab) is a non-governmental organization in Malawi that develops and deploys a national electronic medical record system (EMRS). It began its work in 2001, and following an agreement with Luke International in 2012, took the EMRS to national scale. As of 2014, 1.9 million Malawians had been registered in the system. The EMRS targets high-HIV burden facilities and has several modules, including an antiretroviral therapy (ART) module. This supports the clinical management of HIV patients and populates the National HIV Monitoring and Evaluation System. Antenatal care and maternity modules inform the prevention of mother-to-child transmission of HIV (PMTCT) and reproductive health programs. The system also includes an outpatient care module and additional modules for the management of tuberculosis, diabetes and hypertension, for laboratory management, and for national registration and vital statistics. The architecture is open-source and standards-based. Zambia’s SmartCare is also an EMRS system. The system was introduced in 2005 based on more than five years of prior EMRS work in Zambia, and was conceived primarily to improve continuity of care. It remains a patient-oriented system though it has data aggregation capabilities for reporting and analysis at other levels as well. All ART sites in Zambia are required to utilize SmartCare per MOH instructions. The SmartCare system includes multiple clinical modules including ART, voluntary counseling and testing, maternal and child health, and outpatient services. Additional modules are under development. Data from each visit are copied to a local database and to a portable SmartCard that is retained by patients. This dual data collection system allows transfer of an individual’s medical record across facilities. As of November 2013, the program had been deployed to approximately one-third of the country’s 1800 facilities. Zimbabwe’s Human Resources Information System (ZHRIS) was launched in 2009 in collaboration with Emory University. The goals of the system include providing an integrated and interoperable system to routinely produce accurate, high-quality workforce surveillance information for effective decision-making and to advance Zimbabwe’s health leadership capacity in tracking their workforce. The current Zimbabwean National Health Information Strategy calls for a single, central data repository system that integrates routine data on logistics, laboratories, administration, transportation and human resources. The Health Informatics Training and Research Advancement Centre (HITRAC) at the University of Zimbabwe is contracted to develop and deploy ZHRIS, which (as of August 2013) was utilized in all eight provinces and both of the two main cities’ central hospitals in Zimbabwe. Ultimately, the system aims to include real-time data on the training, employment and demographics of the more than 30,000 health personnel working in the country including both those working for the public sector and those employed in the private sector. |
Data collection
Project researchers from CDC and from Harvard T.H. Chan School of Public Health traveled to the study sites between July and November 2013. All data collectors had training and experience in conducting qualitative research, including data collection and analysis. The team held in-depth discussions with stakeholders from the government, health facilities and implementing partners. Example interviewee types included employees at government ministries, clinical and data/clerical staff at health facilities, and those involved “upstream” in the EHIS (such as software developers, managers, advisors and board members). These respondents were identified by experts in the three countries, and were purposively selected to ensure knowledge of the system and to offer a broad range of experiences with the system. The objective of these interviews was to document features of the EHIS and to identify progress toward country sustainability as per the sustainability framework (i.e., constructs in Table
1). In total, 58 interviews were conducted. Interviews were semi-structured, one-on-one with key informants, based on a standard study protocol, and lasted on average one hour. All participants were given a consent form and were asked to provide oral consent before beginning the interviews. Ethical approval for this study was obtained from CDC, the Harvard T.H. Chan School of Public Health, and host country government offices (Malawi, Zambia and Zimbabwe).
Data analysis
Interview notes were written by hand. These data were typed up, and were independently analyzed by the interviewers as subsequent interviews were still taking place. This allowed for initial analysis, for any emergent themes to be further evaluated, and served as a method of organizing interview transcripts. Additionally, by beginning to analyze transcripts while data collection is ongoing, researcher(s) can become grounded in the data and additional interviews can be modified to focus on better understanding weak themes and to validate unclear or new ideas brought forward in prior interviews [
30].
Patterns were identified from the interview texts, first by inductive analysis for emergent themes, which were then grouped into the constructs of the sustainability framework. This initial coding and sorting of results was done independently by the 3 interviewers [CM, AS, CB] and then this group examined and discussed the preliminary thematic sets of results, particularly focusing on commonalities across all study countries and areas of difference, for example by system type. The narrative synthesis for each construct was written and then checked by revisiting the original interview text to ensure data were not de-contextualized, and that verbatim quotes were interpreted correctly based on their original context. There was also a member checking process, whereby the main results were discussed with the study participants to confirm information received by the interviewers. Other secondary data sources (i.e. program planning documents, progress reports) were also used to triangulate and ensure validity of the results.
Discussion
This study aimed to better understand how to build a promising environment for EHIS investments by exploring constructs from health systems sustainability frameworks. Many studies of health program sustainability have looked at traditional clinical or public health services, but EHIS have unique characteristics – from technical complexity to interactions with broader societal and technological trends – that may differentially affect program sustainability, especially in resource-poor settings.
These case studies highlight the importance of aligning perspectives across partners and across levels of the health system. Such harmonization will improve EHIS sustainability by aligning incentives and setting manageable expectations while more robustly integrating EHIS activities into the health system. Agreement on system vision and goals, and on how to measure these, was frequently noted by participants to be a crucial step in introducing a successful and sustainable EHIS. Accordingly, clarity and concordance on goals also enabled championing whether top-down or bottom-up; this has been found to be important in other instances of EHIS implementation as well [
27].
Attaining such universal agreement on the mission and role of an EHIS may be an important goal, but it is a difficult one to achieve [
19]. Different partners may enable this in a variety of ways. Donors can encourage participatory and consensus-building activities to set comprehensive project goals and metrics. Governments should also take an active role in such processes, engaging stakeholders from different sectors, including clinical areas, monitoring and evaluation, and planning departments. Finally, EHIS organizations can strive for transparent strategic planning and monitoring processes. Other studies have cited institutionalizing EHIS and other information systems as an important aspect of sustainability [
29], citing the risk of such projects remaining in “pilot mode” if they do not have a long-term, multi-partner implementation commitment [
21].
System users can play a central role in project design, and their inclusion may help improve sustainability. A main objective of EHIS is to reduce burden on health workers. The additional burden of parallel paper- and computer-based systems can be, and has been, a key challenge to the introduction and implementation of these systems in these three countries, and has been cited elsewhere in the literature [
24,
28]. Managers and policymakers in all three countries emphasized the systems’ potential for simplifying and accelerating data collection. However, those tasked with operating the systems frequently expressed frustration that the dual management of paper and electronic records resulted in duplication, rather than elimination, of efforts. Frustrations were especially high among health workers who had been promised that EHIS would streamline their workflow. Abandoning paper in favor of electronic recordkeeping requires a leap of faith, and EHIS implementers should request such a switch only when the system can reasonably handle this demand. Nonetheless, such a transition should be a near-term goal even if it can only be achieved incrementally by specific modules within the software or in only some geographic areas of a country. Respondents from each of the three countries emphasized the fact that system users must perceive tangible benefits if we expect them to use, let alone rely upon, the EHIS. The sooner this can happen, the more reasonable it is to move the health system towards computer-only information systems. Perceived usefulness has emerged in the broader literature as an important determinant of EHIS success [
25].
The dynamic nature of the health workforce also emerged as an important issue. The low level of computer literacy within the health workforce is an important constraint to fully effective EHIS deployment. High staff turnover exacerbates this challenge, resulting in constant pressure for training on new EHIS systems. EHIS projects can mitigate the effects of this on system sustainability by giving health workers portable skills, for example, via improved pre-service trainings and standardized software and use protocols throughout the health system. This would reduce the need for constant re-training as health workers move around within the system. It is also crucial to increase public sector salaries to avoid losing trained staff (from EHIS developers to downstream health workers) to private sector employers also in need of their skills. Computer literacy is likely a waning challenge as more youth enter the workforce already familiar with computing, and EHIS trainings may need to adapt their focus from basic skills-building to improved use of technology for providing efficient and high-quality care. EHIS training should be viewed as having an important spillover effect in general increased computer comfort. Human resources has also been cited as a critical factor for successful EHIS implementation and sustainability in other studies from low-resource settings [
18,
20,
22].
Lastly and particularly notably, this study highlights the importance of understanding sustainability as a concept that reaches beyond financial stability. There are many other essential determinants of programs’ likely perpetuation and donors should invest in activities that support these additional pillars of sustainability.
This assessment has some limitations. First, the three selected cases are a small sample, and were purposively selected for their successful broad national or near-national EHIS implementation and adoption. Therefore, these cases likely reflect stronger, more effective implementations of EHIS projects than average. Conversely, this arguably makes the experiences of this set of selected systems more important than more limited implementations of EHIS projects, given the high stakes for the health system if such widely-implemented systems are not sustained. All were funded by the same organization (PEPFAR). This eliminates variability in donor characteristics, which might have affected outcomes, but may limit the external validity and generalizability of the results. Additionally this assessment interviewed a limited number of stakeholders per country. It should be noted, however, that interview responses were highly consistent across countries. As a qualitative exercise, there is no statistical test on the internal validity of these results and no way to control for omitted variables or other sources of bias. The diverse study team participated in different ways throughout the research process (including study design, data collection and analysis) which may have helped mitigate this problem, but it cannot be eliminated completely. Lastly, these systems are undergoing many changes and this research captured only a “snapshot” of these efforts. An ideal study design would include future assessments including after a change or termination in funding, to empirically evaluate the predictions generated here.
Our goal was not to assess the merit of EHIS investments, which is a separate and important discussion that should be informed by national priorities as well as data from impact evaluations and cost-effectiveness analyses [
31,
32]. These have been acknowledged to be largely lacking in the literature around EHIS [
33]. We encourage further study on these topics, as well as on other important EHIS-related issues such as security, data confidentiality, and the appropriateness of EHIS investments by donors given many competing program priorities and the potential for misalignment of needs among national and international stakeholders.
Conclusions
A broad range of stakeholders confirmed the importance of a number of sustainability determinants, both in guided interview questions corresponding to the framework, and in open-ended questions to elicit their own unprompted perceptions of critical determinants. These findings underscore the importance of creating an enabling environment for program sustainability, including by fostering communication between stakeholders for aligning perspectives and agreeing on a system’s goals, engaging users in the design and implementation process, and taking a broad view of sustainability that looks beyond financial dimensions to other important determinants. This will be critical for donor-led investments such as EHIS in low-resource settings. Achieving sustainability is a resource-intense endeavor, but will be necessary to ensure the long-term success of these programs and to see improved health outcomes.
Acknowledgments
This work could not have been completed without the support and instrumental assistance of staff from all three country CDC offices, the Ministries of Health and implementing organizations, including EHIS developers, in Malawi, Zambia and Zimbabwe. We are most grateful to all interviewees and study participants, as well as to Andrew Mitchell and Callae Snively for research assistance in early study phases.