Background
Health system strengthening (HSS) is a growing area of development assistance for health that aims to support WHO’s six health system building blocks of service delivery, health workforce, supply systems, financing, health information systems and leadership [
1]. In 2015, $2.7 billion in development assistance went to HSS, an 80% increase in yearly spending since 2004 [
2].
One important HSS funder is Gavi, a public-private alliance that supports new and underused vaccines in developing countries. In 2004, a report and stakeholder meeting determined that weak health systems were undermining Gavi’s immunization program investments [
3]. In response, Gavi began offering HSS grants in 2005. HSS support has continued to expand, and in 2015 $170 million of Gavi’s $1.6 billion in grants went to HSS [
2]. The objective of Gavi HSS grants is to increase and maintain immunization coverage by overcoming health system bottlenecks that impede progress. While each country chooses to spend Gavi HSS funds differently, most interventions target relatively downstream activities such as service delivery and supply procurement [
4]. Some countries have also used HSS funds to improve health management information systems.
As a relatively new form of funding, less is known about the challenges of implementing system-level programs, as compared to knowledge of medical and behavioral interventions at the individual, interpersonal, or community level [
5]. To build this body of knowledge and evaluate their funds and programs, Gavi requires HSS grant evaluations [
4,
6,
7]. While findings vary by country, common barriers to HSS implementation include poor program management; lack of guidance from Gavi; weak monitoring and reporting systems; lack of clarity about the scope and goals of HSS support; and poor understanding of reprogramming, which is Gavi’s process for course-correcting programs by revising activities or funds [
7].
Gavi’s HSS support in Chad and Cameroon, two of the lowest-performing countries in terms of meeting the Millennium Development Goals by 2015, had not been evaluated prior to this study. Out of 188 countries, Chad and Cameroon ranked 175th and 176th for child mortality progress, and 181st and 178th for maternal mortality progress, respectively [
8]. Both Cameroon and Chad face significant health system limitations. For instance, the World Health Organization (WHO) recommends 23 doctors, nurses and midwives per 10,000 population, but Chad and Cameroon have only four and six per 10,000, respectively [
9]. Recognizing that such systemic conditions inhibit immunization programs, Gavi began providing both countries with HSS support nearly a decade ago. However, quantitative analyses found no evidence that Gavi HSS funding positively impacted immunization coverage in either country. In Chad, coverage changes were small and did not differ across HSS and non-HSS districts [
10]. Coverage increased slightly in Cameroon, but at the same rate in HSS-priority and other districts [
11]. These null findings raise pressing questions as to why HSS funds did not achieve the desired effects.
The growing field of implementation science offers a set of tools that should be applied to answer such questions and maximize the impact of future HSS programs, especially as Cameroon and Chad prepare for second rounds of HSS funding [
12,
13]. One of the most common implementation science frameworks is the Consolidated Framework for Implementation Research (CFIR), which was developed by Damschroder et al. based on a review of published implementation theories and empirical reports [
12]. CFIR is specifically designed for assessing complex, multi-level implementation contexts, which makes it well suited for evaluating HSS programs.
This study is the first to evaluate the Gavi HSS implementation process in either Cameroon or Chad. Identifying the drivers of and barriers to implementation will build a useful evidence base to improve the effectiveness of a large and increasingly important stream of development assistance in these priority countries and similarly challenging settings.
Methods
We triangulated quantitative and qualitative data to retrospectively evaluate the implementation of Gavi’s HSS support in Chad and Cameroon, and conducted a Root Cause Analysis (RCA) guided by the CFIR.
Financial analysis
We compared the proposed, planned, and actual budget of Gavi HSS funds. We defined proposed budget as the budget included in a country’s original approved Gavi HSS proposal. We defined planned budget as the budget for the upcoming year specified in the Annual Progress Report countries submit to Gavi each year. Each year’s planned budget can vary slightly from the original proposed budget. We defined actual budget as the country’s actual expenditure, as reported for the past year in the Annual Progress Report or by an external audit. Data analyzed included: Chad and Cameroon’s HSS proposals; Cameroon’s 2008, 2009, 2010, 2011, and 2014 Annual Progress Reports; a financial audit report for Cameroon [
14]; and Chad’s 2008, 2009, 2010, and 2013 Annual Progress Reports. Source documents are available at
www.gavi.org/country/cameroon/documents/ and
www.gavi.org/country/chad/documents/.
Document review
A comprehensive document review was performed to refine research questions, identify stakeholders for interviews, refine topic guides for key informant interviews (KIIs), and collect factual information about the HSS programs’ components and timelines. Documents analyzed included grant proposals, Gavi responses to these proposals, Gavi HSS guidelines and reports, country annual progress reports, Expanded Program on Immunization (EPI) annual plans, Millennium Development Goals progress reports, national health development plans, and past evaluations of Gavi HSS in other countries.
Field visits
Field visits were conducted to evaluate the extent to which HSS activities had been implemented. We selected the sample to capture variation by location and type of facility in Cameroon, and variation between HSS and non-HSS targeted districts in Chad.
In Cameroon, we first selected regions to represent the country’s linguistic variations (Anglophone vs. Francophone). We then selected districts within these regions that represented Cameroon’s variation in terms of demographics (rural vs. urban) and vaccination coverage (high coverage vs. low coverage). In each district, we randomly selected public Integrated Health Centers (IHC), which are targeted through HSS funds, and private IHCs, which are not. In Chad, we first selected HSS-targets districts that represented the country’s variation in terms of demographics (rural vs. urban) and vaccination coverage (high coverage vs. low coverage). We then selected non-HSS-target districts from the same regions as the selected HSS-target districts, matched based on the same demographic and vaccination coverage criteria. In each HSS-target and non-HHS-target district, we randomly selected health centers. The list of districts visited are shown in Additional file
1: Appendix Table S1.
Questionnaires were developed around checklists of activities proposed in the original and reprogrammed HSS applications. Chad’s questionnaires did not reflect the activities from the second reprogramming, which was only just being implemented during data collection. The questionnaires were tailored to different levels of the health system, including administrative and service-delivery sites at the health systems’ central, regional, district and health center levels. Trained research assistants administered the questionnaires.
In each country, a list of key informants (KI) was developed based on document review, with input from Gavi and the ministries of health (MOH). KIs included stakeholders from the central, regional, and district MOH; partners from non-governmental and bilateral organizations; and Gavi Independent Review Committee (IRC) and Secretariat staff. Topic guides were developed using the results of document review and Gavi’s evaluation domains, and included questions to understand the process, barriers, and drivers of implementing programs supported by Gavi HSS. Topic guides were customized depending on the respondent, and interviewers probed beyond the standard questions as appropriate. Verbal consent was obtained from all participants. Interviews were conducted by senior researchers and trained research assistants in person, expect for four via Skype. Interviews were audio-recorded and transcribed verbatim.
Analysis
Key informant interview transcripts were analyzed through thematic analysis, using an iterative coding process to identify important features of the data. Codes were initially based on the overarching research questions, but researchers also inductively derived new codes based on the data. We referenced the CFIR constructs to identify, revise, and structure themes and results from these codes [
12]. The CFIR is organized into five domains, each of which contains between four and eight constructs, which in turn have up to six sub-constructs each. The “Inner Setting” and “Process” domains were most relevant, as the evaluation focused on understanding and making recommendations related to institution-level barriers during the implementation process. We also identified some factors in the “Outer Setting” domain.
We triangulated data from all sources to conduct an RCA aimed at understanding key HSS implementation challenges [
15]. RCA is a structured method to retrospectively analyze adverse outcomes and identify the chain of events and underlying problems that led to the challenge observed. For this analysis, the researcher asks “why” for each identified cause, and continues to do so until the root cause is identified. As a rule of thumb, the researcher will ask “why” a total of 5 times [
16]. RCA is used in a variety of contexts, from understanding poor patient outcomes in a clinical setting to analyzing programmatic challenges for complex interventions like HSS. We used the CFIR domains, constructs, and sub-constructs to guide and structure the RCA analysis and reporting.
Discussion
This study provides the first insights on the multiplicity of factors hindering implementation of Cameroon’s and Chad’s Gavi HSS programs. Chief among these were unpredictable Gavi processes and disbursements, poor communication between the countries and Gavi, insufficient country planning without adequate technical assistance, lack of country staff and leadership, and weak systems to manage finances and promote learning during implementation. It is notable that many of the weaknesses exhibited by Chad and Cameroon were the same issues their HSS grants intended to address, indicating the need to increase technical, managerial and other forms of non-financial support during HSS implementation [
17].
In Gavi’s early years, countries perceived that the application process was too rapid and complex [
18]. While this study and prior evaluations find the HSS experience has been slower, it has become more complex, uncertain, and difficult to navigate [
7]. Gavi should revise protocols, streamline approvals, and reinforce staff to make application and disbursement processes more predictable and timely. Pooled or joint fund management may be one promising option, having improved processes in Ethiopia, Nepal, and Sudan [
19‐
21]. In tandem, Gavi should strengthen communication frameworks with countries, addressing a common barrier to scaling up programs [
22]. Better communication is needed to clarify the application process, hold countries accountable for communicating budget changes, and guide countries on how to proceed when disbursements are delayed or divided. One concrete action would be increasing in-country presence via Senior Country Managers, as recommended by the recently completed Gavi Full Country Evaluations [
23,
24]. Communication improvements should also be complemented by codified policies on how to proceed with delayed disbursements. Until Gavi’s disbursements become more predictable, Chad, Cameroon, and other countries would be wise to include contingencies in their proposals such as tying implementation start dates to disbursements, developing smaller alternative plans in the case of late or partial disbursements, and/or identifying alternate donor or government funds to cover activities during delays.
A second theme was the need for countries to improve their learning and knowledge-management practices, especially given the high rate of staff turnover. While Gavi, Chad and Cameroon showed some isolated examples of learning from past mistakes (eg: development of financial management guidelines after reprogramming), many critical problems such as unrealistic timelines and unprepared leadership were repeated multiple times. Looking beyond HSS, Chad also failed to learn from a similar experience several years earlier, when a Global Fund-supported program was suspended due to mismanaged funds [
25]. A number of existing frameworks can facilitate productive learning during implementation, many of which are based around the Plan-Do-Study-Act (PDSA) Cycle for quality improvement [
26‐
28]. These frameworks typically advocate a sequential approach that allows for testing and adapting interventions prior to scale up, rather than the single sweeping introductions that occurred in Chad and Cameroon [
26]. Some donors have begun to adopt learning frameworks [
29]; Gavi specifically has used “learning agendas” to study implementation issues for the cholera and rabies vaccines and commissioned several prospective evaluations [
30,
31]. Unfortunately, a learning mindset was not apparent during Chad and Cameroon’s HSS programs, and an intentional approach is needed to ensure the lessons from this current experience are not lost as those from the Global Fund were.
For HSS to succeed, Gavi and countries should prioritize strengthening essential functions necessary for implementation, such as human resources and managerial capacity. The lack of focus on human resources and other “upstream” efforts has been a focus of past Gavi HSS critiques [
32,
33], and this was clearly an issue in Chad, which spent none of the over $200 million intended for human resources. Human resource challenges exemplify the circular logic of HSS grants that implicitly expect countries to possess the very capacities they are aiming to improve [
34]. Human resources and managerial capacity have long been recognized as key barriers program implementation for Gavi, Cameroon and Chad [
18,
35] so it is unsurprising that managerial and health worker training were integral components of these countries’ HSS proposals. However, human resources shortages and weak management severely limit countries’ capacity to absorb funds and preclude implementation of scaled health interventions [
36,
37]. This was clearly seen in the lack of leadership exhibited in Chad and Cameroon. To overcome the paradox of expecting success from weak areas, countries need to strengthen certain key capacities – such as training staff expressly for HSS program management – prior to implementing the rest of the program. This could be achieved during an inception phase, or with sustained support from a partnering organization. Several initiatives have shown successful models for supporting countries’ planning and managerial capacity [
38], but there is a need to adapt these to the HSS context [
39‐
41]. Sequential implementation and learning approaches may also help identify and develop critical functional capacities necessary for implementation such as management abilities, information technology and communication systems. Further, improving countries’ core implementation capacities in the long-term requires policy dialogue, which the Paris declaration for aid effectiveness designates as an integral part of financial assistance for health [
42]. The limited policy-focused activities, and the lack of their implementation in Cameroon and Chad HSS, such as developing financial or managerial guidelines, have weakened the role of the local leadership. This has led to the need for a third party to manage HSS finances in Cameroon, and the decreased potential for sustainable gains and capacity to implement future programs in both countries.
Our study is subject to several limitations. First, the archiving problem made it impossible to access or even verify the existence of some important documents. Second, several KI were not available, especially those who played important roles at the beginning of HSS. Finally, recall bias was a limitation as these programs started over 8 years ago. Nevertheless, our study has two major strengths. First, it is based on a mixed methodology that triangulated data from different sources to provide more robust findings. Second, it was an independent evaluation conducted as a collaboration between interdisciplinary local and international teams to create a collectively deep body of knowledge.
Acknowledgments
The UND group includes Alexis Ngarmbatedjimal, MSc, Haroun Koumakoi, MSc, and Djimet Seli, MSc. The IFORD group includes Hénoque Blaise Nguendo Yongsi, PhD, Julien Guy Ewos Bomba, MD, MPH, Léopold Cyriaque Donfack Mbasso, MD, Mariane Kenmegni Omgba, PhD, Patrice Tanang Tchouala, MSc, and Vivien Meli Meli, PhD. We thank all key informants who participated in this study, as well Adrienne Chew, for copyediting, from the Institute for Health Metrics and Evaluation.