Factors influencing performance by contracted non-state providers implementing a basic package of health services in Afghanistan

Our evaluation of Afghanistan’s CO mechanism for BPHS used a conceptual framework developed by Liu et al. as a foundation and a guide for designing the study, developing data collection tools, and analyzing data [21]. Using the Liu et al. framework provided guidance on methodology. Further, it enables comparisons of the situation in Afghanistan with that of other contracting schemes in other contexts that have also been assessed using the same framework. While the specifics of the geographical and historical situation in Afghanistan are unique, adopting a tested and proven framework contributes to the validity of the findings and makes the findings comparable with other situations.

As the Liu et al. framework suggests, this study sought to develop an overview of the contextual, institutional and contractual arrangements that have influenced NSP performance (see Fig. 1) [19, 21]. The study identifies various factors. The study also reviews program performance measures, including “contractual factors”, “contextual factors” (or the external environment) and “institutional factors” (such as hiring and retention of staff, and interactions between providers and purchaser). It sought to capture both intended and unintended effects.

As suggested by Liu et al., the research team conducting this evaluation was not directly involved in the delivery of services in the provinces where the research was carried out. The research team was external to the provinces of interest, and comprised the primary investigator (PI), three co-investigators (COI), six field investigators (FI) and one research coordinator.

In order to represent the varying contexts in Afghanistan, the research was conducted in six provinces: Balkh, Bamyan, Herat, Kabul, Kandahar, and Nangarhar. Aspects of the context included the level of security, geographical features, ethnic variations (i.e. including both Pashtun and Tajik majority provinces), the donors involved (i.e. The World Bank, USAID and EU), and implementing NSP organization.

Three main data collection methods were used: desk review, individual interviews and focus group discussions.

Our literature review explored a range of documents pertaining to the research objectives, including addressing critical issues and major policy arguments related to the role of NSPs in Afghanistan. The desk review incorporated academic papers, gray literature, reports, and official policy documents.

Liu et al. note that qualitative data provides rich insights on factors influencing program effectiveness. In line with this comment, this study involved two qualitative data collection methods: in-depth interviews with key national and provincial stakeholders and focus group discussions with local-level stakeholders. We used a purposeful sampling technique to ensure diversity among our respondents [22]. The sampling plan was stratified according to different categories of stakeholders: representatives of the MoPH at both the central and provincial levels, donors, UN agencies, NGOs, health care workers, and health professional associations. The variety allowed the team to explore perceptions and ideas from a diverse group, identifying similarities and divergences across the respondent categories.

The stakeholder and focus group interview guides were developed by the core research team (PI and COIs). The guides were translated into both the Dari and Pashto languages and then cross-translated, piloted, and corrected in order to finalize the study instrument. All interviews and discussions were conducted in either the Dari or Pashto language based on participant preference. Transcriptions were generally made on the same day (or as soon as possible) by the field investigators (FIs), who used both their field notes and recordings to ensure accurate transcription of interviews.

The FIs carried out 36 in-depth interviews and 6 FGDs across all categories of study participants. By design, we focused on health workers’ experiences with the contracting out mechanism; no patients or beneficiaries were included in the study. Table 1 lists all types of interviewees and their affiliations. The interviewees for in-depth interviews were selected using purposeful sampling that considered institutional affiliation (i.e. government or NSP), geographical distribution (representing all the provinces where the study was conducted), and function in the system (policy maker, manager or field level worker). Interviews were conducted at the respondents’ workplaces or other locations where the participants felt comfortable.

The participants for the FGDs were also selected through a purposeful sampling process that sought to keep the composition of the FGDs constant across provinces. The members of each FGD were recruited based on predefined criteria and in collaboration with local health authorities. Characteristics of FGD participants are summarized in Table 2. The FGDs were conducted in neutral settings where the participants could freely express themselves.

The FIs who collected the data were recruited and trained in March and April 2016 by the PI and COIs. The field work, led by the PI and coordinated by the research coordinator, was conducted in June and August 2016. Only the research team had access to the data collected and all interviews and FGDs were assigned codes to preserve anonymity when citing quotations.

Interview transcriptions and field and diary notes were included in the data analysis. We used content analysis to consider the key issues, elements and outcomes [23]. Topics and concepts were identified, highlighted and placed in categories of association. Themes and statements were coded according to the conceptual framework. Representative quotes were selected and allocated to the relevant classifications. Common viewpoints were described, and particularly important responses were elucidated. Finally, each category was studied and discussed by the research team to develop interpretations of the data that addressed the aims and objectives of the study.

Findings from the interviews and FGDs were triangulated with other data sources in four ways. First, the research team assessed the consistency of the findings generated using the different data collection methods. Second, we examined the consistency of different data from the same method. For instance, we compared multiple sources’ perspectives about the procurement of medical supplies, a topic we discussed with donors, MoPH policy makers and NSPs. Third, multiple analysts reviewed all findings. Fourth, we used various perspectives and theoretical frameworks when interpreting the data. In all cases we made sure that the personal opinions of the research team members were not reported as part of the results.

Combining multiple observers, theories, methods, and data sources helped to avoid problems created by collecting data using only a single method or from only one source. The breadth of perspectives included in the analysis allowed us to comprehensively assess the program and isolate the impact of CO. This was frequently difficult, given the prevalence and severity of problems such as those posed by the environment in parts of Afghanistan [21].

Contextual factors include any conditions that create either a conducive or unfavorable environment for an effective program of contracting-out. In Afghanistan, the sociocultural and geographic factors were long-standing conditions. On the other hand, the political changes that followed the fall of the Taliban created a new legal and policy foundation for contracting out. Table 4 summarizes the contextual factors that emerged from the study data.

Sociocultural norms at provincial level were identified by all categories of participants, from policy makers at the MoPH to donors to provincial- and field-level health workers, as a key factor influencing the delivery of health services [MoPH-02][DPR-01][HW-07][PPHD-07]. For example, in some provinces it is culturally unacceptable for a male health worker to examine a female patient. Coupled with a relative lack of women with higher education, leading to a shortage of local female health workers, this situation compromises health care access for women. In other provinces, different sociocultural rules about female modesty and gender apply. In these provinces, socio-cultural norms allow women to be examined by male health workers and as a result, women have better access to health care regardless of educational level. For instance, in Bamyan women actively participate in the health care system, which is functioning. Women provide some health services, and female and male health workers working together is the norm. In other parts of the country, such as Nooristan, this would not be considered culturally acceptable, requiring a different health system structure [MoPH-02].

Successful leaders are marked by their ability to maintain close relationships with local people and agencies [24]. The level of, capacity at, and structure of, the provincial health department was mentioned by participants as a key element affecting the delivery of health services in general, and contracted-out services in particular. Participants’ views were similar at the central and provincial levels. For instance, policy makers in Kabul felt that a provincial health director could facilitate better provision of health services by NSPs by making resources available and promoting the success of health service delivery efforts [MoPH-02]. The capacity and structure of the provincial health department is linked to other provincial leaders such as political figures, influential local residents and the provincial governor. However, all participants described unwanted political interference in decisions related to the delivery of health services, such as choosing where to deliver services, pushing for the hiring and firing of certain health workers or contracting with specific companies for logistical support. One respondent stated:

Sometimes the politicians interfere [with the implementation of the health services]. They recommend to the NGO an irrational establishment of a health center. [As a result] underutilized clinics are created due to political reasons. The CHC [Comprehensive Health Center] is established but population around it is not sufficient [to reach the targeted number of clients]. [DPR-01]

Political interference has proven a key challenge to contracting out programs, as NSPs have to work with local officials, warlords, members of Parliament and other influential members of the community on a regular basis. [MGR-03].

Provision of services by NSPs was also considered a challenge to government authority. Several provincial government officials interviewed reported that the CO mechanism has undermined the role of the government in service provision, consequently calling into question the legitimacy of the government. Government officials expressed concern that the population only perceives that the services are provided by NSPs and does not understand the government’s role in providing health care services [PPHD-06] [PPHD-04].

Respondents in all categories unanimously stressed that security is an essential factor in creating an enabling environment for the effective provision of health services by NSPs. Respondents from Nangarhar and Kandahar expressed the most concern about security. Insecurity is debilitating to the delivery of health care. Several interviewees described the impact of worsening security in some provinces after 2007. Increasing insecurity in these areas affected both the delivery of services and reduced the ability of the MoPH to conduct monitoring and supervision [MoPH-02]. Those NSPs with long-established relationships with local communities have generally managed to continue delivering services even in areas controlled by anti-government groups, although many incidents were mentioned when clinics had to close, or were even attacked, during a local conflict. A respondent from one of the least secure areas summarized the problem:

War, and the local situation, have huge impact on [health services]. If somewhere there is war and the situation is not normal then an NGO can’t find qualified staff and can’t provide [health services]. [PPHD-06]

Geographical features also have a significant impact on the reach and effectiveness of health services. Each province of Afghanistan has distinct geographical characteristics that affect the distribution of health facilities and provision of health services. It is particularly difficult to guarantee regular supplies of medicines and medical equipment in hard-to-reach areas in mountainous regions. The difficult geography is compounded by challenges created by the climate. NSPs have to plan ahead to maintain services during often long periods of road closures in winter.

Many NSPs, therefore, prefer to provide services in regions that are easier to access. NSPs with contracts to deliver health services in regions with harsh geography need to develop innovative strategies, in particular to incentivize recruitment and retention of health professionals willing to work in difficult conditions. Several participants from NSPs mentioned instances when they had to offer more benefits to get staff to accept positions in hard-to-reach areas. This was particularly the case for female doctors, whose packages might include also hiring the doctor’s husband, providing hardship payments and offering special vacation opportunities [PMGR-03]. One participant described that:

NGOs’ salary rates are according to geographical grading. It’s different in different provinces. The hard-to-reach areas and conflict-affected areas have more salary. [HW-303]

Contextual factors lay the foundation on which institutional responses are built and in terms of which the contractual factors are defined.

The category of contractual factors includes various aspects of the contracting mechanism: types of services covered, contract formality, contract duration, contractor selection, specifications of requirements, processes of contract implementation, output and outcome indicators and finally the contract payment mechanism.

Respondents were generally able to describe the main types of services contracted out by the government to NSPs (including BPHS and the Essential Package of Hospital Services (EPHS)) and those provided directly such as provincial hospital services. One focus group agreed:

[The] Basic Package of Health Services provided in [our] province includes all the BPHS components, such as maternal and neonatal health services, immunization and child health services, public nutrition, control of communicable diseases services, mental health and disability health services, and pharmaceutical services. There is also EPHS, which provides secondary health services through the regional hospital in [a neighboring] province. The EPHS is a contracting-in mechanism in [our] province. [FGD-02]

Some respondents also mentioned contracting of capacity building programs and research projects.

MoPH policy makers expressed generally favorable perspectives on the contracts in the CO mechanism. Several respondents mentioned that the selection process established for CO and the creation of the Grants and Contracts Management Unit (GCMU) have become examples that other national sectors seek to follow [MoPH-02] [PPHD-02]. From the outset, the BPHS program has emphasized formality in its contracts. They require NSPs to abide by all governmental laws (after undergoing a rigorous selection process). These strictures enabled the NSPs and the government alike to trust each other, and fostered reliability of the services.

Both NSP managers and MoPH officials interviewed noted that contract durations differed by donor, and that contracts were commonly extended beyond the original contractual agreement. While the initial contract durations ranged from 18 to 36 months (with an average of 26 months), extensions lengthened them. One MoPH official explained:

[The durations] are different, normally between two and three years. But these [contracts] were extended. Even if it is for three years, it is subject to the evaluation by the [third party] organizations. Performance review is a condition for the extension. There were extensions up to five years. The PCH [partnership contracts for health], for instance, started in 2009 and in the second year it was evaluated and extended to the third, and finally it was extended to five years. The small project [non-BPHS] did not last more than six or seven months. [MOPH-02]

Respondents had mixed reactions towards the extensions. Some argued that the extensions of contracts had a positive effect on service delivery by preventing disruptions that would occur with another long tendering process. This view was supported by NSPs, who stressed that the longer an NSP worked in a given location, the stronger their relationship with the community [PMGR-02]. However, others presented a different viewpoint. This view was widely expressed by provincial MoPH authorities, who reported that following an extension the NSPs tended to relax, undertaking fewer quality improvement efforts or innovations [PPHD-05]. Another concern raised about contract extensions was that they reduce competition, undermining its benefits.

Funding for the contracts comes from multiple donors with the MoPH now serving as the purchaser; in the earlier stages of the CO program, donors interacted directly with NSPs believing that the government lacked the requisite capacity for financial management and procurement. Indeed, some donors temporarily used their own mechanisms for procurement of NSPs until the government’s procurement capacity was ready to manage a large program like the BPHS [PPHD-01]. Once the MoPH had developed the capacity to handle procurement for large scale programs, a unified system was developed with the leadership of the MoPH. This transition occurred gradually, beginning with the 2003 transfer to GCMU of contract management for all the World Bank funded provinces. In 2010 USAID delegated its contract management to the MoPH, as did the EU in 2013 [5].

The study reviewed the contract specifications from the MoPH. According to these documents, the process for contracting out to NSPs is well-designed and clear. The process is governed by the MoPH with active participation from relevant stakeholders, including provincial health directors. A selection committee (GCMU officers, provincial health director, UN agency representative and MoF representative) reviews and awards contracts, while the administrative aspects are managed by GCMU.

While the process seemed clear on paper, interviewees expressed concerns. Some respondents suggested that there was little real competition. Some felt that the participation of provincial health directors was merely symbolic; moreover, MoPH officials at both central and provincial levels expressed concern that a small number of Provincial Public Health Directors (PPHDs) were unable to be impartial.

BPHS and EPHS documents specify the services to be provided by NSPs. They detail the requirements for all processes, inputs and monitoring, as well as targets for outputs and outcomes. Among our respondents, MoPH managers, donors and central NSP managers had more precise knowledge of these details than health workers and provincial managers.

The process for implementing health services is specified by the BPHS implementation guidelines. Each contract includes a log frame and approved and agreed-on indicators that help guide implementation and monitoring and evaluation of the performance of NSPs. Thus there is a common understanding between the government purchaser (MoPH) and the NSP contractors about what types of services are to be provided and how they should be implemented [MoPH-03].

In this study, all groups of respondents demonstrated high levels of awareness of performance specifications and most discussed performance indicators. The responses from NSP employees in particular showed that these indicators play a meaningful role in ensuring that services are delivered per the plans and expectations of the contracts [PMGR-01],[PMGR-02],[PMGR-03],[PMGR-04],[PMGR-05],[PMGR-06].

Each contract includes specific and clear target output and outcome indicators. These contribute to transparency and clarity on how to measure activities and facilitate quantification of the services provided by NSPs. Output indicators may include number of health workers trained, number of health education sessions conducted or number of institutional deliveries. Output targets are based on the population of a basic health center (BHC) or clinic catchment area. Provincial targets are set using provincial population data. Outcome indicators are captured and measured separately by third party evaluators using the Balanced Score Card (BSC). The BSC has six domains [25]. (Additional file 1 provides information on BSC performance over time for each of the six provinces examined in this study).

Outputs are the primary focus for USAID-funded contracts, which reimburse NSPs for services delivered. This payment system facilitates evaluation, as data are reported. The World Bank -contracts and the current System Enhancement for Health Actions in Transition (SEHAT) program, on the other hand, are based on lump-sum contracts and emphasize outcome indicators. One Ministry official explained:

The three donors have had different performance indicators. For USAID, input process and output and outcome indicators were used. We had a datasheet that contained both output and outcome indicators. The World Bank had more focus on outcome indicators and did not emphasize process or inputs. EU was in between, with a tendency towards outcomes. [MOPH-02]

As noted, two mechanisms have been used to pay contracted NSPs: lump-sum payment and cost-reimbursable payment. The World Bank funded projects were contracts with a lump-sum payment mechanism, as one respondent described:

The contract was lump-sum, with some flexibility in movement across the budget lines. The staff is provided with salary and money for some other items, such as running cost, maintenance and emergency medicine. [PMGR-04]

The cost-reimbursable payment mechanism, on the other hand, is the main model under USAID. In USAID-supported provinces, payments were made based on reported outputs.

The EU contracts fell between the two distinct models. They were cost-reimbursable, but with a greater focus on performance outcomes rather than inputs and outputs.

NSPs managers we interviewed expressed preferences for the lump-sum mechanism, which they see as offering more flexibility and less rigorous reporting and monitoring [PMGR-02, FGD-01]. However, this mechanism risks making evaluation using reported data more difficult. Respondents from the government, therefore, generally preferred a reimbursable mechanism, which entails more scrutiny and closer supervision of the NSPs [MOPH-02, PPHD-05].

The choice of payment mechanism can affect performance. With lump-sum payments, NSPs have more freedom in terms of their implementation processes. They have latitude to initiate innovative approaches to attain the contractually agreed upon outcomes. However, it also creates more opportunities to diverge from the contract.

With the launch of the SEHAT program (2013), however, all payment mechanisms are lump-sum. However, “lump-sum” may mean different things to different partners. One respondent highlighted this conundrum:

Everyone talks about lump-sum mechanism but still there is not enough clarity about it. NGOs have their own definition where they want more freedom and flexibility, while MoPH has its own definition trying to make NGOs more accountable. Both parties should come together and decide what they mean. [MoPH-02]

Frontline health workers understood “payment mechanism” in reference to their salaries, regardless of the contract model used to support the payroll. One provincial worker described:

The payment mechanism for the employees is working in such a way that first the reports from the health facilities are collected by the NGO. Then, the reports are analyzed and the financial report is prepared and finally, the employee payment is deposited into their bank accounts on a monthly basis. In the past, this payment mechanism was different. The staff payments were processed in the form of a cash transfer.

The payment systems for employees have evolved. In the first few years, NSPs determined salaries based on their budgets. In 2005, a national salary scale was established by the MoPH that standardized payments across the provinces and organizations. Most health workers interviewed thought that a Results-Based Financing (RBF) approach would be more appealing than a fixed salary, because they would get both a basic standard salary and extra payment based on performance [HW-05].

The contractual factors establish parameters for how contractors respond to contextual factors, and set limits within which the institutional factors operate.

We classified institutional factors in two categories: internal responses (created by either the purchaser or the contractor) and external responses [21]. Internal responses are further divided into three sub-categories: 1) managing inputs, 2) managing outputs and outcomes, 3) performance monitoring. External response sub-categories are: 1) provider market and 2) public service.

Our interviews found that most stakeholders have a positive impression of performance monitoring for contracting-out health services. A national HMIS system and third-party evaluations are included in the contracts to track input, output and outcome measures, as well as to assess overall impact.

The HMIS is based on a set of indicators gathered at the health facility level by frontline health workers, such as the number of deliveries that occurred in the health centers or were assisted by skilled birth attendants and the number of children vaccinated through routine immunizations. However, since the HMIS data are based on self-reports from providers, their quality and accuracy were called into question by some respondents. The new system for HMIS data verification, which involves a third party, received positive feedback from some respondents, who indicated that it is helping to improve the reliability of HMIS data [DPR-02, FGD-01].

A second concern with HMIS data is its usefulness for decision making. Some respondents mentioned that HMIS data are indeed informing decision making at different levels, from the individual health facility to the ministerial level. One policy area in which HMIS data is considered to be highly valuable is in the rationalization of distribution of health facilities. HMIS data provide information to help assess whether, considering both the investment costs and the needs of communities, proposed locations or functionality levels of new health centers are rational.

Respondents reported that NSPs have also created systems to utilize collected data in improving the delivery of health services at different levels. Data collected from clinics are analyzed and presented back to health facility managers on monthly and quarterly basis. Any indicators that have not been achieved are highlighted and corrective measures discussed. For instance, if the number of deliveries in a facility is low, the NSP conducts a follow-up assessment to understand why. This informs decision making on how to address problems so corrective measures can be integrated in the plans for the next cycle.

In summary, the MoPH in collaboration with donors and its development partners has established a comprehensive, intensive and responsive HMIS to measure and provide timely feedback on the contracted NSPs’ performance on indicators. Some concerns remain about the quality of the data and the efficiency of monitoring and evaluation (M&E) processes. However, on the whole the system covers all aspects of the project and is well integrated, thus constituting the backbone of CO for health services.

Liu et al. note that systematically understanding the interaction of factors requires comparators [8]; this was beyond the scope of this individual country level analysis. Other limitations related to three aspects of the research process. The study design focused on collecting and analyzing qualitative data to generate an in-depth picture of the contracted health care delivery system in Afghanistan. However, the findings could also have been triangulated with quantitative data, in particular to understand the CO program’s outcomes.

Execution was limited by insecurity, the geographic size of the catchment areas and difficulties posed by transportation. Further, given time and resource limitations, the qualitative research design used purposive sampling of provinces and participants in order to capture a breadth of experiences in terms of payment mechanisms, contracting processes and KIs’ roles. However, we cannot make claims about how common or widespread any of the perspectives were. During data collection, we faced particular challenges when interviewing PPHDs. In some cases, they lacked institutional memory about contracting out, while others were not reachable. In an exceptional case, one director of health was interviewed while hospitalized and recovering from a roadside explosion.

Finally, our main objective in this study was to present a description of the factors influencing a specific intervention. However, analyzing interactions among the factors proved beyond of the scope of this study. Future studies are recommended to delve further into this.

Our relatively narrow case study on the BPHS allowed us an in-depth view of the factors that affect NSPs’ performance. We omitted discussion of the contracting-out of EPHS or other programmatic, training and research services. We sought to highlight this gap by mentioning them in the background section, and recognize that they present areas for additional research.