Methods and design
Overarching study design and methods
This study will rely on mixed-methods, applying both quantitative and qualitative methods of data collection and analysis (detailed in Table
3). The study will be partially prospective, collecting and analyzing primary and secondary data during the year of study implementation, and partially retrospective, relying on secondary data existing at the time leading up to and including the study launch. For a synopsis of methods and research questions employed see Table
3.
Table 3
Research questions, methodological approach, data collection activity
1. Service Provision. How has the intervention affected quality of service provided and why? | Quant & Qual | a. Structured Checklist; Data extraction lists for (i) routine surveillance databases (HMIS) and (ii) Service Provision Assessment (at baseline) | a. IDIa& FGDa |
b. Health facility (intervention and control) | b. Clients, Community members |
c. Primary and Secondary data | c. Primary data |
2. Provider Motivation. How has the intervention affected health worker motivation? | Quant & Qual | a. Health worker survey | a. IDI |
b. Health workers in intervention and control | b. Health workers in intervention facilities |
c. Primary and secondary data | c. Primary data |
3. Fidelity of Implementation. How has the intervention aligned with intended design, and what factors have affected this? | Qual | -- | a. IDI & Document Review |
b. IDIs with MoH, Funders, PBI desk officers, DHMTs, SSDI employees, health workers in Intervention Facilities; Document Review of implementation planning and monitoring material |
c. Primary and Secondary data |
4. Costing. What are the costs of implementing the intervention in relation to outcomes produced? | Quant | a. Data Extraction | -- |
b. Implementer materials (SSDI costing data) |
c. Secondary data |
Changes in facility routines and Provider’s lives
PBI’s impact on service provision and health worker behavior
Drawing on a mixed-methods approach, this component will focus on changes produced by the SSDI-PBI intervention on quantity and quality of the health services provided in relation to measured changes in the work environment. Following our theoretical model, changes in physical and psychosocial work environment are expected to positively affect EHP service delivery. Specifically, we expect to observe changes in the quality of care of targeted services.
Quality of care measures will include measures of service input, process, and output elements, such as the availability of functional equipment, stock-outs of essential supplies and medicines, procurement and maintenance procedures at both the individual service and the facility level, human resource availability, client satisfaction with essential service components, and service coverage.
Given our explicit ambition to work with a design closer to implementation by building on health information that is collected on a routine basis, most of the data used in this study component will stem from secondary sources (health management information system (HMIS), service provision assessment (SPA), SSDI baseline data, as well as SSDI monitoring and evaluation data). Additional information collected through direct facility inventories and health worker surveys will be used to further enrich the data sources for this component.
This portion of the study will use mixed methods, collecting the above quantitative but also qualitative data in parallel with an aim to explore different facets of the same research question. The quantitative component will adopt a controlled time-series design, including controls that are matched to be comparable in terms of facility type, zone (or district, if feasible), distance to a main road, and PQI intervention status. The inclusion of controls is meant to minimize potential time-dependent biases. For all selected indicators, we will therefore collect information going as far back as 12–24 months prior to and 12–15 months following the PBI program’s start. The qualitative portion of this component will be used to examine clients’ perception of service quality over the course of and in response to the PBI intervention. Qualitative methods of data collection will include IDIs with clients (i.e. service users) and FGDs with community members (including those who can convey community-held knowledge, attitudes and practices in relation to the health facility).
PBI’s effect on health workers’ motivation
Drawing on mixed methods, this component will focus on health workers’ perceptions, satisfaction and motivation in relation to the implementation of the SSDI-PBI intervention. Specifically, we will assess whether and how changes in the working environment are perceived by health workers, and how these changes might have resulted in changes in motivation to provide high quality and quantity care. The qualitative approach will entail a series of IDIs with selected health workers from selected intervention health facilities, taking a retrospective approach of asking them to recall their perceptions of changes that have taken place in the implementation period, relating them to specific aspects of the intervention, and explaining pathways of change. We will conduct interviews only in intervention facilities with an aim of interviewing at least two providers per facility. We will also collect quantitative survey data that will assess perceptions of change in the working environment in both control and intervention facilities.
The process evaluation component
Drawing on qualitative methods, the process evaluation will focus on fidelity of implementation (FOI), and consider adherence (defined in terms of content, schedule, and coverage) to the original intervention model and the contextual factors that mediated and affected adherence [
34,
35]. This analysis will explore to what extent
in itinere modifications - inevitable when concerned stakeholders are responding to and implementing a given intervention - hinder or enhance the effectiveness of the intervention itself. The process evaluation component will allow us: to delineate to what extent the SSDI-PBI intervention was delivered as initially planned and to explore heterogeneity in implementation processes across districts and facilities; to understand how the various stakeholders responded to the intervention and acted to modify, for better or for worse, its content; to identify contextual elements that affected the implementation of the SSDI-PBI intervention and to what extent these elements can explain heterogeneity in outcomes across districts and facilities.
Within this component we will pay particular attention to the implementation processes related to: the verification and counter-verification system, to understand its suitability and effectiveness within the framework of the SSDI-PBI intervention; the facility management structures and the interaction with the SSDI team in charge of the facility business plans, to understand the process of receiving and re-investing the performance rewards and the potential for “gaming” induced by these structures; and the role that district management teams play in relation to supervision, human resource allocation, and distribution of equipment and commodities, to identify potential changes in behavior in favor of incentivized facilities and services.
This portion of the study will rely on qualitative methods, conducted using a prospective approach. In line with standard practice in process evaluation [
36] and with our own work in similar settings [
37,
38], we will begin our work by convening a workshop with the SSDI-PBI core design and implementation team to develop a shared Theory of Intervention (TOI) and to identify the core 20–25 activities which fit this TOI, and are key to the intervention success from a theoretical point of view. Qualitative data will be collected to examine contextual factors that mediated the implementation and as such affected fidelity, and to explore how the various stakeholders responded to the intervention through adaptation and adoption. By “stakeholders” we are referring to anyone involved in the design and implementation of the intervention (i.e. the funding agency and its implementing partners, Ministry of Health (MoH) directorates, district management teams, PBI desk officers, as well as at least one health worker from each concerned facility). Sampling will be done via the snowball method wherein the study team will first be introduced by the funding agency and key programmatic personnel to those engaged in the SSDI-PBI program. This will assist the research team in identifying initial respondents. These respondents will be asked to assist in the identification of other respondents, who could facilitate an understanding of the themes and issues raised in the interview or related to the program generally.
The costing component
The main objective of this component is to evaluate costs of the SSDI-PBI scheme to provide insights on the value for money associated with the intervention. The costs associated with PBI include the following cost categories: a.) start-up costs (design of program, training, initial dissemination); b.) ongoing management (including verification and counter-verification); c.) the cost of incentives themselves. These costs are incurred at the level of the Ministry of Health, its development partners (USAID), and the implementing agency (SSDI). The cost analysis will allow estimating the incidence of different cost categories, accruing the total costs of PBI. More specifically, we will be able to compare the total costs of start-up, implementation and management of the PBI scheme to the costs of financial incentives. For the majority of cost components, costs will be evaluated using the micro-costing approach, which requires that for each cost category, quantity of resources is identified and then multiplied by its unit costs. To do this we will rely heavily on support from the implementation team, asking to access their financial and cost management data. We will also request lists of participants who were engaged in key implementation activities (and approximate salary ranges across cadres of employment) in order to account for an allocation of staff time toward PBI activities in the event that time investment was not paid for.
Overarching analysis & data protection
The ultimate aim of this research is to assess the effect that the SSDI-PBI intervention produces on the quantity and quality of care in light of the intervention implementation processes and at what cost. While the work is divided into components to ease data collection and analysis, the ultimate aim of this study is to triangulate and integrate information across components and data sources in order to address the overall study objective, and to provide a more nuanced interpretive analysis. Lead researchers across study components will convene to discuss and share findings upon completion of analysis of a given component.
Quantitative data analysis
Quantitative data will be analyzed using Stata. Our analytical approach will largely rely on a controlled time-series analysis, building on data from 17 intervention and 17 control facilities. As such, the actual sample size will be 34. For each of the quantitative outcome indicators, we will include monthly data from the HMIS for the period Sept 2013 to Dec 2015 (or 12–24 months prior to and 12–15 months following PBI start). This will generate a total of 28 observation points for each of the 34 facilities included in the study. For each service provision indicator, we will compare developments over time across intervention and control facilities. Given that we could not assess actual data availability and completeness prior to data collection, we will do ex-post power calculations. We will assess the impact of the intervention on targeted service provision indicators using an interrupted time-series model with independent controls. In addition, given the availability of two single time points, we will rely on difference in differences modeling to assess changes in quality of care, using for example changes in staff numbers and qualification, availability of equipment and supply as proxies. This analytical component will draw upon primary data that will be collected in intervention and control facilities, as well as SPA data collected in January 2014, which will serve as a pseudo-baseline.
Qualitative data analysis
All qualitative interviews will be tape-recorded, transcribed and translated into English. Qualitative data will be analyzed using a hybrid approach. We will first identify themes from a selection of rich and nuanced transcripts – an inductive approach [
39]. We will later apply this template of codes to the remaining transcripts through a deductive approach [
40]. We will adopt in-vivo coding (using NVivo 10), with codes, categories, and themes emerging as we proceed through the data, although the initial coding process will be guided by the research questions. We will apply data triangulation as we will compare information across data sources to capture multiple perspectives of the same research question, and analyst triangulation as at least two researchers will code and interpret each set of data.
Discussion
PBI programs are complex and questions remain in terms of the potential for such programs to improve the quality of care and spark reductions in morbidity and mortality. The aim of SSDI-PBI in Malawi is to increase service utilization and stimulate better quality of care by linking payments to strategies aimed at improving providers’ working environments and in doing so, empowering providers with the means to provide care of an adequate standard.
The overarching purpose of this study is to contribute robust evidence to a growing body of literature examining the process, impact and cost-effectiveness of PBI programs in SSA. We also aim to generate knowledge regarding how programs such as SSDI-PBI are woven into the health system, how they are perceived by patients and providers alike, and to what extent they are cost-effective. We view this research as particularly useful because it examines a non-conventional PBI model, which does not entail monetary incentives or financial autonomy at the facility level. Such a model has not been studied in the literature.
This study design was conceptualized with a relatively small budget and a relatively short study period (approximately one year) to rapidly meet the need of informing the implementation of the intervention. Given the broad scope of the PBI intervention in respect to the range of services being incentivized, time and funding constraints do not allow for a detailed assessment of cost, nor is it feasible to undertake time or labor intensive endeavors such as community-based household surveys to assess health-seeking behavior, or direct observations to assess quality of care processes. In respect to producing robust effect measures, the main limitation of this study design is that it had to be adapted to a non-randomized intervention. Given that study districts and intervention facilities were chosen by the MoH based on political and logistical considerations rather than a random selection of health facilities and districts, selection bias was introduced. Furthermore, our study sample is limited to the relatively small scope of the intervention (17 health facilities), potentially resulting in low statistical power. As a consequence, we might not be able to demonstrate statistical significance of observed impacts. Finally, the fact that the start of the PBI program is not aligned with the external evaluation research further restricted the design, as baseline information on the performance of essential health service provision could not be based on primary data collection on sets of indicators directly linked to specific outcome measures. As a result, different quasi- and non-experimental study approaches and a heavy reliance on secondary data sources was required to address potential threats to validity.
In terms of contending with the aforementioned limitations, our evaluation team seeks to be nimble and resourceful. We are drawing from multiple methods within both quantitative and qualitative approaches. Quantitatively, we collect primary data related to infrastructure and equipment, staff numbers and trainings, as well as health worker perceptions. Qualitatively, we directly collect IDI and FGD data across a broad range of respondents at facility, community and policymaker levels thereby inhibiting the domination of one single respondent type. We are also undertaking triangulation to identify points of convergence and divergence in the data. This study draws upon a substantial amount of secondary data. The utilization of secondary data, while imperfect, can present a means to capitalize on existing data thereby reducing costs while contending with time constraints. Furthermore, secondary data presents a means to overcome the absence of an evaluator-initiated baseline survey, to capture trends over time (in the case of HMIS data, specifically), and to fulfill a tenet of meaningful implementation research [
41]. While having a short turnaround period poses limitations in terms of research, we also view it as an opportunity to generate knowledge and to inform implementation partners in a timely fashion—even as decisions are being made regarding how to expand an intervention.
This study also seeks to draw upon a theory as a means to inform the nature of the research and to guide the research team in terms of sampling and data analysis. We view the development of theory as an essential though under-emphasized concept within PBI literature. This evaluation places an emphasis on the health facility (inclusive of the staff, infrastructure and equipment therein), which the study team and implementing team have determined represents a meaningful starting point for an examination of PBI (given the manner in which the program compels tremendous shifts in terms of how, when, how often and at what cost providers are capable and expected to provide care). Furthermore, the actions of providers and the nature and quality of their engagement with patients are critical factors in determining whether and to what degree morbidity and mortality are ultimately reduced. While our theory emphasizes the effect of PBI within the health facility, we see several points of entry to inform future consideration of the theoretical underpinnings of PBI. For example, PBI lays the foundation for national level policies and priorities that ultimately redirect practices at zonal, regional, district and community levels. On an interpersonal level, PBI affects how patients and providers interact with and perceive one another (and possibly how the parties perceive the health system more generally). Given the web of entry points wherein one could consider theoretical underpinnings of PBI, it is challenging to focus on a particular sphere or set of indicators, but we view an expansion of theory-driven research as necessary in guiding future research in this field.
Our study design contributes to the evaluation literature by providing an example of how sound implementation research can be done despite time and budget constraints as well as constraints due to the size of the intervention. We view this contribution as crucial particularly as PBI programs continue to expand and ministries of health and finance -- as well as funders and program implementers -- are faced with decisions regarding whether or how programs should be adapted or expanded and at what cost. We view a more well-rounded examination as a powerful tool in not only guiding programmatic and policymaker decisions, but also as a means to guide fellow researchers as they consider how to evaluate health programs in a comprehensive, affordable, meaningful and timely manner.
Abbreviations
CoM, college of medicine (of Malawi); EHP, essential health package; FGD, focus group discussion; FOI, fidelity of implementation; HMIS, health management information system; IDI, in-depth interview; MDG, millennium development goal; MoH, ministry of health; PBI, performance-based incentives; SPA, service provision assessment; SSA, sub-saharan africa; SSDI, support for service delivery integration; TOI, theory of intervention; USAID, United States Agency for International Development
Acknowledgements
The authors would like to thank Sophie Witter for her helpful insights in relation to our theoretical model. We also thank Supriya Madhavan for her insights on the protocol. We are grateful for financial support from the Deutsche Forschungsgemeinschaft and Ruprecht-Karls-Universität Heidelberg within the funding program Open Access Publishing.