Using a quality improvement model to enhance providers’ performance in maternal and newborn health care: a post-only intervention and comparison design

This study employed a post-only intervention and comparison evaluation design to assess the impact of the SBM-R intervention on provider performance at public health facilities located in four major regions of Ethiopia: Tigray, Amhara, Oromia and Southern Nations, Nationalities and Peoples’ (SNNP). An SBM-R process to improve the performance and quality of MNH services was introduced at 116 government health facilities (15 hospitals and 101 health centers) from March 2011 to June 2014 with a phased-in approach. This study focuses on the 21 health facilities (15 health centers and 6 hospitals) in the first phase. All of these facilities had proceeded through the entire SBM-R process within three years of implementation and were ready for external assessment.

Guided by national MNH policies and guidelines, performance standards in 10 technical areas were developed for providing quality MNH services, including 80 standards for health centers and 79 standards for hospitals. Technical areas included: routine ANC, uncomplicated labor and delivery care and post-partum care; management of antenatal, intrapartum, and postpartum complications; infection prevention; laboratory; pharmacy; human, physical and materials resources; information, education, communication and community participation; and management system. Each standard comprised a series of specific items that could be scored as a measure of achievement. After authorities endorsed the standards, they were introduced in each health facility. Health managers and providers received three rounds of SBM-R modular trainings focusing on how to use and implement the standards, measure progress, and recognize performance.

After completing the first round of SBM-R training, each facility established a quality improvement team to implement the intervention. This team conducted a baseline assessment of services at the facility using standard tools to identify performance gaps and analyze root causes. Commonly identified performance gaps included deficits in health providers’ skills and knowledge as well as shortages of supplies and equipment. Based on the initial gap assessment, intervention facilities received essential equipment and supplies (such as newborn weighing scales, autoclaves, examination beds, delivery beds, baby towels and hats, and infection prevention supplies), training on Basic Emergency Obstetric Newborn Care (BEmONC) for providers, and regular follow-up from regional health bureaus and partners, including site mentoring, review meetings and quarterly supportive supervision. Quality improvement teams at each facility conducted assessments every six months to track performance using direct structured observations, document review and provider interviews. Based on the results, the teams developed action plans to address remaining performance gaps. All target facilities completed the four stages of SBM-R and were eligible for external assessments verifying that they had achieved at least 80% of the standards.

Sample sizes were calculated for observations of providers’ performance in each of three service areas (ANC, labor and delivery, and immediate PNC) based on 95% level of statistical confidence, 80% statistical power, 20% expected difference in health providers’ performance between intervention and comparison groups and the recommended value of 1% intraclass correlation coefficient for median value of primary health care research [11]. The intraclass correlation coefficient is the correlation between responses given by randomly selected respondents within clusters or facilities to measure degree of similarity [12].

Health providers’ mean performance scores from a 2010 quality of care study in Ethiopia [13] informed the reference values used to calculate the three sample sizes. For ANC, a mean percentage performance score of 38% for preventive treatments, including the administration of tetanus toxoid and prescription or provision of iron/folic acid was used. For uncomplicated labor and delivery care, a mean percentage performance score of 29% on active management of the third stage of labor was used. For immediate PNC, a mean percentage performance score of 47% on immediate newborn care services was used. Based on these, we calculated sample sizes of 236 provider-client interactions (observations) for ANC, 226 for uncomplicated labor and delivery care, and 232 for immediate PNC. Each sample was divided equally between intervention and comparison groups.

A total of six hospitals and 15 health centers had completed all stages of SBM-R. Half (three hospitals and eight health centers) were randomly selected for the study sample. Comparison facilities that had not implemented SBM-R (three hospitals and eight health centers) were selected from the same region and zone as the intervention facilities, but from different districts. Comparison facilities were matched with intervention facilities on staff size and client volume, including the number of first ANC visits and deliveries attended by skilled birth attendants.

The estimated number of MNH providers (those who offer ANC, labor and delivery, and/or PNC services) at each facility was 12 per hospital and five per health center, for a total of 152 across all intervention and comparison facilities in the sample. Because the number of MNH providers at each facility was small, all of them were invited to participate in the study. Women who sought ANC or labor and delivery followed by immediate PNC at these facilities during the study period were invited to participate.

Observations of service delivery using structured checklists were conducted to measure provider performance in ANC, uncomplicated labor and delivery care, and immediate PNC. The clinical observation tool listed essential maternity and newborn care tasks and was adapted from an earlier quality of care study in Ethiopia [13], national BEmONC training guidelines [10], and SBM-R tools. The ANC portion of the tool consisted of 53 tasks in eight skill areas; the labor and delivery portion consisted of 105 tasks in 10 skill areas; and the PNC portion included nine tasks in two skill areas. Observers coded each task as “Yes, task performed” or “No, task not performed.” The tool also captured basic characteristics of health providers.

Data were collected from July 7 to August 3, 2014 by 20 experienced assessors. Assessors consisted of midwives and health officers with at least a Bachelor of Science degree and national BEmONC master trainers. The assessors attended a four-day training on consent, study instruments, observation techniques, and data quality, and demonstrated their competency through role plays on consents and observation checklists before deployment. A team of two assessors conducted observations at each facility. Five supervisors and one study coordinator supervised them. To minimize observer’s bias, assessors were not assigned to health facilities where they worked.

Assessors collected information on providers’ socio-demographic characteristics and conducted observations while providers performed each step of routine ANC services, all stages of uncomplicated labor and delivery care, and/or the full range of immediate PNC services within the first six hours after delivery. Assessors stayed overnight at facilities to conduct observations, remaining up to five days at each facility until they observed at least one provider-client interaction per facility. Providers could be observed with more than one patient during this time. If no delivery took place in the facility during the assessors’ visit, the study coordinator instructed assessors to move to the next sample facility and observe more providers to compensate. Before traveling to the next sample facility, assessors and supervisors checked the completed observation tools for completeness and consistency.

Data was cleaned and entered into CSPro V 5.0 and exported to STATA V 13.1 for statistical analysis. Descriptive statistics, including frequency and percentages, were computed to identify outliers and missing values. Data were nested in three levels: clients (first level) were clustered by provider (second level), and providers were clustered by facility (third level); this nesting structure creates dependency of data. The analysis examined three outcome variables: provider performance scores on ANC, uncomplicated labor and delivery care, and immediate PNC, calculated as the percentage of essential tasks in each service area completed by providers. For this study, health providers’ performance is defined as the extent to which actual work practices reflect the skills and knowledge taught in training using standardized operating procedures [14].

Multilevel analysis was used to calculate adjusted mean percentage performance scores and standard errors to compare intervention and comparison groups. The multilevel model is appropriate to generate adjusted standard errors of the estimates and test statistics or p-values for clustered data, instead of conventional linear models that overestimate test statistics and lead to misleading inferences [15, 16]. Mean performance scores ranged from 0 to 1. The mean percentage performance score was calculated by dividing mean performance scores by the maximum mean score and multiplying by 100.

Interclass correlation coefficients (ICC) were computed to assess variations explained by facility and health provider clustering effects on three outcome variables: health provider performance on ANC, uncomplicated labor and delivery care, and immediate PNC. ICC ranges from 0 to 1 and is used to estimate the degree of providers’ performance similarity within and among clusters, where an ICC value close to 1 indicates similarities between individual’s performance within clusters and an ICC value close to 0 shows independence of individual’s performance within clusters and/or an indication to use single level analysis [17]. ICC values suggest the use of multilevel analysis even if they are close to 0.05 [18].

Study approvals were obtained from the Ethiopian Ministry of Science and Technology National Research Ethics Review Committee and the Johns Hopkins Bloomberg School of Public Health Institutional Review Board. Trained assessors informed health providers and women sought MNH services about the study and asked for their written consent to be observed. Woman was asked to allow trained assessor to observe only her labor and delivery services with the provider however, assessors did not collect data on her name, identification number and pregnancy history. For those women who arrived to the facility with labor pain, her next of kin was asked for written consent for observation. If a woman had complication, assessors did not ask written consent from a woman or her next of kin; and they waited another women who came for labor and delivery services to use for providers observations.

The sample consisted of 147 MNH providers (79 in the SBM-R intervention group and 68 in the comparison group). Most providers were female in both intervention (77.2%) and comparison (70.6%) groups, and one-third were over age 30 (Table 1). A large majority were midwives and nurses, and about half worked at hospitals and half at health centers. Almost half had fewer than five years of work experience in both intervention (48.1%) and comparison (45.6%) groups. There were no significant differences in provider or facility characteristics between intervention and comparison groups.

As shown in Table 2, assessors observed 232 ANC clients who were attended by 63 different health providers, 226 labor and delivery clients who were observed with 117 different providers and 229 PNC clients who were observed with 109 providers. ICC values vary, but are higher in the intervention than in the comparison group with one exception (PNC services at the facility level). This suggests that the quality of services offered was more uniform at intervention than comparison facilities. The greatest differences between intervention and comparison groups were seen in labor and delivery services: provider performance was more similar within each facility in intervention (ICC = 0.775) than in comparison facilities (ICC = 0.454), and performance was also more similar for multiple observations of the same provider in intervention (ICC = 0.632) than in comparison facilities (ICC = 0.474).

Strengths of the study included its reliance on direct observations of routine services by experienced assessors using comprehensive and verifiable tools to assess quality of care by comparing intervention and comparison groups. The biggest limitation is the lack of baseline information on provider performance, which makes it impossible to compare changes in provider performance over time between study groups. In future, pre-post study designs are needed to examine the full effects of SBM-R quality improvement tools. In addition, qualitative interview data that could provide a deeper understanding of providers’ performance and clients’ perspective on the quality of services are lacking. Findings cannot be generalized to all facilities across all regions, and it is possible that other quality improvement approaches fielded by international non-governmental organizations may have contaminated effects in comparison sites. Finally, the study may have introduced the Hawthorne effect, in which providers perform better than usual in response to being observed by external assessors; however, this would have affected both study groups equally.