Measuring the quality of maternal and care processes at the time of delivery in sub-Saharan Africa: development and validation of a short index

The current study used the comprehensive facility-based QoPIIPC index developed through earlier analysis as a reference point. The process of developing and validating the comprehensive index is briefly summarized here and has been reported in detail previously [39]. The comprehensive measure was developed following a modified Delphi process with maternal and neonatal care (MNC) experts to identify consensus dimensions of QoPIIPC. MNC experts also rated the ability of items, i.e., actions during intrapartum and postpartum care, to reflect these dimensions. The five consensus QoPIIPC dimensions identified by the expert group were technical quality, screening and monitoring quality, interpersonal care quality, the quality of infection prevention/control, and the avoidance of harmful or non-indicated interventions [39]. Indices containing combinations of highly-rated items were developed based on MNC expert ratings and evaluated for face, content, and criterion validity. Secondary data obtained from surveys observing L&D care at health facilities in sub-Saharan Africa were used in index validation. The comprehensive QoPIIPC index of 20 items was selected based on comparison of performance on several validation benchmarks [39]. The secondary data source and validation benchmarks are described further below.

For the analysis reported in this paper, the 20 items in the comprehensive QoPIIPC index were evaluated for whether they could be assessed at or immediately following delivery, thus avoiding observation of client intake and the unpredictably long first stage of active labor and early second stage of labor. Items meeting these criteria were retained in a “delivery-only” index.

The Maternal and Child Health Integrated Program (MCHIP), a USAID-funded global project implemented by Jhpiego, conducted the QoC Assessments, a set of observational surveys in sub-Saharan Africa from 2010 to 2013. QoC Assessment data were used to evaluate the delivery-only index were obtained from a series of observational surveys of QoC in sub-Saharan Africa between 2010 and 2013. Specifically, the study used data from QoC Assessments conducted in 2010–2011 in Kenya, Madagascar, and Tanzania, including Zanzibar; as well as a repeat survey in Tanzania alone in 2012–2013. The countries were selected due to similarity in their maternal health services and indicators [40‐42].

As described in reporting the earlier study to develop a comprehensive QoPIIPC index [39], a structured checklist was used for delivery observations in the QoC Assessments, based on World Health Organization recommendations and other global guidelines and surveys [7, 15, 20, 43, 44]. The checklist included items about essential L&D care as well as care for maternal and newborn complications [45]. There were 131 routine care L&D items in the L&D observation checklist [39, 45].

The QoC Assessment sample sizes, at least 250 deliveries in each country, were intended to provide national estimates of routine L&D care practices. Details of sampling approaches and data collection tools are provided in each country’s survey report [45]. Analytic samples in this study were restricted to L&D cases observed across intake, active labor, delivery, and the immediate postpartum period. The Zanzibar and Round 1 Tanzania samples were merged for analysis, as the number of deliveries observed in Zanzibar was small. Data were not weighted for analysis.

The delivery-only index was evaluated within each country and across countries; it was compared to the comprehensive QoPIIPC index using QoC Assessment delivery observation data. As in the prior study, each observed delivery was assigned a comprehensive index score and a delivery-only index score. Each index item had a value of 1 if performed and 0 if not performed. These item scores were summed to create comprehensive and delivery-only index scores for each delivery. A total QoC score was also given to each delivery based whether each routine intrapartum and immediate postpartum care item in the full L&D observation checklist was performed.

The delivery-only index was assessed across six validation domains, each with multiple benchmarks. The domains were: representation of QoPIIPC dimensions; association of the index score with overall QoC performance; relation of each item in the index to overall QoC performance; ability to discriminate between poorly and well-performed deliveries; inclusion of items that ranged in frequency of performance; and variability and distribution of the index score. These validation domains evaluate the degree to which an index measures and is informative about QoPIIPC. Benchmarks are specific, quantifiable, and comparable criteria within each validation domain. A total of 28 benchmarks were assessed across the six validation domains. Validation domains, benchmarks, and selection criteria are identical to those used to validate the comprehensive QoPIIPC index in the earlier study, and have been described previously [39]. A threshold of p < 0.05 was used in tests of statistical significance.

A particular focus of assessment was content and criterion validity. Content validity describes how well the index represents QoPIIPC, specifically the consensus dimensions identified through the Delphi process described above. Criterion validity is reflected by the relation of the index score to a reference measure of QoPIIPC. In this analysis, the total QoC score across all routine care items served as the reference measure of overall QoC performance.

To be useful, a quality measure must be able to discriminate between poorly and well-performed deliveries. Therefore, this domain accounted for a substantial proportion (15 of 28) of the validation benchmarks. To enable assessment of QoC discrimination, level of care quality was described with three dichotomous variables. First, relatively good performance was defined as being in the top 25% of the total QoC score distribution. Second, absolute good performance was defined as achieving at least 80% of the maximum possible total QoC score. Finally, relatively poor performance was defined as being in the bottom 25% of the total QoC score distribution. The three dichotomous variables were treated as the dependent variables in separate analyses.

Simple logistical regressions assessed the relation between index scores and the odds of being in each good/poor performance group. The area under receiver operating characteristic (AUROC) curves based on the logistic regression results was calculated for each good/poor performance classification. AUROCs indicate the ability of the index to correctly classify QoC. If two deliveries are drawn from the sample at random, the AUROC represents the proportion of pairs in which the delivery with the higher index score is in the good performance group and vice versa, for classification of poor performance. An AUROC of 0.7–0.9 shows moderate discrimination while over 0.9 is considered excellent discrimination [46, 47]. Predicted probabilities were calculated based on logistic regressions, representing the likelihood of being in the relative and absolute good performance groups at each value of the index score.

Analyses also compared the performance of the delivery-only and comprehensive QoPIIPC indices. AUROC comparisons assessed the relative ability of each index to classify deliveries as good or poor performance. Likelihood ratio tests compared the fit of linear and logistic regression models of the association between index scores and overall QoC performance. Likelihood ratio test assessment was possible because the delivery-only index items were a subset of the comprehensive index items. Comparisons used standardized index scores to avoid differences due to the number of items included in the two indices.

To enable comparison between the delivery-only and comprehensive indices, performance on each validation benchmark was given a score for each index; the index that performed better on each benchmark received 1 point and the other, 0 points. The scores were summed for each domain. The index with a higher score within each validation domain received 1 point and the other, 0 points. Finally, validation performance scores summing across domains (potential range from 0 to 6, with 1 point for each domain) were calculated for each index within each country and across countries.

Because the comprehensive QoPIIPC index was developed through an extensive expert review and validation process, the aim of this analysis was not to determine whether the delivery-only index is a “better” measure of quality. Instead, comparative evaluation of validation performance sought to examine whether the delivery-only index may be a robust alternative in settings of limited resources for quality assessment and observation of care.

The QoC Assessment protocol was reviewed and approved by ethical review boards in each country where the survey was conducted. In the countries whose data are analyzed in this study, these boards were: the Kenya Medical Research Institute Institutional Review Board (IRB) in Kenya; the Ministry of Health Ethical Committee in Madagascar; and the National Institute of Medical Research Institutional Review Board IRB in Tanzania. The Johns Hopkins Bloomberg School of Public Health IRB ruled the protocol for the QoC Assessment study across all countries exempt from review (reference number 00002549).

Written informed consent was obtained from facility directors prior to the QoC Assessment implementation. During data collection, verbal informed consent was obtained from providers and patients or patients’ next of kin. Providers were not asked to give written consent during the provision of L&D care; however, a comprehensive discussion of benefits and burdens was held with the facility directors in a non-service provision context. Patients or next of kin were not asked to provide written consent both because of literacy limitations and to reduce the burden on women during L&D. Verbal consent was recorded in the QoC Assessment data entry applications; each module of questions noted that provider and patient (or next of kin) consent was required before items in that module could be completed. Consent procedures were described in research plans submitted to and approved by the aforementioned IRBs. The names of individual patients and providers were not collected during service observations. The quantitative analyses reported in this study were conducted using secondary data without identifiers.

This study faced limitations related to the QoC Assessment data analyzed, such as a potential Hawthorne effect, lack of generalizability to facilities with a lower-volume of deliveries, and restriction of data to routine L&D/ENC interventions. These limitations have been reported in depth elsewhere [39]. However, considering the resources and effort required to observe L&D care even with the most efficient tools, it may be appropriate for the use of the delivery-only index to be restricted to higher-volume facilities. Additionally, the QoC Assessments were conducted across a diverse sample of facilities, from rural health centers to referral hospitals, possibly reducing the effect of non-random sampling on generalizability.

This study has a number of important strengths. Much research on obstetric QoC and its potential measurement has relied on routine data sources that are not designed for or suitable for quality assessment. Observations, such as those conducted in the QoC Assessments, may provide improvements in completeness, accuracy, and specificity [29‐31]. This study is also one of very few to include validation of quality measures with empirical data from low income country settings, and the only one to focus on the time of delivery.

The delivery-only index may reduce the burden of observation sufficiently to enable periodic L&D care quality assessment at the facility level, complementing other clinical supervision and records-based monitoring activities. All users must be oriented to the fact that this tool is not intended to be a comprehensive clinical guideline, checklist, or job aid; however, it can be used to provide valid information on care quality through targeted observation and may address gaps that have been identified through global MNC research and monitoring.

As greater attention is paid to the fact that QoC must improve if the global targets for maternal and neonatal mortality reduction are to be achieved, understanding of the construct is evolving [48]. A recent study by Souza et al. concludes that coverage with life-saving interventions may be insufficient to reduce maternal deaths without improvements in overall care quality [49]. This nuanced understanding of QoPIIPC suggests that observation of care may be crucial in quality assurance and improvement (QA/QI). Key aspects of QoPIIPC, such as provider-patient interactions and provider vigilance of danger signs, are not captured in medical records and registers. Tools that bring observation out of the research setting and into programs are necessary to address gaps in knowledge about routine L&D care quality, particularly as most assessment of QoPIIPC has focused on adverse events such as deaths and near misses [50‐52].

The need for valid quality assessment becomes particularly urgent as incentives to women for facility delivery, removal of user fees, performance-based financing for providers and health facilities, and other trends increase the use of facility-based L&D care [18‐20, 53]. In-depth verification of QoC contributes to QA/QI initiatives and is essential when providers and facilities are paid for performance [54, 55]. Anecdotal and program evidence suggests that when specific actions are emphasized in policy, their performance may be affected in ways that cannot be detected through record review. For example, partographs may be filled in after labor if completed partographs are rewarded in performance-based financing programs [56]. An improved ability to efficiently measure QoPIIPC may also strengthen the validity of future research on quality assurance and improvement within maternal and newborn health services.