Construction and pilot test of a set of indicators to assess the implementation and effectiveness of the who safe childbirth checklist

A working group was established with the participation of the obstetrical, perinatology and quality management staff from four hospitals in the state of Hidalgo, the state of Mexico and Mexico City, supported by a research team from the National Institute of Public Health. The working group adhered to the WHO call for Collaboration on the implementation of the SCC [12]. Following the WHO SCC Collaboration recommendations, the original SCC [22] was refined and adapted to the Mexican context after several iterative discussion sessions. Adaptation was carried out for both content and format. For instance, in relation to format the original WHO SCC is a single document including childbirth and newborn, while our group decided to have separated checklists (one for mother and another for the newborn) to better accommodate the SCC to the structure of hospital childbirth care responsibilities (see online Additional file 1 in Spanish and Additional file 2 in English). In relation to contents, whereas the original SCC asks for simple checks, we added additional information to complete information and reinforce their use as reminder for best practice. For instance, the original SCC ask to check if there was a cesarean section, but in Mexico we added to check on the reason or justification for the cesarean section, according to a list proposed by the working group based on accepted Practice Guidelines.

The main basis for the set of indicators related to best practices was the evidence-based structure and process-of-care items contained in the original WHO SCC, converted into indicators. These items and indicators were double-checked and compared with the official recommendations of the Clinical Practice Guidelines in Mexico, to reassure validity and avoid inconsistencies between locally accepted evidence-based practices and the WHO standards for maternal and child care [23]. A group of four researchers with expertise in quality improvement classified the resulting indicators according to type (structure, process, outcome) and potential data source (clinical record or survey) and prepared these indicators for pilot testing by defining the technical specifications of the indicators in a standardized format used previously in other studies [24, 25]. Electronic applications for the computerized capture of data using laptops or tablets were also developed. The detailed definitions of the indicators are available online in Spanish (Additional file 3) and English (Additional file 4). For the indicators regarding the SCC utilization and the factors associated with the SCC, we adapted previous research on the implementation of the Safe Surgery Checklist [26] to the SCC.

Different strategies were used to measure the indicators depending on the expected data source (clinical records or surveys):

We performed a cross-sectional retrospective study of the indicators that were estimated from clinical records. In two hospitals, systematic random samples were obtained from cases with corresponding ICD-10 codes identified in the computerized system of hospital discharges (SAEH, acronym in Spanish). First, two samples (n = 15 each) were obtained from hospital A and were independently assessed by two pairs of evaluators; then, a second sample (n = 17) was obtained at hospital B, which was assessed by three evaluators. The seven evaluators did not participate in the construction of the indicators but were trained (10 h) in the use of the indicators and the electronic data capture. An inter-observer reliability analysis was performed by calculating the prevalence-adjusted, bias-adjusted kappa (PABAK) [27] for the pooled sample of 47 cases. An indicator was considered reliable if the PABAK value was ≥0.6, particularly when the observed prevalence was extreme (compliance or non-compliance ≥85%) [28, 29]. The results of this assessment were used to refine the tools and the unreliable indicators. The refined indicators were pilot tested again by four evaluators in a sample of 30 cases (15 cases per couple) in a third hospital. The pilot data were also used to assess the feasibility of measurement. All data were analyzed with the statistical packages Epidat 4.1 [30] and Stata 14.1 [31].

For the indicators obtained from questionnaires, we organized four sessions in two of the participating hospitals following the pre-test cognitive methodology [32], which includes concurrent observation of the behavior of the interviewee and reading/responding aloud. Cognitive methodology was used to ensure respondents understanding of the questionnaire. Reactions and responses were used to change the wording of the questions, when necessary, in order obtain a final version of the questionnaire. The group utilized to test the health personnel questionnaire included 12 professionals. The group for testing the questionnaire for mothers included 15 mothers. Finally, the surveys were fully pilot tested with individual interviews with 5 health professionals and 6 mothers. Two researchers, with previous experience in interviews and surveys, conducted the cognitive sessions and pilot interviews and analyzed the data.

We constructed 53 indicators (Table 1), 15 of which were used to assess the factors associated with utilizing the SCC and the SCC utilization itself (Table 2). The remaining indicators (38), which are described in Tables 3 and 4, were constructed to monitor the impact of the SCC, mainly on good practices but also on outcomes related to complications and adverse events (Table 3). Among these indicators, 25 relate to healthcare for the mother, 11 relate to the newborn and 1 indicator relates to both. Mortality indicators (maternal and neonatal) were excluded from the present pilot study since these indicators are already well established and may be measured from existing databases rather than internally at the institutions using the SCC. The good practice (structure and process) indicators were further classified according to the phases of the SCC: admission and labor, immediately post-partum, and prior to discharge. Indicators that may apply to various stages were grouped in a “general” category.

The SCC includes reminders for interventions with proven positive impacts on quality childbirth and the management of complications and mortality in both the mother and the newborn, but the SCC was launched without identifying indicators that may be used to monitor the utilization and impact of the SCC. However, valid measures allow the monitoring of processes and results of strategies, thereby evaluating the strategies and facilitating decision making [33]; monitoring is necessary for the effective implementation of a strategy and contributes to preventing the strategy from becoming a mere list of intentions [21]. Measuring and reporting results is also important for accountability and transparency [33]. In recent years, particularly after the completion of the Millennium Development Goals, there has been an emphasis on the design of valid strategies and measuring instruments to monitor interventions and enable comparisons at the national and international levels [20, 23, 33]. To better analyze the attributed impact and eventual implementation strategies, we believed that a set of specific indicators closely related to the SCC content was required to evaluate the SCC impact, along with measures of SCC utilization. To date, the largest study on the implementation of the SCC, the Better Birth Trial, assessed the SCC use in 18 of the 39 best practices for which the SCC provided reminders and was based on the comparative observation of coaches and independent observers when coaches were not present [16]. The main objective was to assess the impact of coaching on SCC use, and the analyzed practices were therefore those that could be observed by either coaches or independent observers when the coaches were not present. No attempt was made to provide indicators that could be used to assess the SCC strategy as such, including factors other than coaching that may influence the SCC use. Other studies [17, 18] primarily focused on the completion of the SCC when used, excluding from the analysis all cases in which the SCC was not utilized.

Our proposed indicators are intended for use in monitoring the implementation and effectiveness of the SCC, allowing for comparisons among cases and also institutions where the SCC has not been used. Therefore, the target population was total deliveries, and the data sources were not only those cases in which the SCC was utilized. The numerator and denominator, the terms used and proposed data sources may thus be carefully defined. However, regardless of how well-defined the terms and data sources of an indicator are, pilot testing in actual environments is necessary to guarantee the feasibility of a measurement, an unambiguous interpretation and an eventual technical refinement to increase reliability. A pilot test may reveal that the specifications must be more detailed or that the data may not be available [34‐36]. Skipping this step may result in proposing unfeasible or unreliable, and therefore not valid, indicators [36]. We observed all of these types of problems. In some cases, we discussed and refined the specifications or the seemingly unreliable indicators based on the PABAK value obtained in the first pilot test, and in other cases, we highlighted problems with the data source that may affect the generalization of the proposed measures. It was most difficult to find correctly recorded time specifications, and in some cases, we changed specifications to include different ways in which such recorded data could actually be compliant but disparate. For example, “oxytocin administered during the first minute” was sometimes recorded as “oxytocin given after the liberation of shoulders” or “was given oxytocin immediately after the release of the product”; for this reason, we included these response options as valid compliance in the indicator description. However, the majority of feasibility problems related to deficiencies in the data sources, including the identification of cases for assessment and the quality of the recorded information in clinical records and other existing documents, such as partograms, which underscores the importance of context-specific obstacles to monitoring otherwise potentially valid indicators. In relation to the identification of cases (i.e., deliveries) in the routine hospital discharge system, we observed that in many cases, deliveries were not coded as such; the existence of any previous condition or complication as the cause of hospitalization was preferred in many cases. We also identified (and propose to identify) childbirths by searching individual records for other variables that are related to childbirths, such as the gestational age, in combination with the recording of a birth product. There were women with “gestational age” and “birth products” not coded as deliveries.

After selecting cases, extracting the necessary information may also be complex. The lack of uniformity in the formats of medical records is a major contributor to the confusion. Many times, the required information could be recorded in different places or formats, and it is necessary to clarify what format should be revised for each variable. For instance, gestational age may not be in the medical record but may instead be in the partogram. In some cases, when the same information was recorded in different formats, it was possible to detect inconsistencies (even in relation to simple variables, such as the sex of the newborn or the type or birth, vaginal or cesarean) that contributed to context-specific causes of the lack of data reliability.

These findings from the first pilot study in two hospitals and the entire process of refining the indicators were fundamental to obtaining more reliable tools, which were tested again in a second pilot study in a third hospital. However, two of the proposed indicators, both related to the recording of the appropriate time in which the required actions should be performed, had questionable reliability results in the second pilot study. One of these indicators, related to the contents of the partogram, showed reliability in the first pilot study. This discrepancy suggests the potential need for pilot testing in a given environment, even for indicators that are reliable in other environments. Such a pilot test can be performed with small sample sizes [36], but for national or regional projects, a period of monitoring data quality and availability as well as designing strategies to generate timely and reliable data have been suggested [37, 38] .

Some of the SCC items, although relevant, apply to sub-populations of mothers and newborns defined by a given condition. Routine monitoring of these indicators would not be possible unless the information system allows for the easy identification of cases with that particular condition, given the relatively low frequency in which these cases may occur. We classified these indicators, a total of six indicators, as routinely non-feasible. The timely initiation of antiretroviral treatment for HIV+ women and their newborns is a clear example of this type of indicator, which applies only to the subpopulation of known HIV+ women. The same situation occurs with other cases that cannot be readily identified in the discharge information system, such as the justification of antibiotic treatment, the justified administration of magnesium sulfate, and even the adequate management of postpartum hemorrhage. Larger or specific purposive samples, in cases in which these situations could be identified in routine databases, may be required to measure these indicators. In other SCC studies, based only on those cases in which the SCC was utilized and using the SCC checked items as the data source [17, 18], these indicators, particularly those related to HIV+ women and their newborns, show some of the lowest compliance (compliance meaning checking the item in the SCC). In other cases, such as the in the study by Spector JM et al. [11], it is not clear whether compliance includes checking the item defining the sub-population (i.e., assessing HIV status) plus giving the treatment as an “all-or-none” composite indicator. In addition, in our study, the SCC adaptation distinguished two steps: one, whether a treatment or intervention has been provided (for instance, antibiotics, magnesium sulfate, or episiotomy); and two, the explicit justification for that treatment or intervention. Monitoring the quality of the second step in these practices may not be routinely feasible unless the prevalence of the subpopulation to which they apply is sufficiently high in a routine sample of childbirths. The original SCC simply asks whether the particular treatment or intervention is needed and has been administered (in a single Yes/No item) and then provides some advice (a short list of symptoms or situations) regarding the indications for the treatment. In our adaptation, the group of experts considered it more appropriate to transform these items into two companion and sequential items: first, checking whether the treatment was administered (Yes/No), and then, checking the symptom or situation that justifies the treatment. In this way, the presence of any of these symptoms may also be an explicit reminder for considering the treatment if it has not been administered; at the same time, these questions may be used to assess adequacy.

However, there are important aspects (skin-to skin-contact, the initiation of breastfeeding, information about warning signs, among others) for which we could not expect good registered data. The same situation occurred in the search for potential factors that may affect the SCC use, namely those related to the attitudes of health personnel, the existence of inputs and other structural factors. For both types of measures, we devised and successfully tested short questionnaires for mothers and health personnel. Without these additional tools, a comprehensive monitoring and evaluation of the use and impact of the SCC and the subsequent design of improvement initiatives would not be possible.

The technical specifications of the proposed set (Additional file 1 in Spanish and Additional file 2 in English) are consistent with the findings of the pilot tests and include the proposed data source, making these indicators ready for use in the Mexican environment and also for adaptation to other environments.