Health providers pass knowledge and abilities acquired by training in obstetric emergencies to their peers: the average treatment on the treated effect of PRONTO on delivery attendance in Mexico

The PRONTO simulation-based training, a training course on the management of obstetric emergencies, was developed and piloted in Mexico in 2009. Although the training is not oriented toward the management of normal deliveries, that topic is included in several sessions of the curriculum [17]. A cluster randomized controlled trial was conducted in 24 hospitals in three Mexican states to evaluate the impact of PRONTO. A total of 450 participants (54% physicians and 46% nurses) received the PRONTO training at the 12 intervention hospitals. The selection of these participants was carried out by the authorities in the selected health facilities, but the main selection criterion was that the trainees were personnel who attended deliveries or worked in emergency or delivery rooms [16]. For the original study, hospitals were matched one-to-one based on similar characteristics and incidence of obstetric complications; one member of the hospital pair was then randomly assigned to receive the training. Further details on the original impact evaluation study and the methods used in the training have been previously published [18].

Data collection began in August 2010, and the follow-up concluded in March 2013 in the 24 participating hospitals. Five field workers were instructed to conduct a structured observation of births from the beginning of the second stage of labor until 10–20 min after the third stage was completed, using a paper-based checklist to collect the data. During each data collection period (the baseline and follow-ups at 4, 8, and 12 months after the intervention), the goal was to observe at least 10 vaginal births attended by different providers on different shifts at each hospital over a maximum 5-day observation period. Prior to the observation of births, both the health provider and the laboring woman provided oral consent. The observers were not blinded to the treatment allocation, but they were blinded to the treatment status of individual providers. A more detailed explanation is provided elsewhere [16].

The original dataset analyzed by Fritz, et al. (2017) was a set of 641 observed deliveries [16]. However, to estimate the ATT effect, it was necessary to create a unique identifier for each health provider and to distinguish those providers who were trained using PRONTO from those who were not. Because the original study design was not planned to allow for analysis at the provider level, no unique provider identifier was recorded. Rather, the names of delivery attendants were recorded in an informal, non-standardized fashion. Consequently, there was vast heterogeneity in the way these names were written, which made it impossible to create unique identifiers in a straightforward manner. An additional problem was that the gender of the provider was not recorded. In some cases, we were able to infer gender from the name. However, this was not always possible, especially for interns, who were usually referred to in hospital notes only by the prefix MIP and their family name (e.g., “MIP Perez,” which means “Undergraduate Intern Medic Perez” and is a gender-neutral expression in Spanish).

A uniformization of the providers’ names was performed using the Stata command STRGROUP [19], which matches similar names of providers within each hospital and profession. The algorithm for the matching between all pairwise combinations of strings into the “provider name” variable considered the estimation of the Levenshtein distance, defined as the minimum number of single-character edits (i.e., insertions, deletions, or substitutions) required to change one string into the other [20]. A normalization was then performed by dividing the distance by the length of the shorter string in the comparison. Subsequently, we matched a string pair if the names’ normalized distance was less than or equal to a threshold of 0.25, which corresponded to the 10th percentile of the distribution of Levenshtein distances. The algorithm was also applied to identify name similarities among the list of participants in the training, which were stored in a different dataset. Finally, each set of matches was assigned a unique provider identification number, and this was manually re-checked prior to analysis. The health provider’s gender was inferred from their name or from clues such as the use of “Dr.” or “Dra.” as prefixes for the name. We were unable to retrieve information on the name of the provider for 3% of the observed deliveries, and we were unable to identify the gender of the provider for 12% of cases. We were able to ascertain gender for 100% of the treated providers. The study protocol for the original study (Reference 845), as well as for additional data gathering and analysis on individual providers (Reference 733) were reviewed and approved by the Ethics and Research Committees of the National Institute of Public Health in Cuernavaca, Mexico.

The original dataset of 641 observed deliveries was collapsed by hospital and stage (4, 8, and 12 months following the training) to a dataset of 310 providers who attended at least one delivery after the time of the training. We summarized the outcomes as the proportion of attended deliveries in which routine practice was followed. The unit of analysis was the i^th provider at the t^th stage of the study, for a final analytic sample size of 356 observations, of which 50 corresponded to observations of trained providers, 141 were observations of non-trained providers in treated hospitals, and 165 were potential controls (providers observed in control hospitals). The study flow chart is included as supplementary material [see Additional file 1].

The main outcomes were the routine practices performed during delivery. Three practices can improve patient outcomes and prevent significant childbirth complications, according to the WHO [9]: 1) Active management of the third stage of labor (AMTSL), defined as (1st Step) applying 10 international units of oxytocin in the first minute after the birth of the baby, (2nd Step) traction and countertraction of the umbilical cord, and (3rd Step) uterine massage immediately after the birth of the placenta; 2) use of delayed cord clamping (DCC), defined as a delay in the clamping of the umbilical cord of at least 60 s after the birth of the baby; and 3) Skin-to-skin contact (SSC), defined as immediate contact or attachment between mother and child after birth. Three unsubstantiated and harmful practices that can worsen patient outcomes if used routinely, according to the WHO [9] are as follows: 1) episiotomy, defined as the performance of a surgical incision in the woman’s perineum, including the skin, muscular plane, and vaginal mucosa; 2) fundal uterine pressure (the Kristeller maneuver), defined as the application of manual pressure on the upper part of the uterus directed toward the birth canal; and 3) uterine sweeping, defined as the introduction of a hand or a gauze-wrapped instrument into the fundus of the uterus after the birth of the placenta.

The main explanatory variable was participation in the PRONTO training (yes/no). Important covariates at the individual level were the health provider’s gender, shift, and profession.

The following equation is the mathematical approach used to estimate the ATT effect of the PRONTO training (δ₁), which can be understood as the expected difference on the performance of routine practices (AMTSL, DCC, SSC, episiotomy, fundal pressure, and uterine sweeping) between those in the intervention group and those who were trained:

In this equation, y₁ is the probability of a trained provider performing a routine practice and y₀ is the probability of practice performance with no training (counterfactual), given that the participant was actually trained. The ATT impact estimators were obtained by comparing the prevalence of practices performed by participants with the prevalence among non-participants who would have taken the training had they been offered the opportunity to do so, the prevalence among this non-participants serves as a proxy of the counterfactual: the prevalence on those who were indeed trained in the hypothetical situation were the training never took place. To accomplish this, and because participants in the training were not randomly selected within the treated hospitals, we matched each trained provider with a non-trained provider from the control hospitals, this procedure relies on the assumption that that conditional on the matched covariates, the trained providers and non-trained providers are exchangeable [21]. We performed the matching using one-to-one nearest-neighbor matching in terms of the minimum Mahalanobis distance [22], which was defined using the following covariates: matched hospital pair, state, gender, shift, and profession. Additionally, to control for trends in the probability of performing the practices, we matched exactly on time elapsed since the training (4, 8, or 12 months). Because the accuracy of the estimators depended on the comparability of the treatment groups, we performed a sensitivity analysis using matching to the one, two, and three nearest neighbors with the tebalance Stata routine. We chose the one-to-one matching procedure, because it provided more than twice the bias reduction (in terms of the average difference in covariates between the groups) yielded by the use of two or more neighbors.

To estimate the possible effect of PRONTO on those who were not directly trained (positive externality), we performed an estimation of the ATE for non-treated providers (ATNT), matching those who did not attend the training but were working in treated hospitals to those working in control hospitals, otherwise following the same approach described above. Finally, to test for possible decay (linear trend) of the probability of treated providers performing the examined practices in the post-training period, we fitted a provider-level fixed effects regression model with robust standard errors and clustering correction at the hospital level, including time after training as a continuous variable with time units of four months (time elapsed between observation rounds). All analyses were performed with Stata 14.0 (StataCorp LP, College Station, TX, USA). All relevant data is included in anonymized form as a supplementary file [see Additional file 2].