Development and internal validation of the multivariable CIPHER (Collaborative Integrated Pregnancy High-dependency Estimate of Risk) clinical risk prediction model

Thirteen collaborating sites with ICUs from 11 countries contributed data to the CIPHER cohort. These sites were identified through a literature review of published obstetric ICU cohorts. After initial contact to establish whether the investigators were interested in collaborating, they were sent a survey to ensure suitability of the facility for participation in the CIPHER cohort. This survey evaluated human resources, interventions, and infrastructure available at each ICU site to ensure that sites were similar with regard to type of care provided. The HICs were in Canada, the US, Ireland, the Netherlands, and Australia. The LMIC sites were in Brazil, Argentina, Jordan, Turkey, Pakistan, and China. (For details, please see Additional file 1: Table S1.)

Women were included if they were admitted to a critical care unit for more than 24 h and were known to be either pregnant (diagnosed before or during their ICU stay) or no more than 42 days post-partum, irrespective of pregnancy duration, from 1 January 2000 to 31 December 2012. Women admitted for less than 24 h who were neither pregnant nor recently pregnant were excluded, as were women with 10 or more missing candidate predictor variables or those who were missing primary outcome information or who met the definition of the primary outcome prior to admission.

We carried out a retrospective chart review of both paper and electronic medical records by using standardized data collection forms and protocols for all sites. Data were entered into a customized, secure online RedCAP^® database. All data were reviewed for quality and consistency. When questions arose regarding data reliability, these data were confirmed by re-review of the primary health record.

The composite primary outcome was defined as any one of (i) maternal death during pregnancy or within 42 days of delivery or (ii) organ support for more than 7 days or (iii) life-saving intervention or a combination of these. Organ support and life-saving interventions included in this composite were defined by organ system and are used as a surrogate for severe maternal morbidity. Specifically, organ support outcomes include any one or more of (1) respiratory (continuous positive airway pressure (CPAP), bilevel positive airway pressure (BiPAP), or invasive ventilation); (2) cardiac (positive inotrope or vasopressor use); (3) continuous renal replacement therapy for acute renal failure; (4) hepatic (liver transplantation and other management of hepatic failure (for example, ventilatory and circulatory support), management of elevated intracranial pressure and renal failure, and medical therapies for hepatitis B virus (for example, lamivudine)), and anticoagulation for Budd–Chiari syndrome); (5) hematologic (transfusion of at least 5 units of blood products); (6) neurological (Glasgow Coma Scale score of less than 10); or (7) uterine (uncontrollable hemorrhage or infection leading to hysterectomy). These definitions were arrived at through study working group consensus and are based on definitions of organ support used in the APACHE studies [5] and the World Health Organization (WHO) near-miss approach [10].

This sample size estimate is based on a rule of thumb for developing risk prediction models with unbiased estimates of regression coefficients [14]. The formula—[N = (n × 10) / I]—was used to calculate the sample size where N = the required sample size, n = the number of variables to be tested, and I = the incidence of the combined adverse outcome [14]. We assumed, on the basis of published reports, I = 12% for either maternal mortality or prolonged organ support for obstetric women admitted for critical care [15]. We estimated that to develop a reliable model with minimal overfitting with n = 10 candidate predictor variables at an assumed event rate of 12%, a cohort of N = 833 women was required.

A comparison of women with and without missing data shows that women with missing data were less severely ill and had shorter duration of ICU stay (Additional file 1: Table S3). Imputation models specified for this study included all variables that differed between women with or without one or more missing selected predictor variable to best account for these differences. Univariate odds ratios estimated for all candidate predictor variables in both the imputed data (Table 3) and complete case cohorts (Additional file 1: Table S2) were similar, and an expected increase in precision was seen in the estimates generated using the imputed data.

The final variables selected for inclusion in the CIPHER model were maternal age, surgical status in the preceding 24 h, systolic blood pressure, Glasgow Coma Scale score, serum sodium, serum potassium, activated partial thromboplastin time (aPTT), serum creatinine, and serum bilirubin, and arterial blood gas (ABG) pH (Table 4). The odds of experiencing an adverse outcome increase as serum creatinine, total bilirubin, serum sodium, and aPTT increase; decrease as maternal age and systolic blood pressure decrease; and decrease if ABG pH and Glasgow Coma Scale score increase or if there was surgery in the 24 h preceding ICU admission (Table 3).

The apparent AUROC for the CIPHER model was 0.84 (0.83 to 0.85) (Fig. 2). This model was well calibrated in the development data, as would be expected, with a calibration slope of 1.0 and intercept of −0.001 (Fig. 3). Stratification capacity and classification accuracy of the model as presented in Table 5 were both good; the majority of women identified in the two lowest risk groups (54.1% women, n = 416) had low rates of adverse outcome (3.8%, n = 16). Women in the highest-risk group had a high incidence of adverse maternal outcome (66.1%, n = 59). This is meaningfully greater than the population prevalence of adverse outcome (16.5%), as is reflected by the high LR associated with this category (27.97, 95% confidence interval (CI) 16.91 to 46.27).

After 200 iterations of bootstrapping in each of the 10 imputed datasets, the pooled average optimism for the AUROC was 0.013, which results in an optimism-corrected AUROC for the CIPHER model of 0.82 (95% CI 0.81 to 0.84). Minimal overestimation of risk was identified after bootstrap analysis with a resultant optimism-corrected calibration slope of 0.92 (95% CI 0.91 to 0.92) and intercept of −0.11 (95% CI −0.13 to −0.08).

All subgroup analyses demonstrated uniformity of model fit with maintenance of discriminative performance of the CIPHER model above our defined threshold for adequate performance of AUROC of more than 0.7. When the cohort was restricted to only those cases admitted to the ICU during the antenatal/intrapartum period, versus the post-partum period, the AUROCs were estimated as 0.84 (95% CI 0.78 to 0.91) and 0.83 (95% CI 0.79 to 0.89), respectively. When the cohort was restricted to only those cases admitted for obstetric causes versus those with non-obstetric causes, the AUROCs were 0.85 (95% CI 0.80 to 0.90) and 0.82 (95% CI 0.76 to 0.88), respectively. When the cohort was restricted to either LMIC facilities or HIC facilities, the AUROCs were estimated to be 0.851 (95% CI 0.812 to 0.894) versus 0.774 (95% CI 0.663 to 0.868), respectively. There was a small but meaningful decrease in the point estimate of AUROC when the analysis was restricted to only HIC cases and the lower limit of the confidence interval falls below the 0.7 threshold for a good model.

When the CIPHER model was used to predict maternal death alone, the AUROC was 0.87 (95% CI 0.86 to 0.88). The discriminative performance of the APACHE 2 score for death during pregnancy or less than 6 weeks post-partum in our ICU cohort was also high (AUROC 0.84, 95% CI 0.72 to 0.96). This analysis included only 433 women who had complete data on all 17 APACHE 2 predictor variables.

We have developed and internally validated the CIPHER clinical risk prediction model to accurately assess risk of either death or the need for life-saving prolonged organ support for pregnant or recently pregnant women admitted to an ICU at 13 international sites. The final CIPHER model includes predictor variables that are readily available globally and at relatively low cost. It is a simple model, including only 10 predictors. After internal validation, CIPHER affords high discrimination (0.82, 95% CI 0.81 to 0.84) and good calibration (slope of 0.92, 95% CI 0.91 to 0.92 and intercept of −0.11, 95% CI −0.13 to −0.08). External validation of the model is now required prior to implementation of the model in clinical practice.

The CIPHER model was developed specifically for use in obstetric ICU populations with a globally diverse cohort involving collaboration across 11 countries. Although inclusion of a diverse geographic sample leads to an increase in global relevance, it also likely contributed to a reduction in overall model performance at the local level. A reduction in overall performance is evident when we compare performance seen in the high- and low- or middle-income country subgroups. In both settings, performance is maintained above the threshold for an adequate prognostic model (AUROC >0.7) but CIPHER is better at discriminating between women with and without outcomes in the low- or middle-income population, where the majority of outcomes occurred. A potential weakness of the dataset used is the variability in outcome rates between sites; 88.6% of outcomes occurred in LMIC sites, 57.6% solely in the Pakistan site. For this reason, we recommend external validation and, if required, recalibration of the model in each setting individually before application in clinical care.

Performance of the CIPHER model to predict maternal death alone was similar to the APACHE II model in our cohort. We chose to use both maternal death and the need for life-saving organ support as a primary outcome in order to make the CIPHER model more clinically useful for a pregnant population than a model such as APACHE II, which predicts death alone. Severe maternal morbidity is as significant in its life-altering consequences as mortality within this young and otherwise healthy population of women.

Another limitation is that because clinicians were not masked to the results of the variables assessed in the modeling, CIPHER is vulnerable to treatment paradox [30]; this may be particularly true for the protective nature of post-surgical status that may modify both admission and intervention thresholds and surveillance intensity.

Identifying the variables that predict outcome in pregnant or recently pregnant women admitted to the ICU and developing a prediction model enables estimation of the likelihood of an adverse maternal event in the future on the basis of information available at the time of a woman’s admission to the ICU. The candidate predictor variables for the CIPHER model were those that were routinely and reliably measured, were readily available in hospitals worldwide, and had potential to inform or predict severity of illness or outcome. The definition of severe maternal morbidity was organ- and management-based, reflecting the true burden of disease in the ICU: both the need for organ support and the impact of prolonged duration of organ support and care.

In the development cohort, a threshold CIPHER score of at least 50% was deemed a “positive” test for the combined outcome to define a high-risk group. The LR of 27.97 (95% CI 16.91 to 46.27) for this group is strongly informative. In this group, with an at least 50% CIPHER risk, there is evidence for action as it identifies those women who are most at risk of a combined outcome. In areas where resources are available to manage additional case load or where greater concern exists around impact of missing true-positive cases, setting the threshold for high risk as greater than 25% predicted probability remains informative with an LR of 9.18 (95% CI 6.16 to 13.69) with only a small increase in associated false-positive rate. This means that useful clinical information can be gained from the CIPHER model in order to guide care in a variety of contexts.

Four published studies have focused on development of a maternal ICU outcome prediction model [10, 31‐33]. Developed solely in Brazil, the Maternal Severity Index used predefined, rather than statistically driven, WHO severity markers, identifying seven predictors of maternal death, many of which were themselves composite predictors [10]. A secondary analysis of a cohort of maternal general ICU admissions from the UK that focused on evaluation of APACHE II variables identified medical history, heart rate, systolic blood pressure, and especially Glasgow Coma Scale score as independently predictive of maternal death [31]; however, they did not develop a multivariable prediction model. Nine independent variables predictive of maternal death were identified in a West African (non-ICU) hospital-based study [32]. Of these, many were indications for ICU admission in our study and were not included as candidate predictor variables, including severe anemia, malaria diagnosed during pregnancy, obstetric hemorrhage, pre-eclampsia or eclampsia, uterine rupture, and genital infection or sepsis. Again, these variables were not formally combined to generate a comparable predictive model. A US military cohort was used to develop a 13-variable risk assessment model to predict 38 maternal outcomes, including labor, delivery, maternal morbidity, and death [33]. Performance of this model was good with a reported AUROC of 0.75 for poor maternal outcome, but the model is applicable only to the antenatal period. None of these studies resulted in models overtly applicable to critical care of obstetric patients in both HICs and LMICs.

Future work in this area of research should focus on a number of translational biomarkers poised to become regular components of both maternity and critical care and with potential to modify CIPHER [34, 35]. Prospective external and temporal validation studies of CIPHER are required prior to its broad dissemination into communities of care, whether in LMICs or HICs. To support work towards external validation we have provided a CIPHER calculator (see Additional file 2). An additional goal for validation could be to expand the scope of the CIPHER model to recalibrate in a population admitted to a high-risk maternity unit in order to guide decisions around admission to the ICU.