A machine learning approach for modeling decisions in the out of hospital cardiac arrest care workflow

The Chicago Fire Department (CFD) is the sole EMS agency providing emergency response and transport for 911 calls for Chicago’s approximately three million residents. Upon identification of an OHCA by EMS dispatchers, a basic or advanced life support fire suppression company, and advanced life support transport ambulance are dispatched to the cardiac arrest. The Chicago EMS system responds to over 2500 OHCA incidents annually and has 33 receiving hospitals for OHCA incidents, including 24 ST-elevation myocardial infarction (STEMI) hospitals with interventional cardiology and targeted temperature management capabilities [18]. Per protocol, all OHCA patients treated by CFD EMS who either achieve ROSC or with refractory ventricular fibrillation or ventricular tachycardia are transported to STEMI receiving centers in order to ensure access to early coronary angiography and revascularization as these are a critical component of post-resuscitation care [19].

This study utilizes Cardiac Arrest Registry to Enhance Survival (CARES) data for Chicago [20]. The CARES registry is the largest cardiac arrest data source in the United States, collecting 911 dispatch centers, EMS providers, and receiving hospital data from over 1800 EMS agencies and 2200 hospitals [21].

The CFD EMS treated 12,904 non-traumatic OHCA incidents between September 2013 and December 2018. This study focuses on OHCA workflow decisions made post-ROSC including 2398 OHCA incidents that survived to hospital admission. Figure 1 shows how the data is preprocessed for the study from the full Chicago CARES data. The information used for modeling includes 23 input features for decision modeling and 24 input features for survival modeling (see Additional file 1: Table S2 for detailed lists of the decisions and outcomes in this data source). During the modeling process for the decision and survival models, the information has a sequential nature where the only new piece of information in the survival model is the Coronary Angiography. The input features used to develop and evaluate models are all categorical features. To accommodate ML models, the categorical features are either one-hot encoded or transformed into a vectorized representation [22]. Traditional forms of machine learning algorithms require categorical features to be one-hot encoded where a single feature is transformed into multiple features that represent the original information. For example, if the feature regarding the initiation of cardiopulmonary resuscitation (CPR) has the options of Lay Person, First Responder, or EMS Personnel and the instance receives First Responder CPR the value is encoded as [0,1,0]. In the neural network models, certain architectures allow for the utilization of embedding layers. These embedding layers allow for the model to learn a vectorized representation of the features instead of going through the one-hot encoding process.

This study models post-ROSC decisions and the neurologic outcome of OHCA incidents. CARES includes data on two post-ROSC decisions: (1) whether a patient underwent targeted temperature management (TTM) and (2) whether coronary angiography (CA) was performed. The 2015 AHA Guidelines recommended (Class I) that comatose adult patients with ROSC after OHCA have TTM and CA for OHCA patients with suspected cardiac etiology of arrest and ST-elevation on ECG (Class I). The 2015 AHA Guidelines also stated that CA is reasonable even in the absence of ST-elevation (Class IIa) as CA, alone or as part of a bundle of care, is associated with improved myocardial function and neurological outcomes [23‐25]. Due to wide variation in TTM practice and absence of granular data regarding individual hospital practices (e.g., temperature selected, length of temperature management, rewarming practices), we did not consider the decision to perform TTM as one that could be reliably predicted in decision models. Instead we opted for the post-ROSC decision of CA, where the modeling process aims to determine whether or not a patient is expected to receive a Coronary Angiography. The CA model refers to the decision model throughout this paper.

We also examine the outcome of neurologic function from the hospital record, measured in CARES by Cerebral Performance Category (CPC) score [26]. For this study, neurologic outcome was modified to a binary classification based on CPC score: Class 0 of individuals who survived with functional neurological outcomes (CPC1/2) and Class 1 of patients with non-functional neurological outcomes (CPC3/4/5). The CPC model aims to classify patients into one of these classes based on the care and condition of the patient. The CPC model refers to the survival model throughout this paper.

To classify the OHCA workflow decisions and neurological outcome, we train ML models using a subset of the available data. We then evaluate the performance of the developed model on the remaining data, i.e. data not used in the training phase. For the 2398 OHCA instances in the study, 21.5% receive a Coronary Angiography, and 20.8% have a neurological outcome of Class 0. Figure 1 subsets the 2398 OHCA instances for this study into four sets: training, validation, testing, and cohort analysis. The modeling sets are split randomly across the data timeframe because the decision-making policies implemented during the timeframe should be consistent. The training set consists of 957 events (40% of data), and is used to construct the ML models. The validation set consists of 241 events (10% of data), and is used for model parameter optimization and model comparison. The testing set consists of 600 events (25% of data), and is used to evaluate models on completely unseen data. The cohort analysis set consists of 600 events (25% of data), and is used to perform further analysis regarding post-ROSC decisions for cohorts that will be formally defined in the Cohort Analysis subsection. Additional file 1: Table S1 provides additional details for the data demographics for each set.

To model the decisions and survival outcomes we compare several ML algorithms. LightGBM [27], XGBoost [28], Decision Trees [29], Random Forest [30], Gradient Boosting [31], k-Nearest Neighbor [32], Logistic Regression [33], Support Vector Machine [34], and Deep Neural Networks [35]. ML models including Logistic Regression, Decision Trees, and k-Nearest Neighbor are popular algorithms in a variety of healthcare applications because they are highly interpretable to the user. The remaining developed ML models are generally more powerful algorithms, however they result in low interpretable outcomes. During the training process for all models a grid search of parameters performed to ensure that each model is optimized. Additional file 1: Table S3 provides the detailed parameters that are explored during this process.

For the neural network model, we utilized a modified Embedded Fully Convolutional Network (EFCN) [36]. Figure 2 illustrates the EFCN model architecture for the two types of models: Coronary Angiography and CPC Score. These two models are developed in a sequence where the Coronary Angiography EFCN is trained first, then the CPC Score EFCN. Due to the sequential nature of the modeling flow and the utilization of neural networks, transfer learning is applied by using pre-trained embedding vectors [37]. As shown in Fig. 2, the embedding weights for the Coronary Angiography EFCN model are randomly initialized and updated during training. For the CPC Score EFCN model, the pre-trained embedding vectors (yellow dashed boxes) are used as the initial weights for the embedding vectors and then updated during the training from the Coronary Angiography classification. The only randomly initialized weights for the CPC Score EFCN model is the embedding vector for the Coronary Angiography input, which is the only new feature in this model.

Each model is constructed using the training set and was iteratively improved using the validation set. The model that yields the best area under the receiver operating characteristic curve (AUROC) on the validation data is selected as the best model. The higher the AUROC, the better the performance of the model at distinguishing between classes. This model is then evaluated on the unseen testing data. AUROC is used as the primary evaluation metric. In addition, other metrics including the accuracy, area under the precision-recall curve (AUPRC), F1 Score, Matthew’s correlation coefficient (MCC) [38], and the Brier score loss (BSL) [39] are reported for all results. In addition, the SHapley Additive exPlanations (SHAP) [40] values for the best method is present. This method utilizes game theory to explain the prediction of an instance by calculating the contribution of each feature to the final prediction.

Chicago has 33 receiving hospitals. Using the developed ML models, cohort analysis of post-ROSC decisions was performed. Prior to analysis, the hospitals first are broken into cohorts based on the CPC1/2 rates for applicable OHCA instances. This analysis has the potential to be conducted on all hospitals, but to ensure generalizability we removed 18 hospitals from this analysis due to an insufficient number of OHCA instances to draw conclusions about the post-ROSC decisions. The cohort analysis data set includes 15 hospitals with at least 75 OHCA instances for the study period. The 15 hospitals are broken into three cohorts with five hospitals in each cohort. The CPC1/2 rate of each cohort is determined based on the combined CPC1/2 rates of the training and validation instances of the hospitals in the respective cohort. Cohort 1 consists of the 5 hospitals with the highest CPC1/2 rate of 35.9. Cohort 2 consists of the 5 hospitals with the next highest CPC1/2 rate of 18.3. Cohort 3 consists of the remaining 5 hospitals with a CPC1/2 rate of 13.2. Additional file 1: Table S1 provides a demographic breakdown of these cohorts.

The cohorts were used to retrain our ML models to evaluate how decisions differ based on hospital cohorts. Modeling retraining is done by taking the pretrained weights from the full modeling process and allows the model to reoptimize because on just a specific cohorts training set. The retraining process allows for the models to better reflect a cohorts decision making without started from randomized network weights. These cohort models are used to analyze how different cohorts make decisions when patients arrive with specific field conditions. The potential field conditions for analysis include the field data listed in Additional file 1:Table S2.