A machine learning model to predict critical care outcomes in patient with chest pain visiting the emergency department

Globally, chest pain of acute-onset is one of the most common presenting complaints in the emergency department (ED). It represents approximately 6 to 9 million visits per year in the USA [1, 2]. In this group of patients, initial assessment is guided by vital signs, ECG findings, levels of cardiac enzymes, and estimation of established risk scores. However, this initial risk stratification remains insufficient [3], and contributes to crowding of ED and delay in patient care, ultimately resulting in greater morbidity and mortality [4]. Thus, in an overcrowded ED with limited resources, it is essential to identify critically ill patients presenting with chest pain and take appropriate measures for the preferential management of these patients [5].

In the last two decades, an evolving literature related to the identification of a wide range of critically ill patients has emerged. Previous studies have developed models for predicting the clinical deterioration of patients admitted in wards. They used mortality, cardiac arrest, and transfer to intensive care units (ICUs) as their clinical outcomes, however, they could achieve only moderate performance [6, 7]. The aim for preparing the Acute Coronary Treatment and Intervention Outcomes Network (ACTION) ICU score was to predict the complications requiring ICU care in patients with non-ST elevation myocardial infarction (NSTEMI), but reported low accuracy both in development and externally validated cohorts, thereby resulting in restricted use in clinical prediction models [8, 9]. Several established clinical outcome scores have been used for risk stratification of patients with chest pain presenting to the ED, including the History, Electrocardiography (ECG), Age, Risk factors, and Troponin (HEART) [10]; the Thrombolysis in Myocardial Infarction (TIMI) [11]; and the Global Registry of Acute Coronary Events (GRACE) score [12]. The commonly used prediction outcome is major adverse cardiovascular events (MACE) [13], namely myocardial infarction, percutaneous coronary intervention (PCI), coronary artery bypass graft (CABG), coronary artery stenosis, cardiac arrest, all-cause mortality, etc. However, there are certain differences and partial overlap in the critical care outcomes of these scores. TIMI and GRACE are time consuming and only applicable in patients with acute coronary syndrome (ACS) [14, 15], thus leading to suboptimal use in patients with chest pain of various origin. Amongst various scores, the HEART score is most accurate and widely used for risk stratification of patients with chest pain [16]. It is used for safe discharge of low-risk patients [17], or identifying high-risk patient for occurrence of MACE [18]. Considering all these facts, it can be concluded that a little or no attention is being paid to the prediction of outcomes in critically ill patients presenting with chest pain. Thus, a great challenge lies ahead in constructing a promising prediction model to identify this group of patients.

In order to improve the risk predictive ability, amongst the patients presenting to ED, various machine learning (ML) algorithms (such as support vector machine [19], neural network [20‐22], random forest [21, 22], gradient-boosted decision tree [21, 22], and least absolute shrinkage and selection operator (LASSO) regression [21, 22]) have been used and demonstrated to have a satisfactory performance. Despite the advancement in the ML algorithms, major drawback of majority of these approaches is the lack of physiological sense due to the absence of a visual model [23], which may in turn result in dissatisfaction amongst and reduced use by the healthcare workers. Nevertheless, it is reported that logistic regression with LASSO regularization (LASSO regression) may address this gap. Based on the ML approach, LASSO regularization shrinks the regression coefficients toward zero, effectively selects the important predictors, and improves the interpretability of the model. With the use of this advance technology, we attempted to produce a formula, by using logistic regression, and provided a mechanistic model by nomogram.

The primary objective of this study was to develop a ML model “LASSO regression model”, using routinely available clinical features in patients with chest pain, and to accurately predict the outcomes in critically ill patients presenting at ED. Moreover, we compared the prediction performance of the LASSO regression model with the reference score.

We applied machine learning approach - LASSO regression model - to predict the likelihood of complications requiring ICU care, cardiac arrest, or mortality amongst the patients with chest pain admitted in ED. The LASSO regression model comprises of 7 clinical features available at initial contact of patient with ED i.e., mode of arrival, reperfusion therapy, Killip class, SBP, SCr, CKMB, and BNP. Compared to the reference models, the LASSO regression model demonstrated a superior performance in predicting critical care outcomes, including improved AUC value and other metrics. Moreover, the decision curve analysis revealed that LASSO regression model yields a larger net benefit—the trade-off between appropriate prediction and over-prediction—throughout the full range of thresholds. The use of this objective risk stratification tool may help the hospitals effectively use the limited ED resources while ensuring that high-risk chest pain patients are taken care of safely.

The reasons for the improved predictive abilities observed with the LASSO regression model are multifactorial. Firstly, the present study employed complete set of information; for instance, demographics, risk factors of ACS, ED presentation, initial laboratory and ECG findings, and medical treatment, all of these could have resulted in an improved predictive ability. Raita et al. [22] proposed an ED triage system to predict critical care outcomes—direct admission to an ICU or in-hospital death—based on the limited set of predictors only collected at ED triage, such as demography, triage vital signs, chief complaints, and patient comorbidities. Compared with this ED triage systems, the LASSO regression model outperformed in terms of the value of AUC, sensitivity, and specificity (0.86, 0.80, and 0.76, respectively). This is attributable, at least in part, to the limited set of predictors employed. Nevertheless, the differential study purposes and participants make it impossible to compare both the systems. The risk score generated by Raita et al. was proposed for emergency triage and rapid identification of the priority patients. However, it could compromise the accuracy. Contrarily, the LASSO regression model is proposed to be used in emergency room rather than at triage. We ensured the patient safety first, even though time consuming. The participants enrolled in both the studies make it impossible to compare both the systems, and the risk score proposed for emergency patients cannot be directly generalized to patients with chest pain. Moreover, the risk score, in the present study, had some indicators that overlapped with ACTION ICU score [8], which included SCr, SBP, and reperfusion therapy. The findings of some previous studies are inconsistent with that of the present study, notably, cTnI and CKMB have been used for several decades in diagnosing patients with chest pain and to stratify them into those with myocardial and non-myocardial infarction [28]. Estimation of serum troponin is the golden standard of evaluating the cardiac markers [26], however in the present study, cTnI failed to show a statistically significant difference in recognizing the critical care outcomes, nevertheless serum CKMB was measured. A plausible explanation for this inconsistency may be due to the differences in testing point between these two cardiac markers, where cTnI was estimated at an emergency triage immediately on arrival at the ED, and the CKMB was measured before the occurrence of critical care outcomes during the emergency room treatment. Thus, there was an obvious time difference in their estimation. Moreover, in ACTION ICU score, prior revascularization was associated with lower likelihood of developing in-hospital complications requiring ICU care. While, conversely, in the present study, it served as a risk factor because reperfusion therapy was defined as a current therapy, and may have a higher likelihood of presenting to the ICU post-operation.

Secondly, an alternative approach to enhance the predictive ability is to utilize advanced ML algorithm, which is capable of handling high-order interactions amongst the enormous predictors, remarkably, combing them in non-linear highly interactive ways [29]. Recently, ML approaches have opened up vast possibilities in emergency medicine—e.g., cardiac complications in patients with acute chest pain [30], cardiac arrest in ED patients [19, 20], and an ED triage tool for all adults patients [22] or children [21]. The present study confirms that ML models can attain a superior predictive ability for critical care outcomes in patients with acute chest pain. While over-fitting, often generates spurious correlations in the data, we were seriously concerned and thus, adopted multiple rigorous approaches to mitigate, regularize, and validate the independent cohort. Consequently, the performance of the validation model exceeded that of the development model. Decision curve analysis was used to evaluate the feasibility of the proposed model, the result demonstrated that we need to have an optimum balance between under-prediction and over-prediction. The ML model, used in the present study, enables correct identification of the critically ill patients, which might be inappropriately under-triaged into the lower-risk by the HEART, GRACE, TIMI score. Similarly, our model could rule out stable patient which would be over-triaged into high risk patient with those three reference scores, and thus may require additional resources. This finding supports the generalizability of the LASSO regression model. Moreover, this model can be employed directly during the bedside rounds.

Finally, due to the wide usage of HEART, GRACE, and TIMI scores and specifically recognized ability to stratify the patients with cardiovascular system, it may be the best reference models for the present study. We found that our LASSO regression model outperformed to these three well-known scores, and all of these showed a low discrimination index (AUC). Moreover, HEART score was slightly higher than the other two clinical risk scores in detecting the critically ill patients with chest pain in the testing set. To our knowledge, this is the first study to use HEART, GRACE, and TIMI scores in predicting critical care outcomes with the patient of chest pain. We speculate that the poor performance of these three risk scores might be attributed to the indicator of ECG test result they included, such as ST-elevation or ST-depression. There is a time window of clinical deterioration in patients with chest pain. Riley et al. found that 11% of patients with ST-elevation myocardial infarction (STEMI) had an initial non-diagnostic ECG with median time of 72 min between symptom onset and first medical contact, while 72.4% of these patients had an elevation of ST segment after 90 min [31]. Likewise, in the present study, these three scores was calculated only by applying the clinical data at triage, where some predictors remained normal. Several studies [15, 16, 18, 32] have previously compared the prognostic value of different risk scores for predicting MACE among chest pain patients, which consistently favored HEART score over other clinical risk scores in stratifying high-risk patients regarding the onset of MACE. However, in the present study, this prominence was not obviously evident due to the heterogeneity of the study populations, because more severe patients (high risk patients) with Level 1 and Level 2 chest pain who received treatment in the first aid room were included. A meta-analysis demonstrated that the discrimination accuracy of HEART score for the low-risk group was significantly higher than that of the high-risk group [17]. Irrespective of cause, our ML model resulted in an outstanding performance in terms of the AUC value, results of confusion matrix, and the assessment of clinical use.

The current study had several potential limitations. Firstly, this was a single-center study performed at a tertiary provincial emergency center in China, thus the institutional factors and potential selection bias might have resulted in findings that are less generalizable. Secondly, it is a fundamental case-control study of medical history data, with an intrinsic limitation in precision that entails. Nevertheless, most of the events contemplated are concrete and were truly recorded in the electronic medical records system. Thirdly, though the proposed model has demonstrated perfect performance in internal validity, there is a need for external validation of the scores for routine clinical use. Fourthly, we did not capture other clinical features, such as heart rate variability (HRV). The HRV has been regarded as a promising predictor that is recognized to have a significant relationship between the autonomic nervous system and cardiovascular mortality [30, 33‐35]. Due to the complicated estimation, time consuming procedure, and unsuitability with non-sinus rhythm [35], HRV has not been widely used clinically, especially in the developing country, and thus, was not included in the present study. Finally, the indication and clinical threshold of ICU admission vary depending on the local healthcare resource, such as ICU transfer criteria, ICU bed availability, and the ratio of nurse/patient and nurse/doctor.

To conclude, based on the ML model, we proposed a visualized LASSO regression model using 7 routinely captured clinical features. Compared to well-known clinical risk score—HEART, GRACE and TIMI score, our model had a superior performance in predicting the critical care outcomes in patients with chest pain. Moreover, the model minimized the potential over-predicted and under-predicted critical care outcomes that could result in excessive resource allocation to low-risk patients and insufficient treatment of high-risk patients. While external validation remains essential, the present study may pave the way for the application of ML-based predication models in critically ill patient with chest pain, as a decision-making technological tool.