Real-time AI prediction for major adverse cardiac events in emergency department patients with chest pain

Chest pain is one of the most common complaints that patients present with in the emergency department (ED) and accounts for 5 to 20% of all ED visits [1]. The causes of chest pain range from myalgia to potentially life-threatening diseases, such as acute coronary syndrome (ACS), aortic dissection, or pulmonary embolism [2]. Therefore, ED physicians often face a challenge when they need to decide on a diagnosis and disposition. Meanwhile, ED physicians may arrange a series of cardiac evaluations or an imaging study to exclude the list of differential diagnoses or even liberally hospitalize patients. However, < 10% of ED patients with chest pain are eventually diagnosed with ACS, and the liberal use of serial studies and hospitalization results in a high medical cost [3]. Therefore, developing a useful clinical prediction rule to help ED physicians with decision-making becomes an important issue.

The HEART (History, Electrocardiography, Age, Risk Factors, Troponin) score was created in 2008 to facilitate accurate diagnostic and therapeutic choices in ED patients with chest pain [4]. The original risk stratifications and suggested dispositions according to HEART score are as follows: (1) 0–3 points (2.5% for major adverse cardiac events [MACE]): an immediate discharge; 4–6 points (20.3% for MACE): implies hospitalization for clinical observation; and (3) ≥7 points: (72.7% for MACE): early invasive strategies [4]. The HEART score has the advantages of high accuracy and has been well-validated in many studies [5‐7]. The disadvantage of the HEART score is that it takes time to calculate for prediction; even a calculator is available online [8], which limits its use in the over-crowded ED. There are other scores (e.g., EDACS, T-MACS, and Vancouver), which do not take substantial time to calculate [9‐11]. They also force the acquisition of important data that may not otherwise be acquired. In recent years, artificial intelligence (AI), including machine learning (ML) techniques, has been used more and more often for developing prediction models and risk stratification in the ED [12, 13]. Many studies have shown that ML outperforms traditional metrics [12, 13]. In addition, ML has the advantage of handling more variables that are already available through electronic medical records (EMRs) and providing real-time feedback in terms of risk stratification to physicians if implemented in the hospital information system (HIS) [12, 13]. However, the issue about utility of ML integration with the HIS for predicting MACE in ED patients with chest pain is still unclear. We did not find any studies on this issue by searching for “acute myocardial infarction,” “chest pain,” “emergency department,” “death,” “machine learning,” “major adverse cardiac events,” and “mortality” in PubMed and Google Scholar. Therefore, we conducted this study intending to clarify this issue.

In total, 85,254 ED patients with chest pain in three hospitals between 2009 and 2018 were identified into the present study. The mean ± standard deviation (SD) age was 57.7 ± 17.9 years, and the proportion of males was 55.3% (Table 1). The proportion of the three age subgroups were age 20–34 (12.8%), age 35–49 (19.5%), age 50–64 (30.0%), and age ≥ 65 (37.7%). The proportion of smokers was 21.8% and mean ± SD of BMI was 24.8 ± 3.7. The medical histories of ED patients included hypertension (47.6%), hyperlipidemia (21.1%), diabetes (26.2%), chronic kidney disease (8.2%), coronary artery disease (38.3%), cerebrovascular disease (15.1%), and peripheral arterial occlusion disease (3.6%). The mean ± SD of high sensitive troponin-I, hemoglobin, and serum creatinine were 113.9 ± 2603.2 pg/mL, 13.1 ± 2.1 mg/dL, and 1.4 ± 2.4 mg/dL, respectively. AMI < 1 month was 20.3%, and all-cause mortality < 1 month was 0.3%.

The random forest model with the SMOTE preprocessing algorithm showed that the AUCs for predicting AMI < 1 month and all-cause mortality < 1 month were 0.915 and 0.999 (Table 2). Tree-based estimators were used to compute the feature importance value, which helps to discard irrelevant features [15]. In our random forest model, we included all 14 feature variables based on the feature importance values (Supplement Figure 1 and Supplement Figure 2) and expert judgment. Comparisons of predictive accuracies among the random forest, logistic regression, SVC, and KNN revealed that the random forest model had the best AUC, accuracy, precision, sensitivity, specificity, and F1 of all the models (Table 3). In the prediction of AMI < 1 month, SVC has the poorest AUC of 0.631. Logistic regression has the poorest AUC of 0.716 in the prediction of all-cause mortality < 1 month. We integrated the AI prediction model with the HIS to assist the ED physicians in real-time decision-making. The time taken to generate prediction results is < 1 s when the ED physician pushes the AI button (Supplementary Figure 3).

We identified 3741 new ED patients with chest pain between Jan 1, 2019 and Oct 31, 2019 to validate the AI prediction model (Table 4). The AUCs of AMI < 1 month and all-cause mortality < 1 month were 0.907 and 0.888, respectively. Between June 1, 2019 and May 31, 2020, the use rate of AI prediction model was 12.4% with a satisfaction of 4.98 ± 0.15 by 5-point Likert scale.

We used a big-data-driven approach, ML, and integration with HIS to build a real-time prediction for MACE in ED patients with chest pain. In the prospective validation, the AI prediction model has an AUC of 0.907 for AMI < 1 month and an AUC of 0.888 for all-cause mortality < 1 month. Compared with other ML algorithms, the random forest model had better accuracy than logistic regression, SVC, and KNN. Validation of the AI prediction model using new patients also showed excellent accuracy.

Risk stratification and disposition of ED patients with chest pain is always a difficult challenge for ED physicians, especially in overcrowding situations [6, 7, 16]. A missed diagnosis of AMI is a nightmare for ED physicians; therefore, developing a clinical prediction rule becomes a useful method to assist with decision-making and disposition [16]. The Thrombolysis In Myocardial Infarction (TIMI) score was originally developed to predict adverse cardiac outcomes (death, [re]infarction, or recurrent severe ischemia requiring revascularization) within 14 days of presentation for patients with unstable angina or non–ST-segment-elevation myocardial infarction [17]. The TIMI score was also used to predict MACE in ED patients with chest pain [18]. Another famous tool, the GRACE score, was also developed in patients already diagnosed with AMI [19]. In contrast to the TIMI and GRACE scores, the HEART score was developed more specifically for rapid risk stratification and disposition in ED patients with chest pain [4]. The HEART score has the benefits of ease of recall and use, ready availability of predictors, a focus on short-term outcomes, specificity for ED management, and identification of three risk groups [20]. Compared with the TIMI and GRACE scores, the HEART score performed better to discriminate MACE in ED patients with chest pain [21].

The real-time AI prediction model in the present study has good accuracy for risk stratification in ED patients with chest pain. In addition, its features of automatic and rapid capture of the predictors from EMRs and improving ML algorithms provide ED physicians with new hope for improving care in this population. The random forest model adopted in the present study is a common algorithm for predicting outcomes and selecting predictors in the ED [13, 22]. A study reported that ML outperformed CURB-65, MEDS, and mREMS as well as traditional analytical techniques for predicting in-hospital mortality in ED patients with sepsis [22]. Another study reported that ML-based variable selection is promising method of discovering relevant and significant predictors of MACE in ED patients with chest pain [13]. In addition to the random forest model, we used the SMOTE algorithm, an oversampling approach, to adjust imbalanced data in the present study [23‐25]. In the SMOTE algorithm, the minority group is over-sampled by creating synthetic examples [23‐25]. The SMOTE is suggested to be better than the under-sampling approach, which has the drawback of disregarding potentially useful data [23‐25].

Implementation of AI prediction model in healthcare raises many new issues, including malpractice liability, insurance coverage, and EMR platforms. Legal issues include technology manufacturers’ and health care professionals’ liability, particularly if they can’t explain the suggestions generated by AI prediction model [26]. New legal doctrine is necessary for AI-related medical malpractice [26]. For healthcare insurance, a clear and effective data governance framework is critical [27]. In the design and use of EMR and AI, legal standards need to be enacted and insurance company should be encouraged to adopt a human-centered approach [27]. Regulators and policy-makers and insurance company need to work together to ensure that the big-data driven AI are transparent and accurate [27].

The major strength of this study is that we developed the first real-time AI prediction model integrated with the HIS for predicting MACE in ED patients with chest pain. Another strength was that we identified new patients to validate the AI prediction model. The limitations are as follows. First, we did not include electrocardiography in the analyzes due to the difficultly of interpreting unstructured data. However, our AI prediction model using structured data only performed well. Further studies by including electrocardiography may be needed to evaluate the additional value. Second, ML has the problems of interpretability and inferences about variables [28]. However, we believe that the random forests used in this study provide for good interpretability through the feature importance plots and the individual trees. Third, some data were not available due to the retrospective nature of this study. We used expert consultation to decide how to manage missing values. Fourth, we did not evaluate the impact of the AI prediction model on clinical practice. Although it is beyond the scope of this study, it may be necessary to evaluate the impact on physicians’ decisions, including acceptance and rejection, skepticism, concerns, change their clinical practices, and patient outcomes. Fifth, we used ICD-9-CM of chest pain to recruit the patients into this study, which may exclude patients who presented with chest pain but had, for instance, AMI. However, patients with immediate diagnosis of AMI are not the target population for the AI prediction model (e.g., significant ST elevation on the electrocardiography). The main difficulty is evaluating the patients with equivocal finding in the initial presentation. Therefore, the AI prediction model may play the important role to assist ED physicians for making decision in such a circumstance. Sixth, the AI prediction model may not be generalizable to other hospitals. However, this study was supposed to be a proof of concept of using AI for this purpose. We also suggest the same algorithm might be used at other institutions. Re-training and testing in other hospitals are suggested to overcome this issue.

We integrated the first real-time AI prediction model with the HIS to predict short-term MACE in ED patients with chest pain. The AI prediction model has excellent accuracy and a rapid response for assisting ED physicians in decisions and disposition. Validation on new patients also showed excellent performance. Further studies about the impact of an AI prediction model on clinical practice and outcomes, as well as including electrocardiography in the analyzes, may be needed.