Background
Chest pain and related symptoms are the most common reasons for patients to present to the emergency department (ED) [
1,
2], and present extremely heterogeneous with a wide spectrum of underlying conditions ranging from lethal diseases such as acute myocardial infarction (AMI) to minor acute problems such as intercostal neuralgia. Ruling in or ruling out high-risk conditions in a timely manner is of great importance and a great challenge [
3‐
6]. Furthermore, the majority of undifferentiated acute chest pain patients are low risk and do not require further invasive tests or admission [
6,
7]. Therefore, risk stratification for chest pain patients at EDs has been recommended in several guidelines [
6,
8] to not only identify as many true low-risk patients as possible but also avoid missing major adverse cardiac events (MACEs).
The Global Registry of Acute Coronary Events (GRACE) score is an objective prediction tool for definite acute coronary syndrome (ACS), incorporating age, vital signs, kidney function, ECG and troponin levels [
9]. This tool has been validated for risk stratification of individuals with acute chest pain [
10‐
17]. In particular, the 0 h/3 h algorithm with the GRACE score incorporated into is recommended (Class I, Level B) for risk stratification and rule-out of AMI in patients with suspected non-ST-elevation ACS by 2015 European Society of Cardiology (ESC) guideline [
6].
However, there are six formulas used to calculate the GRACE score for different outcomes, including those for predicting in-hospital death [
9], in-hospital death or myocardial infarction (MI) [
18], death within 6 months after discharge [
19], death or MI within 6 months after discharge, death from admission to 6 months later [
20], and death or MI from admission to 6 months later [
20]. None of these formulas are specific for rule-out/rule-in of high-risk conditions in patients with undifferentiated chest pain presenting to the ED. The GRACE models have been compared with other scores for stratifying undifferentiated chest pain, such as the History, ECG, Age, Risk Factors, and Troponin (HEART) score and the Thrombolysis in Myocardial Infarction (TIMI) score [
21,
22]. Generally, GRACE was inferior to the HEART score, and the most common GRACE score applied was the one for predicting in-hospital death [
11‐
16]. The questions of why this model is selected and whether it is the most appropriate one remain unanswered. No study has comprehensively assessed these scores in detail in chest pain patients. Therefore, the superiority of certain GRACE scores remains unclear.
Using a range of model performance indices, we aimed to evaluate the performance of six GRACE models and compare their discrimination, reclassification and diagnostic accuracy with those of HEART and TIMI scores to rule out/rule in 30-day MACEs among acute chest pain patients presenting to the ED.
Discussion
This study provides the first comprehensive evaluation and comparison of all six GRACE risk-prediction models in patients with undifferentiated chest pain. In the two Chinese EDs included in this study, all six GRACEs showed a positive linear correlation with actual MACE rates, and the five models had good calibration. All the C-statistics were ≥ 0.70. The GRACE (IHDthMI) and GRACE (IH6mDthMI) exhibited very strong relationships with actual MACE rates (r > 0.9) and showed excellent discriminatory capability (AUC > 0.80). Improvements in AUC, NRI and IDI indicated that GRACE (IHDthMI) and GRACE (IH6mDthMI) were comparable to the HEART score and superior to the other models.
The GRACE risk scores were developed using multivariable regression to assist cardiologists in estimating the risk of different outcomes in hospitalized patients with ACS and have been indicated to provide the most accurate stratification of risk of ACS both on admission and at discharge [
31,
32]. One model is specific to one kind of outcome, including death or composite of MI and death during hospitalization, within 6 months after discharge and from admission to 6 months later. The MI referred to here is the subsequent AMI occurring after the index ACS. However, for undifferentiated chest pain, the high-risk conditions mainly present a composite endpoint of index AMI, subsequent AMI, death, emergency revascularization, cardiac arrest and cardiogenic shock within 30 days after presentation to the ED [
33]. The incidence of index AMI is much greater than that of subsequent AMI, as shown in our study. Our results suggested that the GRACE models showed at least a moderate correlation with the actual incidence of MACEs in the undifferentiated chest pain cohort. In particular, very strong correlations appeared in GRACE (IHDthMI) and GRACE (IH6mDthMI). Furthermore, the predicted probabilities of an event and the observed event rates were significantly similar across deciles of five GRACE models. Therefore, there are foundations for the GRACE models to provide accurate stratification of patients with acute chest pain.
In previous studies, C-statistics for predicting 30-day MACEs in chest pain patients were merely evaluated according to the GRACE (IHDth) with AUCs of 0.60 to 0.83, which were always inferior to those of the HEART score [
10‐
16]. Consistently, we found that the AUC of the GRACE (IHDth) was 0.75 (0.73, 0.76), which was actually lower than that of the HEART score in this study. However, GRACE (IHDth) was neither the only GRACE model nor the best GRACE model for stratifying chest pain. The GRACE (IHDthMI) and GRACE (IH6mDthMI) had better total discriminatory capability (AUC > 0.8) and reclassification without difference from the HEART score. Although the performance of all these models was not good in the rural hospital as in the urban hospital, the advantages of GRACE (IHDthMI), GRACE (IH6mDthMI) and HEART were consistent in both EDs. Significantly positive NRI and IDI in this study showed that the GRACE (IHDthMI) and GRACE (IH6mDthMI) could provide a higher predicted probability of an event for high-risk patients and a lower predicted probability for low-risk patients than the other four models. The possible explanation may be that events predicted by these two GRACE models referred to attacks of AMI rather than merely death, though not the index AMI. Compared with the models for events after discharge, the periods in the hospital or from admission to 6 months later were closer to the 30-day follow-up after presentation to the ED. Our results did not refute previous conclusions but complemented them by providing more complete recognition of the GRACE models.
Exact cutoff values should be determined for clinical use to identify low-risk patients for safe and early discharge without compromising the immediate treatment of high-risk chest pain. Reaney et al. found that GRACE (IHDth) 0–55 could reach a sensitivity of 95.2% and NPV of 95.8%, identifying 21.2% patients as low risk. GRACE (IHDth) ≥119 defined 16% of patients as high risk (specificity 89.8%; PPV 48.1%) [
16]. Poldervaart et al. determined GRACE (IHDth) ≤72 as the cutoff and 19.1% patients were classified as low-risk (sensitivity 95%; NPV 96%) [
14]. Cullen et al. chose the cutoff of GRACE (OH6mDth) ≤50 to determine low-risk (24% patients) with a sensitivity of 98.9%, and the cutoff for recognizing high risk (28% patients) was ≥100, with a specificity of 76.2% [
17]. In our study, the performance of GRACE (IHDth) was relatively consistent with that of previous studies, with a value of ≤79 identifying 18% patients as low risk (sensitivity 95.1%; NPV 94.4%) and a value of > 145 defining 16% patients as high risk (specificity 90.0%; PPV 49.9%). At the same sensitivity and specificity, the GRACE (IHDthMI) and GRACE (IH6mDthMI) outperformed the GRACE (IHDth) and other GRACEs. Although there is no rigorous standard for the sensitivity of risk-stratification models for chest pain, an international survey suggested that clinicians may expect a sensitivity of 99% or higher for AMI or other MACEs [
34]. If the sensitivity was set at ≥99%, GRACE (IHDthMI) and GRACE (IH6mDthMI) were still superior, but the proportions of low-risk patients would drop below 10%. A meta-analysis demonstrated that the pooled sensitivity and specificity of a HEART score ≤ 3 for predicting MACEs were 96.7% (94.0, 98.2%) and 47.0% (41.0, 53.5%), respectively [
35]. HEART≤3 in our cohort had a similar sensitivity (96.8%) but a lower specificity (27.0%). The sensitivity of HEART ≤2 was higher at 98.8%(97.9, 99.7%) at the cost of a lower proportion (11%) of patients identified as low risk. In our previous report, the HEART score would not appear to provide additionally helpful risk stratification to the usual care for discharging low-risk patients [
36]. Regarding the high-risk category, HEART ≥7 did not perform as well (specificity 87.0%; PPV 53.4%) as in Reaney’s study [
16].
For ruling out and ruling in MACEs, the HEART score illustrated a certain advantage over the GRACE (IHDth) but not the GRACE (IHDthMI) or GRACE (IH6mDthMI). The strengths of GRACE are still noteworthy. Possible explanations might be the detailed class and objectivity of components of the GRACE beyond HEART and TIMI. Although the HEART score was directly developed for undifferentiated patients, the assignment of every variable only included three qualitative classes (i.e., 0,1,2) [
21]. The classes for each component of TIMI score are even lesser (only 0 or 1) [
22]. In contrast, the GRACE scores included many more quantitative variables, such as age, SBP, pulse and creatinine, which are supposed to identify subtler differences and result in more exact stratification. As highlighted by the 0 h/1 h algorithm from the ESC guideline recommendations, quantitative interpretation overcomes the qualitative interpretation of high-sensitivity troponin levels for ruling out and ruling in AMI in chest pain patients, and the cutoff levels are assay specific [
6,
37]. Furthermore, some “soft” variables are included in the HEART score, such as the medical history, risk factors and symptoms. It has been shown that these variables do not have a sufficient discriminatory ability to rule in or rule out ACS in the ED [
38]. The combination of symptom variables as a “history” component in the HEART score was still not clearly stated and not assessed systematically [
39]. The GRACE score can avoid this situation due to the absence of subjective variables. The popularity of handheld devices has made the complexity of GRACE no longer a disadvantage.
The results from the assessment of mini-GRACE were mainly in accordance with those of the complete models. Although the correlation of the mini-GRACE (IHDthMI) and the calibration of the mini-GRACE (IH6mDthMI) were lower than the complete models, the discrimination and reclassification of these two mini scores remained excellent and significantly outperformed other models. This illustrates that the differences in model performance may be due to the disparities in weights of shared variables.
Limitations
This study had several limitations. First, the performance of different GRACE scores was assessed in chest pain patients from two hospitals in China. Although urban and rural hospitals were both included, the validation of each score in wider patients should be determined by further studies of heterogeneous groups. In particular, the cutoff levels of the GRACEs are not the same in different studies due to the disparity of inclusion and exclusion criteria and the incidence and definition of MACEs. Determination and validation of the specific cutoff values in clinical practice in certain hospitals are needed. Second, the cardiac marker used in the calculation of scores was the contemporary cTn assay arranged by emergency physicians in their daily work. The ability of scores combined with high-sensitivity cTns to stratify chest pain still needs to be evaluated in future studies. Third, all components used in the risk scores were calculated automatically through a computer algorithm. The ECG variables were based on the standard interpretation from senior cardiologists. This calculation process deviated from clinical reality. Further studies to evaluate the discrimination of scores calculated immediately by the treating physicians are needed.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.