A simple statistical model for prediction of acute coronary syndrome in chest pain patients in the emergency department

Lund University Hospital, Sweden, is a 1200 bed institution which serves as the primary hospital for some 250,000 inhabitants, has a cardiac intensive care unit with 19 beds and an intermediate care ward with ECG monitoring at 19 beds. Percutaneous coronary intervention (PCI) and coronary bypass surgery (CABG) are available 24 hours/day. There is a traditional ED with approximately 50000 patients per year. During the patient inclusion period, there was no systematic diagnostic protocol for patients with suspected ACS, and no dedicated chest pain unit.

Six hundred and sixty-five consecutive visits for chest pain for which electronic ECG data could be retrieved were retrospectively included at the ED of Lund University Hospital from July 1 to November 20, 1997. Characteristics of the 634 unique patients are presented in table 1. From the clinical data collected for each patient, 18 variables available immediately at ED presentation (table 1) were chosen for further study based on their likely importance for ACS prediction.

The ED visit ended in hospitalization in 422 (67%) cases, and among these a discharge diagnosis of ACS was assigned in only 130 (31%), of which 28 (22%) had Q-wave AMI, 53 (41%) had non-Q-wave AMI and 49 (38%) unstable angina. Discharge diagnoses were made by the senior ward physician or the ED physician (in cases discharged from the ED), reviewed by a senior research nurse, and when ambiguous, further reviewed by a senior cardiologist (HÖ). In the review of diagnoses for cases discharged from the ED, available data from the patient records indicated that the rate of missed diagnosis of ACS was low (not more than 2%). AMI was defined by the WHO criteria [25] where the biochemical criterion was at least one measurement of CK-MB>10 μg/l or Troponin T>0.1 μg/l. The criteria for unstable angina were ischemic symptoms (chest pain >15 min., syncope, acute heart failure or pulmonary edema) together with at least one of the following: a) Electrocardiogram (ECG) changes: transient or persisting ST segment depression (≥ 1 mm) and/or T-wave inversion (≥ 1 mm) without developing Q waves or loss of R wave height, or b) Biochemical markers: CK-MB 5–10 μg/l or Troponin T 0.05–0.1 μg/l.

This study was approved by the Lund University Research Ethics Committee.

Two methods of ECG analysis were used, machine reading and expert reading.

The statistical analyses were conducted using SPSS release 12.0.1 (SPSS Inc, Chicago, U.S.). Odds ratios (OR) were calculated for the association between each potential risk factor and ACS. We regarded 95% confidence intervals (CIs) that excluded unity, or, equivalently, p < 0.05, as statistically significant. In the multivariable analysis, the probability of ACS was predicted using multiple logistic regression [27]. All independent variables except age were entered in the regression models as categorical variables, categorized as shown in tables 1, 2. In order to obtain meaningful baseline odds for ACS in the models, we chose 40 years old as reference value for age. Thus, we used the number of years above 40 as age variable in the models and allowed for both positive (patients above 40) and negative (patients below 40) numbers. Starting with the full multivariable model with all independent variables included, we excluded one insignificant independent variable at a time, starting with the variable with highest p-value, until only significant and important predictors remained. Categorical variables with more than two categories were kept in the model if the OR associated with any of the categories was significant. Variables with estimated OR of at least 2.5 (or, equivalently, at most 0.4) were considered as important predictors and thus kept in the model even if they were not statistically significant. The ECG variables were not considered for exclusion. For the final set of independent variables, categories with ORs close to one were collapsed with the reference category for that variable. Unknown response to one of the variables (hypertension) was also added to the reference category. We then assessed the significance of the interaction with age, implemented as cross-product terms together with main effects, for the following variables: sex, chest discomfort at presentation, symptom duration, and previous myocardial infarction. For comparison, we also established a prediction model where we replaced the ST amplitude characteristics of the ECG with the consensus ECG assessments by the two experts (LE and HÖ). Differences in specificity between the two models were tested using McNemar's test for correlated proportions.

The area under the receiver-operating-characteristic curve (ROC) was used as an overall measure of the discrimination abilities of the prediction models [28]. The area under ROC, measured in percent, can be interpreted as the probability that a randomly chosen patient with ACS has a higher outcome probability than a randomly chosen patient without ACS. For calculation of specificity and predictive values, the sensitivity was set to 95%. This somewhat arbitrary level was chosen because with current standard evaluation, some 2–5% of the ACS patients are erroneously discharged from the ED [29, 30], which implies a sensitivity of at least 95 % for the routine ED work-up. In order to get more correct estimates of the generalization performance, i.e. the performance of the prediction models on an unseen study population, we used a k-fold cross-validation procedure [31]. Here we repeated times estimated the regression parameters based on randomly chosen training sets, comprising approximately 80% of all patients, and then evaluated the obtained model by calculating the area under ROC as well as the specificity at 95% sensitivity among the remaining 20% patients in a validation set kept completely isolated from the corresponding training set. This k-fold cross-validation procedure was implemented as follows: 1) the data set was randomly split in 5 groups of approximately equal size 2) the parameters of each model were estimated based on the patients of the training set only, which was established by excluding group k [k = 1, 2, ...., 5] 3) the group left out from parameter estimation was used as a validation set. We repeated the steps 1–3 above 20 times, which implied 20 * 5 = 100 sets of validations sets, 100 areas under ROC, and 100 estimates of the specificity at 95% sensitivity. Median values, together with 2.5 – 97.5 percentiles, from these 100 validation sets were used as measures of the generalization performance of the prediction models.

The ECG assessments made in consensus by the two experts were strong predictors of ACS (table 5), yielding an area under ROC of 76.1% with no other variables included in the prediction model. Adding other clinical data yielded markedly higher discriminating percentage (area under ROC 88.0%; table 6 and figure 2). For this model, a sensitivity of 95% would yield a specificity of 50%, which at an ACS-prevalence of 21% would produce a NPV of 98% and a PPV of 33%. The specificity in this model (50%) was significantly higher (p = 0.001) than in the model based on ECG amplitude data (43%). The discriminating percentage increased only marginally when the interaction terms related to age were incorporated (area under ROC 88.3%; regression model and ROC curve not shown).

Cross-validating the model including clinical and ECG amplitude data decreased the area under ROC from 80.6% to 76.8% in median (2.5 – 97.5 percentiles 69.1 – 84.1%), and lowered the specificity from 43% to 35% in median (2.5 – 97.5 percentiles 13 – 60%) if the sensitivity was set to 95%. This median specificity at 95% sensitivity would, with a prevalence of ACS of 21%, diminish the NPV from 97% to 96% (2.5 – 97.5 percentiles 91 – 98%) and the PPV from 30% to 27% (2.5 – 97.5 percentiles 22 – 38%). Cross-validating the corresponding model based on expert ECG assessments lowered the area under ROC from 88.0% to 85.7% in median (2.5 – 97.5 percentiles 78.5 – 93.0%). By contrast, the specificity at 95% sensitivity was unchanged (50% in median, 2.5 – 97.5 percentiles 21 – 74%), yielding the same NPV (98% in median; 2.5 – 97.5 percentiles 94 – 98%) and the same PPV (33% in median; 2.5 – 97.5 percentiles 24 – 49%) as before.

The patients included in the present model were retrospectively collected and from one center only. Because of the retrospective nature of the study, there may of course be errors despite a careful collection of all data. For example, the diagnoses of the patients discharged from the ED were not tested with routine post-discharge ECG or blood samples for cardiac markers. Therefore, before clinical implementation, the model clearly needs to be validated prospectively, preferably at multiple centers. An old definition of AMI was used in the present study. Newer definitions of AMI have lower cut-off values for biochemical markers, and some of the unstable angina diagnoses in this study would currently be classified as AMI [37]. However, the total number of ACS cases would probably be little changed. Furthermore, the baseline risk (odds) for ACS of the model has to be tuned in each population before implementation. Since it is impossible to analyze all information with a potential impact on the likelihood of ACS, it is of course possible that variables in addition to those included in our model could be important for ACS prediction. The size of the sample might also have limited our abilities to detect clinical or ECG characteristics of low prevalence that nevertheless are important for ACS classification.