The HEART score in predicting major adverse cardiac events in patients presenting to the emergency department with possible acute coronary syndrome: protocol for a systematic review and meta-analysis

In any diagnostic accuracy study, the index test (test under evaluation) and reference standard (best available method for establishing presence or absence of target condition) should be described in sufficient detail to permit replication [23]. In this review, HEART score 0–3 represented a negative index test and HEART score 4–10 represented a positive index test in calculating measures of diagnostic accuracy (see Table 3). This cut-off was selected as the authors considered patients with HEART score 0–3 at low risk of developing MACE and potentially eligible for immediate discharge from the ED.

The authors defined the reference standard as follows:

The third universal definition of AMI, consistent with a rise and/or fall of a cardiac biomarker with minimally one result above the 99th percentile upper reference limit in the context of a patient presenting with cardiac ischaemia.

The index test and reference standard descriptions illustrate the basic diagnostic accuracy study design eligible for inclusion in this review (see Fig. 1).

The rationale for choosing this reference standard was not provided. If one chooses to conceptualise the HEART score as a diagnostic test, the passage of time seemingly represents a more reliable standard for detecting MACE. The authors suggest that to diagnose or predict future MACE, the reference standard is the “third universal definition of AMI”. This is problematic because the diagnosis of MI by a rise and/or fall of troponin is also, by definition, a MACE. This diagnostic accuracy systematic review’s reference standard is thus also part of its target condition.

The reference standard also introduces incorporation and verification biases [24]. Troponin levels are components of both the HEART score and reference standard. This incorporation bias elevates the risk of overstating both the sensitivity and specificity of the HEART score. Experience and clinical practice guidelines suggest patients with a history concerning for ACS, an abnormal ECG, advanced age, or many risk factors for coronary artery disease are very likely to have multiple troponin levels measured while in the ED [1]. These patients will also have a high HEART score. When multiple troponin levels are measured over a period of ED observation, a patient is more likely to have a positive result, be diagnosed with AMI, and subsequently undergo revascularisation by PCI or CABG (see Fig. 2). This verification bias similarly risks overstating both the sensitivity and specificity of the HEART score. A false-negative HEART score 0–3 is less likely to occur if only one troponin measurement occurs. Likewise, a false-positive HEART score 4–10 is less likely when multiple troponin measurements occur. The impact of these biases could have been explored via a sensitivity analysis comparing trials standardising an observation period or multiple troponin levels for all participants irrespective of HEART score (lower risk of bias) to those trials entrusting that decision to the discretion of the treating physician (higher risk of bias).

Several large studies have been published beyond the search window of the May 2017 review, including at least one randomised trial and multiple prospective observational studies [17, 25‐27]. Nine of the 11 studies included in this review were retrospective in nature. In general, retrospective studies of the HEART score have a higher risk of bias [28]. For example, lack of outcome blinding combined with unclear predictor measurement criteria might encourage an assessor to rate the history as “highly suspicious” if a patient was known to have sustained a MACE. This bias likely overestimates the predictive value of the HEART score and should be explicitly addressed in any systematic review. The importance of this bias could have been evaluated by comparing trials with clearly stated outcome blinding (lower risk of bias) to those lacking outcome blinding (higher risk of bias) via a sensitivity analysis. Alternatively, the authors could have made trials without outcome blinding ineligible for inclusion in the review.

The external validity of many observational studies of the HEART score is also of concern. For example, it is debatable whether history and ECG scoring by a researcher or cardiologist reviewing patient charts is reliable as a proxy of how the history and ECG would have been scored by the attending EP. In the context of this review, the strongest study design to evaluate the utility of the HEART score as a predictor of MACE would be a prospective observational study mandating the attending clinician at the time of initial ED assessment determine the HEART score. Studies incorporating this design should be emphasised in a prognostic systematic review. Similarly, across studies included in the May 2017 systematic review, there are potentially important differences in the baseline risk of MACE (range 7.3–29.1%) and type of troponin assays used [22]. The reviewers could have better assessed the impact of clinical heterogeneity in study populations and HEART score measurement by subgroup analyses.

Original research articles on the external validation of the HEART score will be eligible for this review. Derivation and interval validation studies will be excluded. Randomised controlled trials, prospective observational studies, and retrospective observational studies will be considered for inclusion. For a retrospective study to be included, assessors of the patient history and ECG must be blinded to the outcome of MACE. If this is not clearly stated in the study’s methodology, authors will be contacted to clarify if assessors were suitably blinded. If uncertainty regarding blinding persists following an attempt to contact study authors, the study will be excluded from this review. There will be no restrictions on language or timeframe of publication.

Studies evaluating patients with possible ACS (see Table 4) at initial physician assessment in an ED or chest pain unit will be considered for inclusion. Chest pain units exist in some, but not all jurisdictions. These units are typically located in or near the ED and provide a setting for observation and investigation of acute chest pain patients [30]. As our population of interest includes all patients presenting to an ED with possible ACS, studies evaluating only a subset of the population of interest (e.g. only patients with HEART score 0–3) will be excluded. Similarly, studies failing to include all patients who sustain a MACE while in the ED or chest pain unit will also be excluded. An exception will be made if a study excludes patients with definite STEMI or ACS at initial assessment as diagnostic uncertainty is lacking, and these patients are typically immediately transferred to the nearest cardiac catheterisation facility.

All patients must have a HEART score determined using data obtained at the initial physician assessment.

The types of comparisons are not applicable.

The primary outcome will be MACE within 6 weeks of initial ED or chest pain unit assessment. Major adverse cardiac events are a composite outcome encompassing death, MI, PCI, or CABG. The primary outcome will be stratified by HEART score 0–3 and HEART score 4–10.

The intent of the HEART score is to identify a low-risk group of patients that can be safely discharged from the ED. HEART score 0–3 represents the low-risk group as defined in the prediction model’s derivation study. The authors of the May 2017 review agreed with this stratification approach, thus identifying 4101 of 11,217 (36.6%) patients as low risk in their analysis [22]. The pooled prevalence of MACE in these 4101 patients was 1.6% (95% CI 1.2–2.0%). This seems a sensible risk level at which to initiate a shared clinician-patient decision on discharge versus continued hospital observation or investigation. While alternate stratification strategies may further lower the risk of MACE, this must be balanced by subjecting a higher proportion of patients to unnecessary observation and investigation.

Secondary outcomes will be death within 6 weeks of initial ED or chest pain unit assessment and MI within 6 weeks of initial ED or chest pain unit assessment. If a study does not include sufficient information to determine secondary outcomes, the authors will be contacted. If after attempting to contact study authors this information remains unavailable, the study will be excluded from secondary outcome analysis. Secondary outcomes will also be stratified by HEART score 0–3 and HEART score 4–10.

The electronic database search will include the following:

ClinicalTrials.​gov, the ISRCTN registry, the World Health Organization International Clinical Trials Registry Platform (ICTRP), and PROSPERO will be searched for unpublished and ongoing trials. Databases will be searched using the free text terms “HEART score” or “HEART pathway” without field, language, or date of publication limitations. The search strategy with its corresponding preliminary results is provided in Additional file 1: Appendix S1.

Reference lists for primary studies eligible for inclusion and review articles will be scrutinised to identify potentially relevant citations. Conference proceedings from the Canadian Association of Emergency Physicians (CAEP), American College of Emergency Physicians (ACEP), Society for Academic Emergency Medicine (SAEM), and International Conference on Emergency Medicine (ICEM) will be hand-searched. The conference proceedings search will be restricted to start in 2008, the year the HEART score was first derived and published

Results from the search strategies will be combined into a reference manager programme with duplicates excluded. Two authors (CB, CT) will perform the title and abstract screening, excluding obviously ineligible studies. From the potentially relevant articles identified, two authors (CB, CT) will independently perform the full-text reviews and select trials for inclusion using a standardised article inclusion form (see Additional file 1: Appendix S2). Disagreements will be resolved by discussion to reach consensus or third-party adjudication (TH). A list of included trials will be completed (see Additional file 1: Appendix S3).

Two authors (CB, CT) will independently extract the data from the included studies using an electronic data extraction form (see Additional file 1: Appendix S4). Disagreement not resolved by consensus will be adjudicated by a third author (TH). The data extraction form is based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) [13], modified according to the scope of this review (i.e. external validation studies). This checklist provides a framework for extracting key information from studies of prediction models.

To assess the risk of bias within individual studies, the Quality in Prognosis Studies (QUIPS) tool for prognostic studies will be used [28]. Five bias domains (see Table 5) will be rated as having a low, moderate, or high risk of bias according to QUIPS tool criteria. Two independent reviewers (CB, CT) will assess every included study for bias. Disagreements will be resolved by discussion or third-party adjudication (TH). Results from the risk of bias assessment will be presented in table format with colour coding for easy visualisation.

Descriptive statistics will be presented as means with standard deviations (SD) and medians with interquartile ranges (IQR). Outcomes in this review are dichotomous and will be presented as pooled prevalences, Mantel-Haenszel risk ratios (RRs), and absolute risk reductions (ARRs) with 95% confidence intervals (CI). Prediction model performance will be summarised using measures of discrimination, calibration, and a decision curve analysis (DCA).

Discrimination refers to a prediction model’s ability to distinguish between patients developing and not developing the outcome [31]. In the context of this review’s primary outcome, the C-statistic provides the probability a randomly selected patient who experienced a MACE had a higher risk HEART score than a patient who had not experienced a MACE. The C-statistic is equal to the area under the receiver operating characteristic (ROC) curve.

Calibration refers to a model’s accuracy of predicted risk probabilities, indicating the extent to which expected and observed outcomes agree [31]. Summarising estimates of calibration is difficult because calibration plots are often not presented, and studies tend to report different types of summary statistics in calibration [32]. As a result, calibration will be quantified by individual study and summary observed to expected events ratios (O:E events ratios) with 95% confidence intervals stratified by HEART scores of 0–3 and 4–10. The summary O:E events ratio provides a rough indication of overall model calibration across the entire range of predicted risks [31].

Decision curve analysis (DCA) is a method for evaluating a prognostic tool with competing benefits and harms across a range of patient preferences and risk tolerances [33]. The decision to observe or continue to investigate a patient with possible ACS depends on how confident the clinician is in a patient’s prognosis, the efficacy and complications of additional observation or investigation, and the patient’s willingness to accept the burden of an observation or investigation plan that might be unnecessary. To address these clinical decision challenges, DCA uses a measure called the threshold probability [34].

In this review, the threshold probability represents the probability of MACE at which an individual believes the harms of unnecessary observation or investigation (e.g. coronary angiogram) if MACE will not occur are equal to the benefits of observation or investigation if MACE will occur (e.g. early recognition and treatment of MI or obstructive coronary artery disease by PCI or CABG). The probability threshold will vary from individual to individual. If additional observation or investigation is perceived by a patient to have high value and minimal morbidity, inconvenience, and cost, the threshold probability for proceeding with this management plan will be low. Conversely, if this plan is perceived to be minimally effective or associated with adverse effects, the threshold probability will be high [35]. Threshold probabilities thus provide a framework for processing benefits and harms into a patient-centred, evidence-based clinical decision.

This review’s DCA will compare the following approaches: (1) observe or investigate all patients presenting to ED with possible ACS regardless of HEART score or (2) observe or investigate only those patients with HEART score 4–10. In this DCA, the desirable outcome or “benefit” is additional observation or investigation for those patients who will have a MACE (true positives). The undesirable outcome or “harm” is additional observation or investigation for those patients who will not have a MACE (false positives). The net benefit was calculated by determining the difference between the expected benefit and the expected harm. The expected benefit is represented by the proportion of patients who will have a MACE and be observed or investigated (true-positive rate). The expected harm is represented by multiplying the false-positive rate by a weighting factor based on the patient’s threshold probability (see Fig. 3) [35].

For example, if a patient’s threshold probability of a MACE is 10%, the weighting factor applied to the proportion of patients observed or investigated who will not have a MACE would be 0.1/0.9, or one ninth. This minimises the effect of false-positive results because the burden of an unnecessary management plan is perceived by the patient to be low. Graphically, the DCA is expressed as a curve, with a net benefit score on the vertical axis and risk thresholds on the horizontal axis. This analysis is intended to help tailor clinical decision-making to an individual patient’s threshold probability.

Heterogeneity will be assessed by analysing methodological, clinical, and statistical diversity.

Methodological diversity will be judged primarily by the risk of bias assessment as per the QUIPS tool (see Table 5). Sensitivity analyses will be performed to assess the robustness of results after accounting for the impact of subjective methodological assumptions and inclusion of studies at high risk of bias (see the “Sensitivity analysis” section).

Anticipated sources of important clinical diversity include variable baseline risks of MACE, use of conventional/contemporary versus high-sensitivity troponin assays, and attending clinician versus researcher or non-attending clinician determined HEART scores. Subgroup analyses will be performed to explore the impact of clinical heterogeneity (see the “Subgroup analyses” section).

Statistical heterogeneity will be visually displayed in forest plots, with a poor overlap of confidence intervals for the results of individual studies indicating the presence of heterogeneity [36]. More formally, statistical heterogeneity will be assessed using the chi-squared (χ²) test and I² statistic. The χ² test assesses whether observed differences in the results are compatible with chance alone [36]. A low P value (or large χ² statistic) provides evidence of variation in effect estimates beyond chance [36]. The I² statistic quantifies the percentage of total variability in effect estimates that is due to heterogeneity rather than chance [36]. A rough guide to interpretation of the I² statistic is as follows: 0 to 40% might not be important, 30 to 60% may represent moderate heterogeneity, 50 to 90% may represent substantial heterogeneity, and 75 to 100% represents considerable heterogeneity [36]. There are challenges in interpreting the I² statistic in the context of prognostic studies, where large sample sizes of included studies result in very narrow confidence intervals. As a result, I² for pooled risk estimates can be extremely high even in the presence of modest inconsistency in risk estimates between individual studies [37]. If statistically significant and considerable heterogeneity exists (P < 0.10 and I² > 75%) and a meta-analysis is nonetheless deemed appropriate, rationale for performing a meta-analysis with potential explanations for statistical heterogeneity (e.g. differences in the prevalence of MACE between studies) will be explored.

Reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results [36]. A funnel plot will be generated if at least 10 studies are included in the meta-analysis, with measures of effect size plotted on the horizontal axis and the standard errors of these measures plotted on the vertical axis. To evaluate reporting biases, funnel plot symmetry will be visually inspected, with measures of effect size plotted on the horizontal axis and the standard errors of these measures plotted on the vertical axis. Evidence of small study effects will be assessed with Egger’s test and visually displayed as an Egger’s plot.

Meta-analyses implementing a random effects model will be performed. External validation studies typically differ in design, execution, and case-mix [38]. Random effects models allow for the presence of heterogeneity between studies by assuming that the effects being estimated in different studies are not identical, but follow some distribution [36]. Mantel-Haenszel RR and ARR analyses will be conducted using Review Manager (RevMan) software [39]. The summary ROC curve and C-statistic, pooled O:E events ratio, and DCA analyses will be conducted using Stata/IC software applying MIDAS and DCA commands [40].

As the prognostic performance of the HEART score in patients presenting to the ED with possible ACS may depend on the baseline risk of MACE, the troponin assay utilised, and who determines the HEART score, the following subgroup analyses will be performed: (1) low versus intermediate versus high baseline risk of MACE, (2) conventional/contemporary versus high-sensitivity troponin assay (see Table 6) used, and (3) attending clinician determined versus researcher or non-attending clinician determined HEART score. These subgroups were selected to maximise the generalisability of results to variable ED clinical approaches and populations.

A sensitivity analysis is a repeat of a meta-analysis, substituting alternate decisions or ranges of values that are subjective in nature [36]. For example, some investigators have deviated from the definition of a low-risk HEART score as described in the score’s derivation study (HEART score 0–3), instead of defining low risk as those with HEART score 0–2 [22]. There have also been variable approaches to outcome measurement, with some studies standardising an observation period or multiple troponin measurements regardless of HEART score (lower risk of bias) and other studies leaving that decision to the discretion of the treating physician (higher risk of bias). Similarly, some HEART score studies assess the primary outcome of MACE at 30 days, while others assess for this outcome at 6 weeks [22]. To test the robustness of this review’s findings, the following sensitivity analyses will be performed: (1) low versus moderate versus high risk of bias assessment, (2) low-risk HEART score of 0–2 versus 0–3, and (3) primary outcome of MACE assessed at 30 days versus 6 weeks.