Diagnostic accuracy of an artificial neural network compared with statistical quantitation of myocardial perfusion images: a Japanese multicenter study

Data from 1001 patients who underwent stress/rest MPI were accumulated from 12 institutions in Japan. The entry criteria included confirmed and suspected coronary artery disease, and a confirmed diagnosis of the presence or absence of myocardial ischemia and/or infarction. The exclusion criteria comprised patients aged <20 years, arrhythmia causing inappropriate electrocardiographic gating, left bundle branch block (to avoid false positive defects), idiopathic and other types of severe cardiomyopathy, as well as moderate or severe valvular heart diseases. Patient data included age, sex, height, weight, risk factors, results of coronary angiography or coronary CT angiography (CCTA), and a history of percutaneous coronary intervention (PCI) or coronary artery bypass grafting (CABG). We tried to include either stress-induced ischemia or infarction in at least 50% of the patients when the MPI data were collected to train the ANN system.

After the initial creation of the database, additional patients were registered using criteria similar to that of the training database. Data from 364 patients collected from nine hospitals served as the validation dataset.

Datasets were derived from all patients using exercise or adenosine stress MPI with electrocardiographic gating and non-gating (Table 1). ^99mTc-labeled hexakis-2-methoxyisobutylisonitrile (MIBI) was used in all participating institutions, and the second injection dose was 2- to 3-fold higher than the first, resulting in a total administered dose of 740 to 1110 MBq. All institutional protocols for stress and image acquisition followed standard exercise and adenosine stress MPI protocols [1, 14]. All image data were anonymized in the "Digital Imaging and Communication in Medicine" (DICOM) format at each institution and uploaded into a custom-designed server. The only patient information in the image database in the server was serial number, age and sex.

The left ventricle (LV) was segmented using a three-dimensional heart-shaped active-shape model reconstructed with short-axis slice images [15, 16]. A specific software algorithm was used for this analysis (cardioREPO software, FUJIFILM RI Pharma Co. Ltd., Tokyo, Japan; EXINI Diagnostics AB, Lund, Sweden). Areas of possible perfusion abnormalities in stress and rest images (stress and rest defects, respectively) were segmented using a method that mimicked analyses by physicians who visually interpret defects. Candidate regions with abnormalities were identified using deformable models of circular topology based on an active contour framework, and fit to the image based on edge and intensity information [17]. Stress images were subtracted from rest images to identify stress-induced ischemia. Appropriate features were extracted using the characteristics of shape, extent, location, count, perfusion homogeneity, regional motion, wall thickening and sex. The best combination of features was determined for stress, rest and difference datasets, and these features served as inputs for the ANN analysis.

Nuclear cardiology experts at each institution judged all regions in MPI data as abnormal (true) or normal (false). The software algorithm identified 5685 candidate regions, but whether or not an abnormality is true was determined by nuclear cardiologists and nuclear physicians. The interpreters used original myocardial stress and rest images and polar maps, but not quantitative data, such as defect scores for detecting abnormalities. After all data were accumulated at the core center, six nuclear cardiology experts reconfirmed the appropriateness of the judgment without clinical information. Two experts interpreted the data during this process and, if necessary, modified the judgment with referral to other expert opinions to reach consensus. The final judgments of the candidate regions made at the core center served as the gold standard with which to train the ANN system.

Based on the initial ANN training, discordance between visual and ANN findings were again selected and used for appropriately adjusting diagnostic thresholds. In the final output of the software, the ANN value indicated the probability of an abnormality under conditions of 0.0 (definitely normal), 0.5 (borderline) and 1.0 (definitely abnormal) for stress defects, rest defects and induced ischemia.

Four DICOM stress/rest gated and ungated images were transferred from the nuclear medicine dedicated computer system to an on-line connected (off-line acceptable) Windows computer with the aid of a custom-made launcher. After selecting a patient dataset, all processing steps including LV segmentation, polar map generation, detection of abnormal regions, judgment of the abnormality with the ANN system, scoring based on Japanese Society of Nuclear Medicine (JSNM) working group normal databases [18, 19], LV functional analysis including wall motion, volume, ejection fraction and diastolic function, and phase dyssynchrony analysis were performed within 10 s (64-bit operating system, Windows 10) [10, 16, 19‐21]. This processing was automatic without operator interaction, but LV contour detection can be modified if required. The ANN system displays a list of regions including the location, extent, severity and ischemia/infarct with the probability of an abnormality (ANN value) in each region.

Myocardial perfusion defects were scored using a standard 17-segment model, and abnormalities were defined as normal, slightly, moderately and severely decreased, and defective (scores 0, 1, 2, 3, and 4, respectively) in each segment [3, 4]. Scores were validated in automated analysis using cardioREPO software, and the correlation was good compared with that determined using QPS (Cedars Sinai Medical Center, Los Angeles, CA, USA) [10]. The maximum score was 68 points for summed stress/rest/difference scores (SSS/SRS/SDS). Normal MPI databases were constructed according to the JSNM working group database that includes stress/rest normal SPECT data with non-attenuation-corrected Anger camera images [18, 19].

Data are shown as means ± standard deviation (SD). Differences between groups were assessed using the one-way analysis of variance, Student's t tests and F tests. A contingency table was analyzed with Fisher exact tests. Receiver operating characteristics (ROC) were analyzed, and the area under the curve (AUC) was calculated. The cutoff was determined using either an ANN value (probability of abnormality) of 0.50 or the highest value of (sensitivity + specificity − 1) depending on the analysis. The statistics software was JMP version 12 (SAS Institute Inc., Cary, NC, USA), and Mathematica 12 (Wolfram Research Inc., Champaign, IL, USA) was also used for some mathematical calculations. A p value <0.05 was considered significant.

Table 1 summarizes the characteristic of the training databases that included stress defects (71%), rest defects (57%) and ischemia (59%). Old myocardial infarction (OMI) and revascularization were found in 27% and 42% of the patients, respectively. The AUC of the training database calculated to obtain results concordant with those of the experts were 0.912 (sensitivity 86%, specificity 77%, accuracy 82%) for stress defects, 0.834 (sensitivity 83%, specificity 74%, accuracy 78%) for rest defects and 0.888 (sensitivity 82%, specificity 77%, accuracy 80%) for ischemia.

Background conditions and associated diseases did not significantly differ between the validation and the training databases for most of the variables (Table 1). Rates of stress defects, rest defects and ischemia were similar. However, the validation database had a higher incidence of pharmacological stress and of coronary stenosis, and a lower ejection fraction than the training database.

Figure 1 shows the standard output generated using the ANN. This patient had 99% stenosis of the circumflex coronary artery (#11). Six months after stenting he developed chest pain, which was assessed by MPI. The ANN judged the high lateral region as abnormal in both stress and subtraction images with high probability.

The AUC values of the stress defect were 0.815 and 0.916 (χ² = 34.4, p < 0.0001) for the scoring system and the ANN, respectively (Fig. 2). Similarly, the AUC was 0.857 vs. 0.926 (χ² = 21.9, p < 0.0001) for rest defects, and 0.754 vs. 0.895 (χ² = 34.2, p < 0.0001) for ischemia, respectively, indicating a higher AUC for the ANN compared with the scoring method.

Average scores for stress defects were 13.2 ± 10.6 vs. 3.9 ± 4.3 (p < 0.0001, F ratio = 99) in patients with and without stress defects, respectively (Fig. 3). The values for stress defects determined using the ANN in patients with and without stress defects were 0.84 ± 0.24 vs. 0.31 ± 0.29 (p < 0.0001, F ratio = 347). Similarly, although both scores and ANN values under conditions of rest, stress and ischemia significantly differed between positive and negative expert readings (p < 0.0001), the F ratios were higher when determined by the ANN than the scoring method.

The ability of the ANN to detect abnormalities in patient subgroups with and without OMI or revascularization was generally good, with an AUC of ≥0.89 (Table 2). The AUC values for the ANN to detect stress defects in patients with and without OMI were 0.976 and 0.892, respectively. In patients with OMI, the AUC values to detect rest defects using SRS and the ANN were 0.905 and 0.974, respectively. The AUC values for detecting stress defects with and without a history of revascularization were 0.939 and 0.900, respectively. The AUC in all subgroups was higher for the ANN than the scoring method.

According to ≥75% of coronary stenosis, we divided the patients into groups according to their having 0–1-, or 2–3-vessel disease. The AUC values were 0.895 and 0.785 when determined using the ANN and SSS, respectively (χ² = 24.2, p < 0.0001) for patients with 0–1-vessel disease and 0.945 and 0.838, respectively, for patients with 2–3-vessel disease (χ² = 11.0, p = 0.0009), indicating that the ANN had better diagnostic accuracy.

Figure 4 shows a non-linear relationship between scores and the ANN values in the validation group. Although the threshold of abnormality using the ANN was a probability of 0.5, the scores steeply increased when the ANN probability was ≥0.8. The SSS was 3.4 ± 3.5 in the range of ANN probability <0.80, but 15.8 ± 10.3 when in the probability range of ≥0.80 (p < 0.0001) determined by the ANN. The SDS values were 1.8 ± 4.7 and 4.7 ± 3.8 when the ANN probability was <0.8 and ≥0.8, respectively, and the SRS values were 2.3 ± 3.0 and 14.2 ± 9.8 when the ANN probability was <0.8 and ≥0.8, respectively (p < 0.0001).

Diagnostic accuracy using criteria of significant coronary stenosis of ≥75% with either coronary angiography or CCTA was examined in a subgroup of patients (n = 220), excluding those after coronary revascularization. Using these criteria, 0, 1-, 2- and 3-vessel disease was found in 98 (45%), 44 (20%), 41 (19%) and 37 (17%) patients, respectively. The AUC values using stress defects for detecting 1-, 2- and 3-vessel disease were 0.69, 0.67 and 0.69 by expert interpretation, and those by the ANN were 0.68, 0.66 and 0.72, respectively, showing an overall AUC of 0.66 (95% CI: 0.60–0.72) by expert interpretation and 0.65 (0.58–0.72) by ANN (p = 0.74) for detecting at least one coronary stenosis.

Basic approaches of artificial intelligence-based diagnostic systems differ from the diagnostic steps based on conventional statistical methods [13]. Normal or standard databases are created from individuals with a low likelihood of coronary artery disease for statistical analysis, and the limits of normal distribution are determined for all myocardial data points. Combinations of the average and the deviation or ratio (%) of threshold counts are used to measure degrees of abnormalities, which are expressed as defect scores. Such statistical analysis provides consistent diagnosis using common databases fitted for populations [22]. It also serves as an aid to clinical diagnosis and has, therefore, been used in most quantitation software packages [5, 23‐25]. Quantitative diagnosis using a scoring method is convenient for measuring amounts of ischemia, and for guiding appropriate indications for coronary revascularization [6].

Here, we scored defects using a JSNM working group database of myocardial SPECT images for this purpose [18, 19], which was compared with ANN-based diagnosis. On the other hand, the ANN uses a completely different approach [9, 10, 12]. The software in this study first identified candidate areas of abnormality from >5000 candidate regions based on a specific algorithm, and learned the expert interpretation. Based on the notion that the expert diagnosis is true, the most appropriate features were determined for stress, rest and difference images. In the clinical setting, MPI diagnosis is not simply based on average counts on polar maps, but on the integration of myocardial counts, size and shape of defects, wall motion and systolic thickening, gender and other parameters. Although experts might have unintentionally taken these factors into consideration, the ANN was capable of learning this comprehensive diagnostic process.

Artificial intelligence such as a support vector machine (SVM) and machine learning technologies has generated promising results for the diagnosis and prognosis of patients with coronary artery disease. The AUC for SVM (0.92) is significantly better than total perfusion deficit (0.90) when the SVM algorithm is applied, and its diagnostic accuracy is comparable to the overall accuracy of visual readers [8]. Machine learning that included a combination of clinical and CCTA data predicted 5-year all-cause mortality more effectively than existing clinical or CCTA metrics alone [11]. Since artificial intelligence considers a greater number and complexity of variables than humans, it might be applied for wider indications in the near future.

Since the purpose of this project was to create an ANN with diagnostic ability that was similar to that of experts, true abnormality was defined by consensus among expert interpretations. Popular methods for evaluating diagnostic accuracy to date have used coronary stenosis as the gold standard by which sensitivity and specificity are calculated. However, all three coronary territories might not be judged as abnormal in patients with three-vessel disease. If the most severe stenosis is a culprit lesion and only one territory appears abnormal, even experts might judge only one coronary territory as abnormal. Moreover, some patients who have been diagnosed with myocardial infarction might have recovered perfusion to some extent. Others might have insufficient coronary blood flow supply even from the revascularized coronary artery. Recent indications for MPI include not only patients who are untreated, but also those with many complex pathophysiological conditions such as myocardial ischemia, fibrosis and modifications after revascularization and medical therapies. In fact, 30% of patients included in the present study had OMI and 40% had a history of revascularization. Although we did not use coronary stenosis as the gold standard, the diagnostic ability of the ANN was significantly better than scoring, and comparable to that of experts, indicating a good foundation for further clinical application.

However, even when coronary stenosis of ≥75% was used as a diagnostic criterion, the diagnostic accuracy using ROC analysis was comparable between the expert interpretation and ANN. The reason for excluding patients with history of coronary revascularization was that “truth” regarding the presence of remaining ischemia or defects was not confirmed in these patients. Although this analysis was not the goal of the present study, strict criteria of ischemia using coronary stenosis and fractional flow reserve could be evaluated in further studies.

The interpretation of myocardial perfusion images during ANN training could affect final diagnostic accuracy. The diagnostic accuracy of the ANN in a previous study of a Swedish database was 92% for abnormalities at stress and 87% for ischemia [9]. In addition, the same ANN generated an AUC of ≥0.88 in a Japanese population [10]. After accumulating clinical experience with this software, we found that a minor degree of ischemia was overlooked in a Japanese population. We therefore tried to judge even a slight degree of abnormality as abnormal, which is in agreement with our clinical practice. The ANN learned this tendency of expert interpretation during this project. However, the probability of an abnormality, namely the calculated ANN value in regions with a minor degree of ischemia, might be somewhat lower than that of a definite perfusion defect or ischemia. Overall, since a common stress/rest protocol using ^99mTc-MIBI was applied, we suppose that the software can be applied to routine clinical investigations at many hospitals. The applicability of the ANN-based system to diagnostic ²⁰¹Tl and other SPECT studies using equipment such as cadmium-zinc-telluride detectors should be separately investigated.

The non-linear relationship between ANN values and defect scoring was a notable feature that seems in agreement with human decision-making processes. Nuclear cardiologists do not usually judge findings with defect scores <4 as abnormal, but will definitely do so when scores are ≥8. Thus, the likelihood of an abnormal probability steeply increasing between 4 and 8 is understandable. Considering this non-linear relationship, the scoring method cannot be replaced by the ANN. Although 10% ischemia of the left ventricle is used as a guide to coronary revascularization [6, 26], the ANN does not provide a comparable threshold of ischemic severity. However, the ANN can emphasize a possible abnormal region with a degree of probability, which can suggest a diagnosis. Avoidance of overlooking positive findings and over-diagnosing minor abnormalities might be helpful for nuclear cardiology trainees. Since cardiologists and radiologists are not always specialists in nuclear cardiology, an ANN-based suggestion might help enhance confidence in a diagnosis. Even some nuclear cardiology specialists tend to interpret findings with high sensitivity (active reading for abnormality), and others with high specificity (modest reading). The average reading provided by the ANN could thus be a second opinion for such specialists.

Lastly, all perfusion processing, defect scoring and functional analysis are automatically calculated by the software within 10 s. Therefore, if the data transfer protocol from a nuclear-dedicated computer to Windows PC and connection to an institutional picture archiving and communication system (PACS) are appropriately constructed, the ANN software system could function in any institution.

The territories of the three coronary arteries were not included in the training process. Although the coronary territory and polar maps might roughly correspond, such as the anteroseptal region to the left anterior descending coronary artery, strict correspondence in border zones of coronary territories is difficult to determine. Although we could not determine coronary-based accuracy in this study due to its algorithm, fusion images with CCTA and perfusion maps with their probability of ischemia will effectively integrate information from stenosis and perfusion. As discussed above, we could not generate a severity score indicating 10% ischemia. To develop an ANN that can assess ischemic severity comparable to that of severity scoring, new large-scale training will be required. The judgment suggested by the ANN system was based on average readings of the MPI studies at 12 institutions. The variety of conditions is a limitation from the viewpoint of specific readings at these institutions, but this could conversely provide the advantage of averaged interpretation.

Based on a new multicenter database trained by nuclear cardiology specialists, the diagnostic ability of the ANN was good with an AUC of >0.9 overall, including subgroups with and without MI and coronary revascularization. Further studies of large patient cohorts are indicated based on these promising outcomes.