Our aim was to evaluate the plaque characteristics of coronary arteries related to significant stenosis with coronary CT angiography (CCTA) and to discuss the diagnostic accuracy of CCTA in patients with high calcium scores. After institutional review board approval, 110 patients (63 men; mean age: 67.1 ± 7.9 years) with Agatston scores >400 were retrospectively reviewed. Patients underwent Agatston calcium scoring and 64-slice CCTA, in addition to invasive coronary angiography (CAG). The composition (calcified, mixed, and non-calcified) and configuration (concentric, eccentric) of coronary artery plaques were analyzed on a per-segment basis by CCTA. We analyzed the differences in plaque composition and configuration between significant (≥50%) and non-significant (<50%) stenosis. Additionally, the diagnostic accuracy of stenosis according to plaque composition was evaluated by CCTA, using CAG as a reference method. Significant differences in plaque composition and configurations were observed between the two groups. In cases of significant stenosis, the proportions of concentric, mixed, and non-calcified plaques were significantly higher than those of eccentric and calcified plaques (P < 0.001). The sensitivity and positive predictive value of mixed (97.4, 87.6%) and non-calcified plaques (97.8, 95.7%) were significantly higher than those of calcified plaques (87.6, 67.2%). Although CCTA has limited value due to low diagnostic accuracy of calcified plaques, knowledge about the high frequencies of mixed and non-calcified plaques in significant stenosis help to make an accurate assessment of CAD with CCTA in patients with high calcium scores.