Coronary artery calcium (CAC) is associated with the presence of coronary artery disease (CAD) and cardiovascular risk factors. However, the relation between cardiovascular risk factors and CAD has not yet been fully elucidated in patients with a zero or low coronary artery calcium score (CACS). The purpose of this study was to evaluate the relation of cardiovascular risk factors and angina status to obstructive CAD according to categorical CACS. A total of 753 patients were enrolled in this study. CAC scoring and coronary computed tomographic angiography (CCTA) were performed with dual-source 64-slice CT scanners. The number of patients with a CACS ≤10 and ≤100 were 358 and 528, respectively. Patients with a higher CACS were older and more frequently male, and had a greater frequency of hypertension, diabetes, and hypercholesterolemia. The prevalence of obstructive CAD increased with the CACS. Among patients with a CACS ≤100, age, male gender, diabetes, hypercholesterolemia, and typical angina pectoris were related to obstructive CAD. The presence of hypercholesterolemia was relatively strongly associated with obstructive CAD (OR 6.67, 95% CI 2.91–15.3, p < 0.001) on multivariate analysis. Among patients with a CACS ≤10, men, hypercholesterolemia, and typical angina pectoris were significantly more frequent in patients with than in those without obstructive CAD (p < 0.01). Our data suggest that neither the absence nor low of coronary calcium burden may reliably exclude obstructive CAD in typical symptomatic male patients with hypercholesterolemia. This result may be useful to interpret the relation of CACS to obstructive CAD.