Previous studies demonstrated that men were more likely to have plaque rupture and are at greater risk for myocardial infarction and stroke than women. We evaluated differences in carotid plaque characteristics by MRI between men and women with mild-moderate atherosclerosis and elevated ApoB levels. One hundred eighty-two subjects (104 men and 78 women) with CAD or carotid stenosis (≥ 15% by ultrasound), ApoB ≥ 120 mg/dL and carotid MRI scan were included. Percent wall volume (%WV) was calculated as (wall volume/total vessel volume) × 100%. Three major plaque compositions, fibrous tissue (FT), calcification (CA) and lipid rich necrotic core (LRNC), were identified and quantified using published MRI criteria. Adventitial and plaque neovascularization as fractional plasma volume (V_p) and permeability as transfer constant (K^trans) were analyzed using kinetic modeling. These characteristics were compared between men and women. Men, compared to women, were younger (54 ± 8 vs. 58 ± 8 years, p = 0.01), had higher rate of previous MI (46 vs. 26%, p = 0.005) but lower proportions of metabolic syndrome (37 vs. 59%, p = 0.003). After adjusting for between-gender differences, men were significantly more likely to have LRNC (OR 2.22, 95% CI 1.04–4.89, p = 0.04) and showed significantly larger %LRNC than women (diff = 4.3%, 95% CI 1.6–6.9%, p = 0.002), while %WV, FT, and CA were similar between men and women. There were no statistically significant differences in adventitial and plaque V_p or K^trans. Men were significantly more likely to have LRNC and had larger LRNC than women. However, men and women showed relatively similar levels of adventitial and plaque neovascularization and permeability.