Infants with congenital heart disease (CHD) have delayed brain maturation and alterations in brain volume. Brain metrics is a simple measurement technique that can be used to evaluate brain growth. This study used brain metrics to test the hypothesis that alterations in brain size persist at 3 months of age and that infants with CHD have slower rates of brain growth than control infants. Fifty-seven infants with CHD underwent serial brain magnetic resonance imaging (MRI). To evaluate brain growth across the first 3 months of life, brain metrics were undertaken using 19 tissue and fluid spaces shown on MRIs performed before surgery and again at 3 months of age. Before surgery, infants with CHD have smaller frontal, parietal, cerebellar, and brain stem measures (p < 0.001). At 3 months of age, alterations persisted in all measures except the cerebellum. There was no difference between control and CHD infants in brain growth. However, the cerebellum trended toward greater growth in infants with CHD. Somatic growth was the primary factor that related to brain growth. Presence of focal white matter lesions before and after surgery did not relate to alterations in brain size or growth. Although infants with CHD have persistent alterations in brain size at 3 months of age, rates of brain growth are similar to that of healthy term infants. Somatic growth was the primary predictor of brain growth, emphasizing the importance of optimal weight gain in this population.