Congenital heart defects (CHD) lead to extensive use of healthcare resources. Still, there is little information available regarding readmission rates or associated factors. We sought to evaluate readmission rates and their determinants among patients with CHD hospitalized in a Swiss university hospital. We conducted a retrospective study using data from all non-adult (<18 years) patients hospitalized between 2002 and 2014 at the University Hospital of Lausanne with an International Classification of Diseases version 10 code Q20–Q25. Overall, 996 patients (460 girls, 332 undergoing surgery, mean age 2.7 years) were assessed, 96 of whom (9.6%) were readmitted within 30 days after discharge. Among the 96 readmissions, 83 (86.5%) were related to the CHD. Median time to readmission was 10 days (interquartile range 6–20) and median length of readmission was 12 days (interquartile range 6–20). After multivariate adjustment, foreign nationality, greater distance to hospital and length of index hospitalization <14 days predisposed to readmission. Patients who underwent surgery were less likely to be readmitted (8.7%). We conclude that readmissions were frequent, almost 1 in 10 patients, and associated with several socio-clinical factors. Providing patients who live far from hospital with specialized care closer to home may help reduce the rate of readmission.