Adverse childhood experiences (ACEs) are related to depression in older adults. However, little is known about the long-term effect of ACEs or potential gender differences. This study aimed to examine the longitudinal associations between multiple types of ACEs and depression trajectories in middle-aged and older Chinese adults and to compare these associations between men and women. Four waves of data were extracted from the China Health and Retirement Longitudinal Study (n = 4829). We first performed growth mixture modeling to identify four depression trajectories. Multinomial logistic regression analyses were subsequently conducted to examine the relationships between ACEs and depression trajectories, stratified by gender. The four depression trajectories were labeled persistently severe depression (5.2%), increasing depression (14.9%), decreasing depression (12.2%), and stable asymptomatic (67.8%). Among all respondents, ACEs of feelings of loneliness, parents with mental health problems, a lack of friends, and bullying by peers significantly predicted persistently severe or unstable depression trajectories. Among men, poor neighborhood relationships in childhood significantly predicted a persistently severe depression trajectory. Among women, health limitations, poor health status, and low-quality and unsafe neighborhoods predicted persistently severe or unstable depression trajectories. Our findings highlight the detrimental effects of ACEs on persistently severe or unstable depression trajectories in middle age to old age. Long-term and prospective awareness of these gender differences should be incorporated into the provision of mental health services and interventions for older people. More efforts should be made to protect children from abuse and neglect and provide timely interventions for those suffering from ACEs.