Central nervous system (CNS) agents cause fractures among the elderly, but fracture risks of a wide range of CNS agent classes have not been analyzed in a study with the same population and definitions of variables. This study aimed to estimate the degree of fragility fracture risk of a wide range of CNS agent classes in elderly Japanese people. A case-crossover design study, with a case window and three control windows of 3 d each, as well as longer windows up to 15 d, was conducted among opioid non-users who lived without hospitalization for ≥13 months and incurred fragility fractures at ≥65 years of age, using the National Database of Health Insurance Claims and Specific Health Checkups of Japan. Conditional logistic regression estimated adjusted odds ratios (ORs) of CNS agent classes for fragility fractures for groups including and excluding users of pro re nata CNS agents (PRN-CNS agents) and for windows of 3–15 d. Antiepileptic agents had the highest adjusted ORs, 2.4 (95% confidence interval 2.3–2.5) for the group including PRN-CNS agent users (n = 446101). The next-highest classes were anti-dementia agents 1.5 (1.5–1.6), antipsychotics 1.5 (1.4–1.6), anti-Parkinson agents 1.3 (1.2–1.5), and antidepressants 1.1 (1.1–1.2). Similar ORs were found when PRN-CNS agent users were excluded (n = 352828), and slightly higher ORs were found for longer windows, with almost the same order of classes. Elderly individuals who use antiepileptic agents or a combination of antiepileptic agents and CNS agent classes with the next-highest ORs should be carefully monitored.