Venous thromboembolism (VTE) is a multifactorial disease. Cancer and older age are risk factors for both recurrent VTE and bleeding under anticoagulant therapy. Oral direct factor Xa inhibitors (Xa inhibitors) have been widely used to treat VTE. However, their effectiveness and safety in cancer and elderly patients have not been fully elucidated. A total of 187 consecutive patients who started Xa inhibitors for VTE therapy between September 2014 and September 2016 were recruited. Patients’ demographics, changes in VTE amount, VTE recurrence, clinically relevant bleeding, and death until February 2017 were compared between 92 cancer and 95 non-cancer patients, and 57 elderly (≥ 75 years) and 130 non-elderly patients. Compared with non-cancer patients, cancer patients had a significantly higher incidence of deep vein thrombosis (DVT) in the proximal legs, superior vena cava, and upper extremities (p = 0.034), although the patients’ demographics and incidence of pulmonary thromboembolism (PE) were similar between the two groups. There were no significant differences in VTE recurrence (p = 0.328) and clinically relevant bleeding (p = 0.078) between the two groups. Death occurred in 29 cancer patients, 23 of whom died of cancer, while there were no deaths among the non-cancer patients. Elderly patients had a lower body weight and creatinine clearance than non-elderly patients. No significant differences between the two groups were found in relation to PE (p = 0.544), DVT site (p = 0.054), recurrent VTE (p = 0.194), clinically relevant bleeding (p = 0.130) and death (p = 0.241). In comparisons among the four groups (elderly and non-elderly patients with and without cancer), recurrent VTE and clinically relevant bleeding were comparable (p = 0.493 and 0.227, respectively), while death was more frequent in cancer patients regardless of age (p < 0.001). The efficacy and safety of Xa inhibitors as VTE treatment were comparable between cancer and non-cancer patients, and in elderly and non-elderly patients. This suggests that Xa inhibitors may be promising drugs for VTE treatment, irrespective of age and comorbid cancer.