Validation of case definitions of depression derived from administrative data against the CIDI-SF as reference standard: results from the PROspective Québec (PROQ) study

The PROspective Québec (PROQ) study is a prospective cohort that has followed 9188 white-collar workers (49.8% women) since 1991 in Quebec, Canada [23]. Questionnaires, clinical measurements and interviews were used to collect participants’ data (socioeconomic conditions, lifestyle habits and physical and psychological health) at three data collection waves: 1991–1993 (n = 9188), 1999–2001 (n = 8120) and 2015–2018 (n = 6744). For the present study, the sample was restricted to participants who responded to the CIDI-SF instrument at the last data collection wave and who consented to having their data linked with the administrative data provided by the provincial universal healthcare provider, the Régie de l’Assurance Maladie du Québec (RAMQ; n = 5487, 66% of those eligible to this last wave).

Cases of depression were estimated in 2015–18 using a validated French version of the WHO’s Composite International Diagnostic Interview-Short Form (CIDI-SF) [24, 25]. CIDI-SF is consistent with the definition of depression in the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) and, more broadly, with that in the International Classification of Diseases (ICD-10). Following DSM-IV, participants were considered cases if they reported having had at least five depressive symptoms almost all day, almost every day, for a period of two consecutive weeks or more in the past 12 months. The symptoms were 1) depressed mood (dysphoria), 2) loss of interest in most things (anhedonia), 3) fatigue / lack of energy, 4) weight change (loss or gain), 5) difficulty falling asleep, 6) difficulty concentrating, 7) feeling down or worthless, and 8) thinking about death, one’s own or that of others. Dysphoria or anhedonia had to be present among the five symptoms [26]. Validation studies of this instrument observed a > 90% probability of detecting depression when applying this definition [27, 28].

RAMQ provides almost all health services in Quebec [29]. All administrative data was extracted from three RAMQ databases (population registry, hospital discharge abstracts and physician services) for the period from January 1, 1991 to December 31, 2018. Hospital admission and discharge dates and ICD codes were obtained from the hospital discharge abstracts (ICD-9 before 2006, ICD-10 beginning in 2006). Consultation dates associated with ICD-9 codes for ambulant treatment were extracted from the physician services database. For the main analysis, we considered ICD-9 codes 296.x, 300.4, 309.x and 311.x and ICD-10 codes F30-F34, F39 and F43.2. To estimate incidence, dates of death were also obtained from the population registry. These databases were joined with each other and with the PROQ cohort database using unique personal identification codes.

Based on previous investigations [9, 16, 17], we considered any hospital stay associated with a diagnosis of depression sufficient to constitute an administrative case, but evaluated two different case definitions with regard to medical consultations:

For the purpose of comparing with CIDI-SF cases (which referred to the year preceding the last wave), the case definitions of administrative cases were applied to that same year.

Known risk factors for depression were used to assess known-groups validity, which is established when an instrument can effectively distinguish between two groups who are known a priori to differ on the variable of interest [30, 31]. In this case, we used sociodemographic variables that are known risk factors for depression: sex (man/woman); age (dichotomized at the median age at retirement) and level of education at baseline (with or without a university degree). In addition, we defined groups based on psychological distress, measured in 1999–2001 using the validated French version of the Psychiatric Symptom Index [32, 33]. Scores were dichotomized at the median.

Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratios (LR+, LR-) each with its 95% confidence interval (95% CI), were calculated for each case definition of depression derived from administrative data, using the CIDI-SF as reference standard. We also calculated Cohen’s kappa (κ) [34].

Furthermore, additional analyses were carried out in order to compensate for possible effects of selective attrition during the follow-up. We used multiple imputation by chained eqs. (20 imputations, 10 iterations) first to infer the missing data from the covariates obtained at recruitment (< 5% missing data, 1991–1993), and then the missing data from the second wave (< 10% missing data, 1999–2003). Using the multiply imputed data, we adjusted the prevalence of depression at the third wave (2015–2018) by inverse probability of censoring weighting (IPCW), which gives more weight to the participants that are more similar to the ones that were lost to follow-up, and therefore permits inferring a population without attrition. The probability of censoring of each subject was estimated as the predicted value of a logistic regression where the dependent variable was whether the subject was censored, and the independent variables were sociodemographic ones [sex, age (continuous), marital status (married or living together vs. living alone), and health behaviors [smoking (never, in the past, currently), alcohol consumption (≤ 5 doses/week, > 5 doses /week), physical activity (≤ once/month, ≤ once/week, ≥ once/week) and body-mass index (weight (kg) by square of height (m))], measured at the first two waves. Sensitivity, specificity, PPV, NPV and kappa with their 95% CI were estimated from the imputed and weighted datasets using non-parametric bootstrap (2000 replications) [35].

For the known-groups analysis, we estimated the prevalence of depression separately for each stratum of four known risk factors for depression: sex, age, educational level and psychological distress [32]. To quantify the relative risk, the association between each risk factor and each case definition of depression was estimated using a robust Poisson model. This is a convenient approach to estimate prevalence ratios with a binary outcome while being less sensitive to model misspecification and avoiding the non-convergence problems of the log-binomial regression [36]. The analyses were conducted in software R, version 3.6.1 with the tidyverse, epiR, geepack, mice [37] and boot [38] packages.

The strengths of this study include the use of a large cohort of workers, not restricted to those who had sought medical help or those with a pre-existing disease. Moreover, we had access to administrative data for 96% of the cohort. The high rates of participation at recruitment and of consent to access administrative data reduce a possible selection bias. Furthermore, the observed socio-demographical characteristics do not indicate selective attrition during follow-up, suggesting a low risk of selection bias for the CIDI-SF questionnaire data. The results obtained after multiple imputation and IPCW were similar to those found using complete cases, also reducing the possibility of selection bias. Finally, the size of the population was sufficient for estimating sensitivity, specificity, predictive values and concordance between administrative and questionnaire data in different socio-demographical strata and under a variety of case definitions.

Our study also has limitations. First, our population was composed of white-collar workers in Quebec’s public and semi-public sector and might not reflect the general population. However, the one-year prevalence of depression based on CIDI-SF cases found in this cohort is very similar to that obtained for the overall Canadian population [40]. Furthermore, while we used a questionnaire validated for diagnosing clinical cases of depression as reference, we recognize that the ideal reference measure would be the use of the Structured Clinical Interview for DSM-IV [28]. There are medical services claims without diagnostic code in the administrative data; however, two different approaches to treating these missing codes did not fundamentally alter the agreement with questionnaire cases. Finally, since the number of participants below 50 years was low for the last wave, we could not evaluate the validity of administrative cases in this age range.