Harmonization of delirium severity instruments: a comparison of the DRS-R-98, MDAS, and CAM-S using item response theory

The goal of this project was to describe the relationship of three delirium severity instruments to one another. We accomplished this within a prospective observational study of hospitalized older adults using modern psychometric methods including factor analysis and item response theory.

We distinguish the intensity and severity in the context of delirium. As used in the field, delirium severity is a broad concept that encompasses the intensity of multiple symptoms. To allow for the conceptual distinction of the varied aspects of delirium severity, we use the term intensity to describe the distribution of cognitive, behavioral, psychiatric, and functional signs and symptoms associated with a single assessment. Intensity is viewed as existing along a continuum, ranging from the absence of signs and symptoms of delirium to highly disruptive behaviors or severely impairing symptoms. Intensity occupies the same conceptual space as the sum of symptom ratings represented by the total scores on the DRS-R-98, MDAS, and CAM-S. However, each instrument has its own metric that is defined by the particular selection of the number of domains assessed and the specific definitions used to characterize different rating levels of each sign or symptom. The result is that the summary scores may not necessarily have a direct relationship to one another, and thus, it can be challenging to integrate results from different studies using different measurement instruments.

We address this challenge by using item response theory methods to define the metric of underlying intensity of the individual delirium signs and symptoms. We use a common item equating design [14] and item response theory methods [15] to define an underlying delirium intensity metric that is equal across all three instruments. Moreover -- and as a consequence of using item response theory to define the metric for intensity -- the intensity metric can be used to describe the signs or symptoms; that is, the metric can be used to describe the level of impairment expressed by an individual patient at a discrete time of observation. Also, the metric can be used to describe specific items that assess the signs or symptoms of delirium: namely the region of the intensity metric that is measured symptoms are rated in lower or higher categories. In this way, the conceptualization of delirium severity represents a composite of the intensity of individual symptoms.

The DRS-R-98, MDAS, and CAM-S were used to rate delirium severity following brief interviews. The instruments encompass some overlap in features of delirium yet are distinctive. The instruments vary in their coverage of delirium severity domains and the manner in which the intensity of a specific symptom is rated. Thus, the interview source items used to score the three instruments vary.

The DRS-R-98 [16] uses family, chart, and nurses to identify and rate the severity of delirium according to 13 items: sleep/wake cycle disturbance, perceptual disturbances and hallucinations, delusions, lability of affect, language, thought process abnormalities, motor agitation, motor retardation, orientation, attention, short-term memory, long-term memory, and visuospatial ability The ratings for each item include 0 (no impairment), 1 (mild), 2 (moderate), and 3 (severe impairment). An overall score is created by summing the score for each item and higher scores indicate greater severity of delirium.

The MDAS [13] assesses the severity of delirium using 10 items on a four point scale (0 to 3) similar to the DRS-R-98 scale described above. The items assess diagnostic criteria of delirium according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV [17]). MDAS items include reduced level of consciousness, disorientation, short-term memory, impaired digit span, reduced ability to maintain and shift attention, disorganized thinking, perceptual disturbance, delusions, psychomotor activity, and sleep/wake cycle disturbances. Again, an overall score is created by summing each item score; scores of 13 or higher indicates delirium, and higher scores indicate greater delirium severity.

The CAM [9] consists of 10 operationalized items from the DSM-III: acute onset and fluctuation, inattention, disorganized thinking, altered level of consciousness, disorientation, memory impairment, perceptual disturbances, psychomotor agitation or retardation, and altered sleep-wake cycle. The CAM diagnostic algorithm of delirium requires the presence of acute onset and fluctuation and inattention, and either disorganized thinking or altered level of consciousness. The severity scores derived from the CAM include the CAM-S [10] long form (10 items from the full CAM) and short form (four items from CAM diagnostic algorithm). Scoring systems allow for the quantification of the severity of delirium. The CAM-S is scored by rating CAM features, except acute onset or fluctuation, on a three-point scale: 0 (absent), 1 (mild), or 2 (marked). Acute onset or fluctuation is rated 0 (absent) or 1 (present). The composite (sum) short form scores range from 0 to 7, and the long form scores range from 0 to 19, with higher scores indicating greater delirium severity.

The daily hospital interviews in the BASIL study included a variety of formal questions and cognitive tests. These interview items were administered by the same raters, in same ordering, for every participant. In addition, interviewers gathered observational evidence throughout the hospital visits. Inter-rater reliability was assured through initial training and ongoing standardization approaches of the raters throughout the study. We modified the DRS-R-98 to accommodate assessment by trained lay interviews, rather than clinicians as originally intended by its developers. The scoring of the DRS-R-98, MDAS, and CAM-S was informed by source items (e.g., interview questions or cognitive tests) and informal observations. In addition to questioning and cognitive testing, the rated items in the three instruments include symptoms rated based on observations of the patient noted throughout the interview. Interviewers were trained to take extensive observational notes during the entire course of the patient visit. Some rated items, such as abnormal level of consciousness and psychomotor agitation/retardation, were not associated with specific formal source items and were scored using observational evidence only. The rated items of the DRS-R-98, MDAS, and CAM-S have a differing number of response categories. Rated items from the DRS-R-98 and MDAS have four response categories (not present, mild, moderate, and severe), while the rated items in the CAM have three response categories (not present, mild, marked). To facilitate co-calibration across delirium severity instruments, for initial modeling steps, described in the analysis plan, the response categories were dichotomized to not present vs any symptom, except for sleep disturbance. Because most participants endorsed mild sleep disturbance, we dichotomized it as not present or mild vs moderate or severe to identify only the more severe cases of sleep disturbance. The original polytomous items were used during the final scoring procedure. Items that were coded as uncertain, refused or don’t know were set to missing.

Harmonization is a broad term that describes a process of addressing differences in measurement or assessment that could involve procedural, rational, or statistical approaches [18]. Our approach was to use statistical methods for harmonization using item response theory (IRT) methods [14]. IRT is a latent variable technique that describes a large family of statistical approaches that are used to describe the relationship between item responses (in our case, delirium sign or symptom ratings) and the latent trait presumed to underlie those responses (in our case, delirium intensity). We equate the latent trait distributions underlying each of three delirium assessment instruments, and report links (or crosswalks) among the total scores derived from each instrument. We use a blended common person and common item design [14, 19].

Seven items were shared among the three instruments, while the other items were shared among two instruments or not shared at all. Shared means that the content of the symptom was similar across instruments. The shared and unshared delirium items were associated with varying source material, as outlined in Fig. 1. Shared items among the three instruments included attention, disorganized thinking/thought process abnormality, orientation, perceptual disturbance, psychomotor agitation/retardation, sleep/wake cycle disturbance, and memory impairment. The assessment of attention involved digit span repetition (forward and backward), sentence repetition, and days of the week and months of the year backward. Source items used to score orientation assessed orientation to year, season, month, day of the week, date, city or town, name of place, and type of place. Sleep/wake cycle disturbance was assessed using six questions related to sleep cycle in the past 24 h. A three-object recall was used to evaluate memory impairment.

To perform statistical harmonization of the instruments, we used co-calibration based on IRT in four modeling steps.

First, a unidimensional confirmatory factor analysis (CFA) model was fit to each of the three instruments separately using dichotomized versions of the set of seven shared items among the delirium instruments. We used Mplus software (version 8, Muthén & Muthén, Los Angeles CA) and a robust maximum likelihood estimator and logit link for dichotomous items, and thereby implement a two parameter logistic IRT model with this CFA [20]. The goal of this step is to assess the appropriateness of a unidimensional model.

In the second modeling step, another unidimensional CFA model was fit separately to each of the three instruments using dichotomized versions of all the rated items of the instrument, with the factor loading and location parameters of the shared items constrained to the estimated values from the first model. The goal of this step is to determine item parameters for the non-shared items while constraining the measurement model to conform to the unidimensional model estimated using the shared items.

In a third modeling step, the items in the MDAS and the CAM-S were linked to the DRS-R-98 using the Haebara IRT linking procedure [21]. The shared items used to equate the MDAS to the DRS-R-98 were: disorganized thinking, orientation, perceptual disturbance, sleep/wake cycle disturbance, delusions, and memory impairment. The shared items used to equate the CAM-S to the DRS-R-98 were: attention, disorganized thinking, orientation, perceptual disturbance, psychomotor agitation, sleep/wake cycle disturbance, and psychomotor retardation. Using factor loading and location parameters from the set of common rated items in the earlier steps causes the resulting factor scores for delirium severity to be on the same metric for each instrument. This IRT linking procedure places the item parameters on the same metric across the three instruments, using the shared items as anchors.

In a fourth modeling step, after the dichotomous version of the items in the MDAS and CAM-S were equated to the DRS-R-98 in the above modeling steps, a unidimensional CFA model for each delirium instrument was fit to the polytomous version of the items. The factor loading and the first threshold for each item were constrained to the equated item parameters from the third step, and the remaining thresholds were freely estimated. This step allows us to obtain estimates for the higher category thresholds assuming the measurement slopes and first thresholds are valid population parameters. The two-stage approach (first estimate slopes and thresholds for dichotomized items, then given these estimate thresholds for higher categories) was necessitated to stabilize the estimates in the presence of a small number of responses in the higher categories.

Once delirium severity was linked across the DRS-R-98, MDAS, and CAM-S (long form and short form), we described item characteristics for each rated item, the agreement between severity scores, test characteristic curves, and the test information or precision of each measurement model. These methods illustrate the relationship between characteristics of the items and severity scores, thereby informing the complex construct of delirium severity as measured by the present instruments.

As commonly practiced, the mean of the latent delirium severity factor from each model was set to 0 with variance 1. The CFA models used a logit link and a maximum likelihood estimator. Models were also run using the weighted least squares (WLSMV) estimator to obtain Root Mean Square Error (RMSEA) and Comparative Fit Index (CFI) fit statistics [22]. RMSEAs below 0.05 and CFIs above 0.95 indicate excellent model fit. Since models used repeated daily delirium assessments, in factor analyses we used robust estimation procedures in Mplus to address the non-independence of observations resulting from clustering on person in the repeat assessments.