Reliability, validity and administrative burden of the community reintegration of injured service members computer adaptive test (CRIS-CAT)”

The 3-part study involved 1) a cross-sectional field study, 2) a one year follow-up study of OEF/OIF Veterans from the field study (cohort study), and 3) a separate 50 person study of CRIS-CAT administration. The overall organization of the study is shown in Figure 1. The methods for each sub-study are described below. All sub-studies were approved by the Institutional Review Board of the Providence VA Medical Center. All subjects provided informed consent.

The CRIS assesses community reintegration through the assessment of participation in life roles as defined by the International Classification of Functioning, Disability and Health (ICF) [15], an approach consistent with recent recommendations from the VA’s State of the Science Working Group on Measurement of Community Reintegration [16]. Items on the CRIS cover 9 aspects of participation: (1) Learning and Applying Knowledge, (2) General Tasks and Demands, (3) Communication, (4) Mobility, (5) Self-care, (6) Domestic Life, (7) Interpersonal Relationships, (8) Major Life Areas, and (9) Community, Social and Civic Life. The CRIS’s scales measure these aspects in three dimensions. The Extent of Participation scale asks the respondent to indicate how often he or she experiences or participates in specific activities. The Perceived Limitations in Participation scale asks the respondent to indicate his or her perceived limitations in participation. Lastly, the Satisfaction with Participation scale asks the respondent to indicate the degree of satisfaction with participation.

Computer Adaptive Test (CAT) methodologies use a computer interface to select and administer items tailored to the unique level of the respondent [17]. Use of CAT based measures reduces respondent burden by allowing administration of fewer items as compared to fixed form measures. CAT applications require a set of items in functional domains that consistently scale along a dimension of low to high functional proficiency in the domain of interest. In previous work we developed the CRIS-CAT by drawing upon a large item pool: Extent of Participation (77 item pool), Perceived Limitation in Participation (144 item pool) and Satisfaction with Participation (86 item pool) [16].

We employed Item Response Theory methods to develop a CAT version of the CRIS that allows for accurate, precise, and reliable measurement of community reintegration with reduced respondent burden. Our IRT models showed that the CRIS-CAT scales were unidimensional. CRIS-CAT scores were estimated from subjects’ responses to the full item pool based on the Weighted Maximum Likelihood Estimation Method [18]. The CAT stopping rules were set at a minimum of 10 items and a maximum of 20 items and a SE of the latent trait set to 0.32. Reliability of the CAT under these conditions was equal to 0.9. Our data simulations showed that the CRIS-CAT scales had good fit, and had excellent precision as compared to the full item set [16]. These findings suggested that the CRIS-CAT was a sound measure of community reintegration with reduced respondent burden.

We collected data on measures which we believed would be correlated with CRIS-CAT scores if the instruments were measuring community integration. We expected to observe similar relationships to those that we found in our earlier study of the CRIS fixed form measure [14].These measures included two scales from the Craig Handicapped Assessment and Reporting Technique (CHART) [[19], the SF-36 V, [20] and the Quality of Life (QOL) measure [21]. The CHART social integration subscale consists of 6 questions about extent of participation in and maintenance of customary social relationships [22]. The CHART occupation subscale consists of 7 questions about extent of participation in occupational activities customary to a person's sex, age, and culture [22]. The SF-36 V, a version of the SF-36 is adapted for the Veteran population [20]. Using SF-36 V responses we calculated the physical component summary score (PCS) and the mental component summary score (MCS) as well as the Physical Functioning Scale (PF-10) [23]. The Quality of Life scale (QOL) consists of 16 questions that assess satisfaction with independent living and self-care activities [21].

Other demographic information that was collected at Visit 1 included age, gender, marital status, race, ethnicity, education, income, employment, parenthood, presence of children in the home, and housing type. Marital & Relationship Status was assessed using a question from the U.S. Census Current Population Survey (CPS) which classified marital status as now married, widowed, divorced, separated, and never married. We re-categorized this variable into three groups: married, never married and other (widowed, divorced or separated). Race was categorized as Caucasian, Black, Mixed and Other. Ethnicity was categorized as Hispanic or non-Hispanic. Education was categorized as less than high school, high school diploma, GED, some college, college graduate and postgraduate education. Income was assessed using a question from the Behavioral Risk Factor Surveillance System (BRFSS) to estimate total household income in the previous year. Income was classified into three categories: less than $25 K, between $25 K and $50 K, and over $50 K. Employment was assessed by 4 questions drawn from the U.S. Census Current Population Survey (CPS) which ask if the respondent was working, a student, or participating in regular volunteer activities. Employment was classified as unemployed, not working due to disability or on medical hold, working part-time or training, and working full-time. Parenthood was defined as having children or stepchildren yes or no. Housing Type was determined by asking respondents to indicate their type of residence (outdoors, staying with friend, vet home, house, apartment, or other).

Additionally, we collected data on OEF/OIF Veteran status, whether or not the Veteran had ever been deployed, and if so how many times, and the number of months since returning from deployment. We also asked subjects to indicate whether they had ever been diagnosed with depression, alcohol/drug abuse, Post Traumatic Stress Disorder (PTSD), or other mental illness.

Data from the field study were used to examine reliability of the CAT, and test concurrent and known groups validity. We examined descriptive characteristics of field study participants, and compared characteristics for each subsample.

Reliability represents the degree to which the differences across subject scores are due to real differences in scale (true variance) as opposed to measurement error. By assuming the latent trait under the partial credit model has a standard normal distribution, the conditional reliability of the CRIS-CAT scales was examined as 1/(1 + (standard error)^2 ), which was a function of latent trait level. The standard error of the person score estimates was derived from the information function. Any section of the reliability function that was greater than 0.70 served to indicate adequate reliability [26].

Concurrent validity of the CRIS-CAT was examined by exploring Pearson product correlations of the CRIS-CAT scales with existing measures that assess specific community reintegration dimensions. We expected to find similar correlations to those reported in our earlier research on the CRIS fixed form measure. (i.e. correlations between the CRIS fixed form measure and the 36-Item Short Form Health Survey scales of role physical, role emotional and social functioning of 0.25-0.80 [13, 14] and a correlation between the CRIS scores and QOL of 0.57-0.79.) [13].

We performed ANOVAs to examine whether the CRIS-CAT scores differed in Veterans from the 3 groups as expected: the homeless group, the working group and the OEF/OIF group.

Characteristics of the field study sample are shown in Table 1. The sample consisted of 517 Veterans; 69 in Group A (those with a high degree of community reintegration), 99 in Group B (homeless Veterans) and 332 in Group C (OEF/OIF Veterans). There were 17 additional Veterans who were screened into one of the above groups, but who, on inspection of their data, were excluded since they did not meet inclusion criteria for the study.

Figures 2a, b, and c show the level of scale score conditional reliability [32] across each CRIS-CAT scale continuum. Reliability was above 0.75 for most of the scale range, but decreased at both ends of the continuum where there were fewer items available for administration.

Results of the concurrent validity analyses are shown in Table 2. All CRIS-CAT scales were strongly correlated with SF-36 Role Physical, Social Functioning, Role Emotional and Physical Functioning, and with QOL, negatively correlated with the number of difficulties in Activities of Daily Living (ADL) and moderately correlated with the CHART subscales. Known group validity analyses conducted via ANOVAs determined that CAT subscale scores differed significantly between the three sample groups (P < .000 for all comparisons) (Figure 3). Veterans in the homeless group had the lowest CRIS-CAT scores (Extent mn 45.6 ±8.9), (Perceived mn 44.7 ±7.3) and (Satisfaction mn 45.1 0 ± 7.2) and subjects in the working group had the highest scores (Extent mn 55.8 ±8.2), (Perceived mn 56.6 ±11.5) and (Satisfaction mn 54.9 ±11.0). OEF/OIF Veterans had average scores between the other two groups (Extent mn 48.7 ±9.2), (Perceived mn 49.6 ±8.6) and (Satisfaction mn 49.6 ± 9.0).

One hundred and thirty five of the 208 OEF/OIF Veterans from Visit 1 returned for Visit 2. Figure 4 shows the number of subjects who were not reached for follow-up, as well as those who were reached and the reasons that they declined to participate. Table 3 shows the characteristics of those subjects from the cohort who were lost to follow-up and those who completed the second visit. Compared to subjects who completed Visit 2, lost to follow-up subjects were significantly younger (p < 0.001), had been deployed more recently (p < 0.05), had lower incomes (p < 0.01), were less likely to have children (p < 0.05), be married (p < 0.05), or have an apartment or house (p < .01), and had fewer instances of depression (p < 0.05) and alcohol/drug abuse diagnoses (p < 0.05). One hundred twenty two of the subjects included in the cohort had data available from the VA medical record.

Only 3.7% of the cohort group reported a change in their marital status between Visit 1 and one year follow-up. Whereas 11.5% reported a change in employment status (8.4% for the worse) (Table 4). Although the majority of subjects had not moved in the prior year, 22.2% had moved once, and 8.9% had moved more than once. Contrary to our hypotheses, CRIS-CAT scores at Visit 1 were not predictive of change in marital status (not shown) or change in employment status or housing stability one year later (Figure 5). The odds of having any self-reported or abstracted ER visits were reduced for those with higher CRIS scores; Extent, Perceived, Satisfaction subscales respectively (OR 0.94, 0.93, 0.95, P < .05). (Figure 6) The odds of having any abstracted ER visits were reduced for those with higher CRIS Extent or Perceived scores respectively (OR 0.93, 0.93, .097; P < .05). (Figure 6) The odds of having a new diagnosis of a mental health condition one year later were lower for persons with higher CRIS-CAT scores at Visit 1 Extent, Perceived, Satisfaction subscales respectively (OR 0.91, 0.91, 0.90; P < 001) (Figure 6). CRIS-CAT scores at Visit 1 were predictive of SF-12 PCS and MCS scores 1 year later. (Table 5)

Characteristics of the sample who participated in the Administration Study are shown in Table 3. Scores of the CRIS-CAT scales and the number of items needed to complete them are shown in Table 6. On average, the Extent subscale required more items than the Perceived Limitations and Satisfaction subscales. ICCs of each of the scales (95%CI) were 0.947 (0.908-0.969) for Extent, 0.912 (0.850-0.949) for Perceived and 0.967 (0.941-0.981) for Satisfaction. MDC values for 90 and 95% confidence are also provided in Table 6.

Our analyses support the reliability and validity of the CRIS-CAT. The level of scale score reliability across each CRIS-CAT scale continuum was acceptable or better for most of the scale range, but decreased at both ends of the continuum. Our results will be useful for interpreting change at the individual level. Our calculations of the MDC for each scale, are useful measures for clinicians and researchers to know because, from a statistical perspective, a patient is considered to have changed only when the difference between the previous score and the current score exceeds the MDC associated with the measurements [33]. This information is helpful in interpreting changes in scores with repeat administration.

Our study of CAT administration demonstrated that the three scales have excellent test-retest reliability and were administered with reduced respondent burden as compared to the full item set and the CRIS fixed form measure [16]. Together these findings suggest that the CRIS-CAT is feasible for adoption into research and clinical practice. Further studies of implementation of the CRIS-CAT in the clinical setting are needed.

In earlier work we found that the Extent scale had the greatest breadth of items across the entire score continuum compared with the Limitations and Satisfaction scales [16]. The Satisfaction scale showed the poorest match between the distribution of items in the item pool with the score distribution from the study sample. This pattern of item distribution corresponds to the pattern of drop off in reliability at the ends of the continuum for the Satisfaction and Limitations scale but not so much for the Extent scale. These findings suggest that the Satisfaction and Limitations scale may be less responsive to change. Further research needs to be conducted to examine responsiveness of the scales, and to identify appropriate items to add to the Satisfaction scale to reduce the ceiling effect that is likely to occur. Addition of items at the upper range of difficulty is also a potential area of improvement for the Limitations scale.