Psychometric properties of the Children’s Response Styles Questionnaire in a Hong Kong Chinese community sample

Information on the study and parent/guardian consent forms were distributed to Grades 4-6 students of seven public primary schools in Hong Kong between March 2009 to September 2012. Only students who were able to read and write Chinese were included in the study. A final sample of 581 children (280 boys and 301 girls), with ages between 8 to 14 years (M = 10.6, SD = 1.1), was secured with written parental consent as well as students’ assent before data collection. This sample size was estimated to be adequate for performing factor analysis and IRT analysis based on the 25-item CRSQ [39]. In each school, the Chinese version of the CRSQ questionnaire was administered by a research assistant in group format, which the participants took less than 20 min to complete. The questionnaire was filled in anonymously, and participating children were advised to ask the research assistant if they had any questions about the items. Each questionnaire was checked if there was any missing items at the time of submission. Participants were thanked at the end of the session.

The following statistical analyses were conducted: 1) Exploratory factor analysis and confirmatory factor analysis were performed to examine the factor structure of the CRSQ. 2) An IRT analysis was carried out to evaluate the psychometric properties of the questionnaire on the item level, and 3) A IRT-based differential item functioning (DIF) analysis was conducted to detect whether the CRSQ items function applied equally well to boys and girls.

To examine the factor structure of the CRSQ in the Chinese context, exploratory factor analysis (EFA) was first performed, followed by confirmatory factor analysis (CFA). Since the CRSQ has not been validated in the Chinese context, considering the cultural variance, it was necessary to investigate its factor structure starting with an exploratory approach. The numbers of dominant factors were determined based on the point at which the eigenvalues change from a rapid and decelerating decline to a gradually flat slope in the scree plot [42‐44]. EFA was performed in SPSS 20.0 [45] using varimax rotation, assuming orthogonal constructs between the factors based on previous findings [13]. Using a factor loading of 0.30 as the cut-off [46], items with a factor loading smaller than 0.30 or having loadings on multiple factors were eliminated. Next, CFA was performed based on polychoric correlation matrix using diagonally weighted least squares in LISREL 8.8 [47].

IRT item calibration was carried out using the graded response model (GRM; [48]) in PARSCALE [49] for two main considerations: (a) classical item analysis suggested that item discrimination parameters were needed for this dataset, and (b) GRM is one of the widely used IRT models in clinical measurement where the item responses are polytomous and ordered (see e.g. [50, 51]). The degree of IRT model-data fit was evaluated by using both statistical tests of significance and graphical displays, as suggested by Swaminathan, Hambleton, & Rogers [52]. The likelihood ratio goodness-of-fit statistic G ² [53] provided by PARSCALE was used for examining model fit. For the potentially misfit items suggested by G ², they were further evaluated in graphical representation by examining the discrepancies between observed response proportions and IRT based predicted response proportions along the θ continuum.

Given gender differences for childhood depression and coping styles [1, 5], differential item functioning (DIF) analysis was conducted to examine whether the CRSQ items function equally for boys and girls after controlling for the latent trait measured by the CRSQ. The IRT-LR analysis was performed in IRTLRDIF [54]. Iterative purification procedure was applied for identifying the anchor set, which was free of DIF and used to link the two comparison groups on a common metric [50, 55]. The adjustment of Benjamini-Hochberg (B-H) procedure was adopted for controlling the Type I error rate in multiple comparisons and for determining the statistical significance of identifying DIF [50].

To examine the factor structure of the CRSQ in the Chinese context, EFA was first performed. The eigenvalues greater than 1.0 [42] were respectively 4.73, 4.05, 1.43, 1.15 and 1.05, collectively accounting for 49.63% of the total variance. Two dominant factors emerged and they together explained 35.14% of the total variance. In the two-factor solution, items 1, 3, 5, 7, 9, 11, 13, 19, 21 and 25 loaded onto one factor which reflected rumination with factor loading values ranging from .39 to .71 and items 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22 and 24 onto the other factor which reflected distraction and problem-solving with factor loading values ranging from .22 to .71. Using factor loading of 0.30 as a cut-off, items 15, 17 and 23 loaded onto both factors with similar factor loading values, and they were thus removed. In addition, items 4 and 14 had low item-total correlations (with values below .30) and were hence eliminated from the distraction and problem-solving subscale.

Next, a two-factor CFA model was employed. The same structure obtained in EFA was applied to the two-factor CFA solution. Item factor loadings were all positive, ranging from 0.36 to 0.73. The goodness-of-fit indices of the two-factor CFA (CFI = 0.92, GFI = 0.90, RMSEA = 0.07, SRMR = 0.08) suggested an acceptable fit. The correlation between the two latent factors was .02, suggesting that the two subscales were nearly independent. Therefore, IRT and DIF analyses were subsequently conducted on the individual subscales.

Tables 1 and 2 show the descriptive statistics and item-total correlations for the final 20-item version of the CRSQ-Chinese. Cronbach’s alpha was 0.82 for the rumination scale (CRSQ-R), and 0.81 for the distraction/problem-solving scale (CRSQ-DPS).

Dimensionality checking. In the rumination subscale, essential unidimensionality was supported because the first eigenvalue (3.88) was substantially greater than the second one (0.98) and the ratio of the first to second eigenvalue were approximately 4, generally accepted as evidence for unidimensionality [56]. In addition, the goodness-of-fit indices from the single-factor CFA model (CFI = .97, GFI = .97, RMSEA = .06, SRMR = .04) suggested a good fit. Item factor loadings were all positive, ranging from 0.40 to 0.73 (Table 1). In the distraction and problem-solving subscale, similarly, the first eigenvalue (3.69) was substantially greater than the second one (0.996) thereby supporting unidimensional assumption. Moreover, the goodness-of-fit indices from the single-factor CFA model (CFI = .94, GFI = .95, RMSEA = .07, SRMR = .05) indicated a reasonable fit. Item factor loadings were all positive, ranging from 0.36 to 0.64 (Table 2).

In the rumination subscale, the item discrimination parameters ranged from 0.93 to 1.86 (Table 1). The highest discriminating item was item 13 “When I am sad, I think about other times when I have felt sad” (a = 1.86), and the lowest discriminating item was item 19 “When I am sad, I think: I’m disappointing my friends, family, or teachers” (a = 0.93). In terms of item severity (Table 1), item 11 “When I am sad, I think about how angry I am with myself” (b ₁ = 0.05, b ₂ = 1.17, b ₃ = 2.02), item 5 “When I am sad, I think: I’m ruining everything” (b ₁ = 0.07, b ₂ = 1.28, b ₃ = 2.18) and item 19 “When I am sad, I think: I’m disappointing my friends, family, or teachers” (b ₁ = −0.12, b ₂ = 1.51, b ₃ = 2.78) emerged at the higher levels of severity. Similarly, their observed item means were also lower than the other CRSQ-R items. Items 3 “When I am sad, I go away by myself and think about why I feel this way” (b ₁ = −0.88, b ₂ = 0.74, b ₃ = 2.23) and item 25 “When I am sad, I think about how I don’t feel like doing anything” (b ₁ = −0.76, b ₂ = 0.94, b ₃ = 1.89) reflected lower levels of severity in the CRSQ-R subscale.

As can be seen in the item characteristic curves (ICCs) and item information functions (IIFs) respectively in Figs. 1 and 2, among the 10 CRSQ-R items, item 13 “When I am sad, I think about other times when I have felt sad” clearly offered the most information along a wide range of θ roughly from −1 to +1.5. Items 11 “When I am sad, I think about how angry I am with myself” and 21 “When I am sad, I think about all my failures, faults, and mistakes” offered greater potential for discriminating among respondents than most of the remaining items; in particular, item 11 was more useful for differentiating upper θ levels (approximately between +1.5 and +2.4), whereas item 21 provided more information for respondents with moderate θ levels (approximately between −0.5 and +1.2). Item 19 “When I am sad, I think: ‘I’m disappointing my friends, family, or teachers’”, item 9 “When I am sad, I go someplace alone to think about my feelings”, item 25 “When I am sad, I think about how I don’t feel like doing anything” and item 3 “When I am sad, I go away by myself and think about why I feel this way” − the low discriminating items − exhibited a flatter ICC, adding very little to the psychometric quality of the CRSQ-R because the power of these items to discriminate respondents with higher from lower levels was minimal.

In the distraction and problem-solving subscale, the item discrimination parameters ranged from 0.97 to 1.77 (Table 2). The highest discriminating item was item 18 “When I am sad, I try to find something good in the situation or something I learned” (a = 1.77), and the lowest discriminating item was item 8 “When I am sad, I spend a lot of time on my schoolwork” (a = 0.97). In terms of item severity, as shown in Table 2, item 8 “When I am sad, I spend a lot of time on my schoolwork” (b ₁ = −0.09, b ₂ = 1.88, b ₃ = 3.59) and item 2 “When I am sad, I help someone else with something so I don’t think about my problem” (b ₁ = −0.47, b ₂ = 1.47, b ₃ = 3.28) reported higher levels of severity (or strength of the latent trait for adopting the distraction or problem-solving coping style). Consistently, less than 5% of respondents rated “almost always” for both items, as shown in the observed response frequencies in Table 2. Given the community sample in the study, even those near the higher end of latent trait levels (e.g. θ = 3) would reach only an expected item score around the third response category (“often”), and very few cases would endorse the fourth response category (“almost always”), as shown in Fig. 2. Among those with lower level of severity in the CRSQ-DPS subscale were item 10 “When I am sad, I do something I enjoy” (b ₁ = −1.76, b ₂ = 0.22, b ₃ = 1.52), item 12 “When I am sad, I do something fun with a friend” (b ₁ = −1.00, b ₂ = 0.17, b ₃ = 1.29) and item 20 “When I am sad, I talk it out with someone who I think can help me feel better” (b ₁ = −0.98, b ₂ = 0.20, b ₃ = 1.27).

Among the 10 CRSQ-DPS items, item 18 “When I am sad, I try to find something good in the situation or something I learned” provided more information than the other nine items along the wide range of θ continuum approximately between −1 and +2.5. Items 8, 6, 24 and 2 had lower discriminating parameter estimates and thus offered less information than the other items. Between items 16 “When I am sad, I ask a friend, parent, or teacher to help me solve my problem” and 22 “When I am sad, I think of a way to make my problem better” with similar a parameters, item 16 was slightly more useful than item 22 in discriminating among the respondents with moderate latent trait levels (θ between around −0.5 and +1), whereas item 22 offered slightly more useful information in discriminating among the respondents in the higher end and the lower end of the θ continuum.

In addition to the item level, test information and standard errors of measurement along the θ continuum were produced in Fig. 3. The CRSQ-R subscale and CRSQ-DPS subscale offered the most useful information at the θ value around 1.2 and 1 respectively (approximately one standard deviation above the mean). Specifically, the CRSQ-R was more informative in assessing respondents along the θ continuum approximately between −1 and +2 (between one standard deviation below the mean and two standard divisions above the mean), and CRSQ-DPS was more informative in assessing respondents along the θ continuum approximately between −0.5 and +0.5 (between half standard deviation below the mean and half standard deviation above the mean). The performance of test information suggested that the both subscales were informative in assessing a broad range of the latent trait, whereas the CRSQ-R offered great potential in assessing the higher level of the latent trait, and it hence can be particularly useful for screening high risk individuals with elevated latent trait levels.

Furthermore, the degree of IRT model-data fit was evaluated by using both statistical tests of significance and graphical displays. Using G ² provided by PARSCALE, items 2, 3 and 10 were detected as potentially misfit items, and they were further evaluated in graphical representation. The confidence bands of predicted item performances generally covered the observed item performances, except a few outliers in the first, second and fourth score category curves. We thus considered acceptable fit for the three items.

For factor structure, the present study yielded similar findings as those documented with a Western sample by Abela, Aydin & Auerbach [13]. Importantly, the two-factor structure identified was comparable across ours and Abela et al.’s [13], implying that the CRSQ has a similar format in Chinese school age children as in some Western cultures. In contrast to the original form of 25-item CRSQ in English [19], a 20-item Chinese CRSQ emerged from item elimination in our data analyses, with 10 items remaining in the rumination subscale and 10 items in the distraction and problem-solving subscale. The correlation between the two factors (r = .02) suggested that the constructs were nearly orthogonal in the Chinese context, which coincides with the previous findings identified in the Western samples [13, 19]. On a related note, the removal of items loading onto both factors helped to reduce the correlation between the two factors. Also, the Chinese version not only has fewer items, but item 8 “When I am sad, I spend a lot of time on my schoolwork” and item 6“When I am sad, I go to my favorite place to get my mind off my feelings” that had been deleted from the earlier 21-item version in Abela et al.’s study [13], stayed in the distraction and problem-solving subscale in the Chinese version. Specifically, these two items relate to academic performance or personal space, and this may reflect some cultural difference in which achieving well academically and getting away from a crowd could be perceived as “adaptive” problem-solving or distraction skills among Chinese youths. For internal consistency, the Cronbach’s alpha was .82 for the CRSQ-R subscale and .81 for the CRSQ-DPS subscale for this 20-item Chinese CRSQ. Similar moderate levels of internal consistency (alphas = 0.82 and 0.79 for the CRSQ rumination and distraction/problem solving subscales respectively) were reported in previous studies in the Western context [13]. The figures in both subscales suggested adequate internal consistency, with .70 as a commonly accepted minimum reliability estimate in psychological and educational measurement [57].

Our study was the first to utilize IRT methods to evaluate the item properties of the CRSQ. Our IRT analysis suggested that some symptoms reflected higher levels of severity than others. In the rumination scale, items such as “ruining everything” and “disappointing my friends, family, or teachers” emerged at the higher levels of severity, whereas items such as “go away by myself” and “don’t feel like doing anything” reflected lower levels of severity. Importantly, the more severe items are consistent with negative evaluation phenomenon commonly observed in depressive rumination among Chinese youths [18]. In Confucian societies, performing in the best interest of the group is highly emphasized [58]. With a narrowing of focused attention resulted from chronic rumination, the risk of a person feeling worthless due to failure to perform socially-oriented duties increases [18], leading to a sense of sustained helplessness and hopelessness, which can further fuel a vicious cycle of ruminative response.

In the distraction and problem-solving subscale, items such as “spending a lot of time on schoolwork” and “helping someone else with something” reflected higher levels of severity, implying that youngsters endorsing these items may be better equipped with problem solving abilities or resources to endorse action-oriented distraction strategies. Overall, the findings on the various levels of severity, may contribute to the clinical utility of the CRSQ. Individuals endorsing the same ratings on the more severe items and on less severe items are more likely to have different levels of the latent trait.

Results from the IRT analysis also suggested that some symptoms are stronger discrimination indicators than others. For instance, the item “thinking about other times when I have felt sad” in the rumination subscale and the item “trying to find something good in the situation or something I learned” in the problem-solving/distraction subscale were found to have higher discriminating power. Clinicians may do well to pay special attentions to these discriminating symptoms because they are particularly robust in differentiating varied levels of the latent trait ranging from low to high. Interestingly, some items such as “disappointing my friends, family, or teachers” in the rumination subscale and “spending a lot of time on schoolwork” in the problem-solving/distraction subscale showed the lowest discriminating power. We again suggest that this may be related to collectivistic cultural reasons. To consider expectations from others and achieve well academically (i.e. studying for lengthy periods) are commonly highlighted as the primary “responsibility” for youths in Asian Confucian societies. These imperative cultural values have been internalized and deeply-rooted in their mindset, thus making the suggested items comparatively insensitive and less discriminating in differentiating response styles in such societies.

Regarding the test information, both subscales offered the most useful information at the latent trait level approximately one standard deviation above the mean (on the θ scale). The rumination subscale was more useful in assessing the latent trait at a broad range of levels from about one standard deviation below the mean to about two standard deviations above the mean on the θ scale. The distraction and problem-solving subscale was more useful in assessing the latent trait around half standard deviation below and above the mean on the θ scale. In other words, both subscales may be informative and useful in assessing the latent trait for community samples of Asian school-age children where the expected average level is not at the extreme end. It is noteworthy that the rumination subscale assessed the latent trait over a wider range of levels, compared to the distraction and problem-solving subscale, rendering the former particularly useful for screening high risk individuals with elevated latent trait levels. High ruminative tendency had been widely considered a significant risk factor for psychopathology, given its strong association with and predictive power for depressive symptoms [14, 15].

DIF analysis showed that the scale items generally apply equally well to boys and girls, except that item 1 “When I am sad, I think about how alone I feel” exhibited higher discriminating power for girls. Understandably, girls may be more susceptible to evaluating themselves by comparing how acceptable they are by others or how much social ties/ resources they have. Girls’, more so than boys’, self-concepts generally depend on interpersonal aspects [59]. Quite plausibly, this gender difference is exacerbated in collectivistic societies, as social desirability is even more deeply valued. Notwithstanding this significant gender effect in the first item, all other items in CRSQ seem to work equally well for boys and girls in assessing the ruminative trait.

We are mindful of several limitations of the present study. First, we relied on self-report measures in assessing psychometric properties of the Children’s Response Styles Questionnaire. Future studies using information from other sources or structured diagnostic interviews to corroborate self-reports will be helpful. Second, our study only recruited local children between ages 9 and 14 years. It remained to be seen whether the factor structure observed could be generalized to other age groups. Specifically, an important ongoing debate focuses on the age at which children develop stable response styles in light of their cognitive development ([19, 20]; see [60] for review). Further studies are needed to explore if the factor structure uncovered in this study will also turn out to be valid for younger or older children. Such information can have significant implications for the clinical utility of this user-friendly questionnaire as an effective assessment tool for early identification of children at risk for depression.