Creation and validation of the 4-item BriefPCS-chronic through methodological triangulation

A detailed, line-by-line qualitative interpretation of each item and response option was reviewed by two authors who were not involved with the original design of the tool (DW, SM) to provide independent perspectives on the fit of the items with routine clinical practice and the conceptual basis of catastrophizing. Discrepancies in perspectives were resolved through discussion. Items were flagged at this point for reasons of possible mismatch between conceptual theory and wording. These reasons included: being unable to form a meaningful sentence through combination of item (stem) and response options, items that contained multiple concepts (e.g. were ‘double barrelled’), or apparent redundancies. The content of each item was also mapped to the original theoretical framework of catastrophizing as an exaggerated or irrational appraisal of pain, as described by Sullivan [3]. No items were removed at this stage, rather the results of the conceptual analysis were considered alongside the quantitative analyses to arrive at final decisions regarding retention/removal.

Corrected item-total correlations (ITCs) were evaluated through creation of an intercorrelation matrix where each item served as the independent variable and the summed score from all remaining items was the dependent variable. A hypothesis to drive the analysis was that all items should individually be significantly correlated with the summed score from all other items, aligning with our priority of creating a brief tool that would correlate well with the original full version. Out of respect for the well-established value of the PCS we introduced methods to avoid spurious findings or premature removal/retention of an item through use of three independent random samples drawn from the database. Each independent sample comprised n = 130 responses (10 subjects per item). An a priori requirement of a minimum corrected ITC of r = 0.70 for each item across all 3 independent samples was set as a threshold for retention, thereby reducing the likelihood of removing an item by chance. Prior to removal the items were compared against the qualitative interpretation in step 1 and only those that were flagged in both analyses (clearly problematic) were removed at this stage as a first pass scale shortening.

Rasch analysis using the remaining items was conducted in accordance with standard practice using RUMM2030 (RUMMLab, Australia) software. Rasch is a probabilistic modeling approach that assumes location of each respondent (termed ‘persons’) on the continuum of the latent construct can be predicted by virtue of knowing that person’s responses to each item, and that response to each item can be predicted by knowing that person’s location on the construct [23]. Any deviations between predicted and observed scores are termed ‘misfits’ and can be further explored through detailed analysis. Rasch analysis allows exploration down to the level of each item/response combination, and provides evidence to support appropriate ordering of response options (termed ‘thresholds’), targeting of each person to the scale, unidimensionality, and differential item functioning (DIF). We followed the general approach of Pallant and colleagues [24] using the partial credit model. Model fit was primarily evaluated through a corrected chi square test in which predicted and observed scores were evaluated for agreement beyond chance. Where significant misfit was identified detailed exploration was conducted. Response thresholds were explored for proper ordering using probability curves. DIF was explored to determine the effects on predictive accuracy influenced by the following variables: sex (male/female), cause (traumatic/insidious), medicolegal status (involved/not involved), and age (< 55 years/55 years or older) through an ANOVA-based approach that stratifies the sample based on level of DIF variable, then compares the residuals between levels of stratification to identify potential differences in scale function. Unidimensionality was explored by creating two separate scales based on factor analysis of residual (error) terms, then comparing predicted location on the two subscales using paired t-tests. If the difference in predicted person locations was not different between the two subscales, adequate unidimensionality was assumed. This is identical to approaches we have used previously for Rasch analysis of the PCS [13], and readers are directed to the prior work for more detailed description of the analysis. Where problems were identified, the research team considered the best approach to each based on the nature of the problem and the goal of the study. If modifications were made the scale was retested using the existing data and a separate independent cohort drawn from the larger database. Upon arriving at an appropriate scale the person separation index (PSI) was calculated as an indicator of internal consistency, with thresholds of PSI ≥ 0.80 considered adequate for person-level comparisons and PSI ≥ 0.70 adequate for group-level comparisons [23]. Sample size for Rasch analysis is not easily estimable, with published manuscripts describing samples of 100 to 1000 participants. As the primary statistic (chi square) is highly sensitive to sample size, two random draws of 250 responses each (n = 500 total) were extracted from the database, safely above the minimum needed to avoid random bias while below a level at which a significant chi square would be difficult to avoid.

As a second step to validate unidimensionality from another perspective, confirmatory factor analysis (CFA) was conducted from a Classical Test Theory-based perspective using MPlus v6.2 software (Muthen & Muthen Inc.). The goal was to arrive at a set of items that loaded adequately on a single latent construct, but two- or three-factor solutions were considered (where appropriate) in keeping with the factor structure of the original PCS (rumination, helplessness/hopelessness, magnification). An independent random sample of 500 responses was extracted from the larger database for these analyses. Model fit (with acceptable thresholds) was explored through the comparative fit index (CFI, > 0.95) [25], the Tucker Lewis Index (TLI, > 0.95) [25] and root mean square error of approximation (RMSEA, < 0.07) [26]. Modification indices were used as necessary to identify any potential modifications for improving fit (e.g. correlated error terms), always with the goal of a single-factor scale.

Following the triangulation procedures (conceptual, Rasch, CFA), the remaining items were considered a short-PCS and evaluated against the original 13-item version for comparison of scale function. A sample of n = 400 responses was deemed adequate for estimating correlations with narrow confidence limits for comparison purposes. Bootstrapped Pearson’s r correlations with 95% confidence intervals were calculated for the association between the original and shortened version(s) with a minimum correlation of r = 0.90 considered acceptable agreement between the two versions. Associations with secondary variables were calculated for the Beck Depression Inventory – II, and the Physical and Affective Interference subscales of the Brief Pain Inventory. Correlation estimates that were within the 95% confidence interval of the original were considered statistically similar, lending support to construct validity.

ITCs and correlations were conducted using SPSS v22 (IBM Inc., Chicago USA).

On conceptual review of all PCS items, four were flagged due to potentially difficult or ambiguous wording. PCS item 1 ‘I worry all the time about whether the pain will end’ was flagged as all the time is also a response option, effectively leading to a potentially ambiguous response of “I worry all the time about whether the pain will end all the time”. PCS item 2 ‘I feel I can’t go on’ was also flagged on both conceptual and safety grounds. While this item may be informed by the latent construct of catastrophizing (if interpreted as “I feel I can’t go on with my pain this severe”), it is also potentially informed by depression and suicidal ideation. Our anecdotal experience indicates that clinicians rarely consider this potential when administering and interpreting the scale, leaving the clinician vulnerable to documented evidence of possibly undiagnosed depression or suicidality. Two items were flagged as double-barreled (items 3 and 4) that could potentially lead to ambiguity where, for example, a respondent may describe the experience as terrible but not necessarily feel that it will never get any better (item 3). Item 6 ‘I become afraid that the pain will get worse’ was flagged as the only item in the scale that presented a future-oriented rather than current perspective. Items 5 ‘I feel I can’t stand it any more’ and 13 ‘I wonder whether something serious may happen’ were both flagged owing to the clear allusion to serious or severe problems that was arguably a poor fit to the severity-based response options; the team were unsure whether it made sense to expect a response “to a slight degree” for either of these items. These 9 items were flagged but not yet removed. The remaining items mapped adequately to the theoretical construct of ‘catastrophizing’, though some potential for redundancy was noted by the team, expected to arise in the subsequent statistical evaluations.

Table 1 provides the characteristics of subjects in the overall database from which random draws were pulled for the different components of this analysis.

Table 2 shows the 3 corrected item-total correlation matrices constructed from 3 independent random samples of n = 130 each. Using our algorithm of requiring a minimum corrected item-total correlation across the 3 independent samples of r = 0.70 for each item, and in consideration of the prior conceptual review step, items 1 (‘I worry all the time about whether the pain will end’), 2 (‘I feel I can’t go on’), 7 (‘I keep thinking of other painful events’), 8 (‘I anxiously want the pain to go away’), 12 (‘There’s nothing I can do to reduce the intensity of the pain’) and 13 (‘I wonder whether something serious may happen’) were removed at this stage. Items 3 and 4, flagged during the conceptual review, were retained due to high ITCs. The remaining 7 items were carried forward for Rasch analysis.

Partial credit Rasch modeling was conducted using the first random sample of 250 subjects. The 7-item model showed significant misfit to the Rasch model (χ² = 42.65, p = 0.003). Item 6 had a fit residual of 2.93 logits, while no other item showed a residual of > 1.29 logits. Removal of item 6 (‘I become afraid that the pain will get worse’), also flagged in the conceptual review, led to acceptable fit (χ² = 21.58, p = 0.25). Items 5 and 10 then revealed significant DIF by sex, and consistent with the conceptual review, item 5 (‘I feel I can’t stand it anymore’) had a single disordered response threshold where option 1 was never more likely to be chosen than the remaining options. Removal of item 5 led to significantly improved fit again (χ² = 14.38, p = 0.50, PSI = 0.83), and resolved the DIF of item 10. Paired t-test revealed adequate unidimensionality. Repeating the analysis with a second randomly-drawn independent sample led to nearly identical fit results (χ² = 17.30, p = 0.30). The histogram in Fig. 1 shows the logit-transformed person and item location distributions. The remaining 5 items (3, 4, 9, 10, 11) were moved onto CFA.

In another independent random sample of 500 (n = 494 after removal of missing responses), despite evidence of unidimensionality in Rasch modeling, CFA fit indicators revealed unacceptable fit by virtue of high RMSEA when the 5 remaining items were loaded on a single latent factor (CFI = 0.99, TLI = 0.98, RMSEA = 0.19). Model fit reached acceptable thresholds when either a 2-factor (Hopelessness/Helplessness and Rumination) model was tested (CFI = 1.00, TLI = 1.00, RMSEA = 0.05) or when the item with the largest residual (item 3) was removed leaving items 4, 9, 10, 11 (χ²= 3.33 p=0.19, CFI = 1.00, TLI = 1.00, RMSEA = 0.05). As item 3 was also flagged in the conceptual analysis and because of the risk of over-fitting the CFA model by virtue of perfect CFI and TLI, we returned to Rasch modeling to explore model fit with that item removed. Fit to the Rasch model remained acceptable (χ² = 20.26, p = 0.06). With only 4 items the Rasch-based PSI suffered (PSI from 0.83 to 0.76) but remained adequately reliable for group-level use [23]. Revisiting the overall purpose of creating a shortened scale that could be statistically interpreted as a single summative score, the 4-item version satisfied that purpose from newer (Rasch-based) and Classical (CFA-based) perspectives and was in accordance with the conceptual review. Fig. 2 shows the factor loadings of the 4 retained items. The 4-item version was therefore retained for concurrent analyses.

Table 3 shows the bivariate correlations between the full (13-item) and shortened (4-item) versions of the PCS using using an independent random drawing of n = 400 responses. The correlation between the versions was nearly perfect for the sample (r = 0.94). Random replacement bootstrapped correlations with 95% confidence intervals revealed identical associations between the short and full PCS versions and each of external metrics: BDI-II (r = 0.50 full, 0.46 short), BPI Physical Interference (r = 0.47 full, 0.43 short) and BPI Affective Interference (r = 0.58 full, 0.56 short). In all cases point estimates for Pearson’s r were slightly lower for the short version compared to the full, but in no case were the differences greater than r = 0.04 points and none were statistically different by virtue of overlapping 95% confidence intervals.