The quality of working life questionnaire for cancer survivors (QWLQ-CS): factorial structure, internal consistency, construct validity and reproducibility

If cancer survivors wanted information about the study or wished to participate, they consented to being contacted by the research team. Next, all cancer survivors who agreed to participate by telephone received an informed consent form for study participation by post, which had to be signed.

Data collection took place between May 2015 and December 2015. Cancer survivors were asked to complete the preliminary QWLQ-CS in paper or digital form. The digital version of the QWLQ-CS was designed using the online survey software Fluidsurveys (SurveyMonkey Europe, Ireland 2014).

The answers on the digital QWLQ-CS were directly exported from the online survey software Fluidsurveys to the software IBM SPSS Statistics 23. The researchers entered the paper versions of the QWLQ-CS into Fluidsurveys twice. The data entry for two of every ten (20%) paper versions of the QWLQ-CS was checked by exporting the data to SPSS and calculating the margin of error. If ≥2% of the data entry was wrong, all of the paper versions were checked by a different researcher.

An important first step in testing a new questionnaire is to assess its content by determining if the variables of the construct to be measures are related. Therefore it is necessary to assess the underlying factor structure of this new set of variables with an EFA. The EFA was performed on the 104-item QWLQ-CS in seven steps (Table 2). In step 1, each item was removed if it fulfilled one of two conditions. The first condition was aimed at preventing an uneven distribution of answers, which might lead to an inability to detect any improvement or to distinguish between patients [28]. The second condition was aimed at removing non-generic items. For instance, items were removed if ≥20% of cancer survivors had answered ‘Not applicable’. Step 2 assessed the inter-item correlation matrix. Items were removed if they had extremely low correlations (<0.2) with ≥80% of the other items, on the grounds that they were not related to any of the other items and were not measuring the same construct, or if they correlated too highly (>0.9) with other items, which implied that the content of these items was too similar [29].

In order to perform the Principal Component Analysis (PCA) in IBM SPSS Statistics 23, the test assumption had to be met in step 3. The Kaiser-Meyer-Olkin test was used to assess the sample adequacy, and if this value was >0.6, the sample size was sufficient. For items to be correlated, Bartlett’s test of sphericity had to be p < 0.05 [30]. In step 4, the number of underlying factors in the QWLQ-CS was explored by analysing the outcomes on Catell’s scree test [31] and Parallel Analysis (PA). In a scree test, the number of factors are based on the break in the plot [32]. PA was used to compare the outcomes of the PCA eigenvalue of our data set to the mean eigenvalue of 100 random data set with the same number of items and sample size [33]. To determine the best fit for the rotation structure in step 5, the PCA was performed on a fixed set of factors, resulting from the scree test and PA and with various rotation methods (Table 2). Based on the rotation plots, we decided which rotation best fitted the data. In step 6, the final decision on the number of factors was made by carefully examining the number of items, their content and the items’ factor loadings on the different number of factors that had been retrieved in step 4. Items with a factor loading of >0.5 were allocated to that factor [29]. Items with a factor loading of <0.5 were removed, and a new PCA was performed after the removal of each item in order to analyse the new factor loadings of the items. For items with factor loadings of >0.3 on more than one factor, removal was discussed, because the interpretation of this item might be ambiguous [29].

Finally, in step 7 items were removed by analysing the internal consistency per factor. The internal consistency indicates the interrelatedness of the scale of the extent to which items assess the same construct [29]. Multiple parameters of internal consistency were analysed (Additional file 1). An item was deleted if it had an inter-item correlation of ≥0.7 with another item, and if it had low inter-item correlations (0.2–0.4) with half of the items in that factor. Finally, a Cronbach’s alpha between 0.7 and 0.9 was acceptable [29], with >0.9 suggesting a high level of item redundancy [28]. Therefore, items were deleted when the Cronbach’s alpha of the factor was <0.7 and >0.9. One PCA was performed to examine the stability of the factor structure.

In step 1, there were no items for which ≥95% of the responses fell into one category. However, 14 of the 104 items were removed because ≥20% of cancer survivors indicated that this item was not applicable to them (Table 2). None of the items in step 2 correlated ≥0.9 with other items, but four items did correlate ≤0.2 with ≥80% of the other items and were removed. In step 3, the PCA was therefore performed with 86 items. Test assumptions were achieved; the Kaiser-Meyer-Olkin test was 0.86 and Bartlett’s test of sphericity was significant (p < .001). In step 4, Catell’s scree test yielded four factors and PA identified eight factors. Varimax rotation was the best fit for the data in step 5. After carefully examining the number of factors resulting from the scree test and PA, the number of items, their content and the items factor loadings, a five-factor structure was determined in step 6. We removed 21 items because they had a factor loading below 0.5, and two items that showed overly high (above 0.3) loadings on other than their main factor. In step 7, 40 of the 63 remaining items were deleted based on inter-item correlations of ≥0.7, inter-item correlation between 0.2–0.4 of multiple items, or the scale’s Cronbach’s alpha of <0.7. This resulted in a total of 23 items that were divided into the subscales: 1) Meaning of work, 2) Perception of the work situation, 3) Atmosphere in the work environment, 4) Understanding and recognition in the organisation, and 5) Problems due to the health situation. These five factors explained 51% of the variance and the QWLQ-CS had good internal consistency (Cronbach’s alpha = 0.91). The Cronbach’s alpha of the subscales varied between 0.83 and 0.86 (Table 3).

Data were collected between March 2016 and April 2016. At baseline, cancer survivors were asked to complete the questionnaire that comprised the QWLQ-CS, the Copenhagen Psychosocial Questionnaire subscales (COPSOQ) [34], the return-to-work self-efficacy scale (RTW-SE) [35], the 36-Item Short Form Health Survey subscale (SF-36) [36, 37], three items measured on a Visual Analogue Scale (VAS), demographic items, and health- and work-related items. At the four-week follow-up, cancer survivors were asked again to fill in the QWLQ-CS and two anchor questions. Employed people without cancer or other physical/mental limitations affecting their job performance also completed a paper or digital questionnaire comprising the QWLQ-CS, demographic items and work-related items.

The sample at baseline consisted of 130 cancer survivors (Table 1). The score average and standard deviations on the QWLQ-CS and its subscales are displayed in Table 5.

Of the sample at baseline (N = 130), 100 cancer survivors completed the questionnaire at follow-up (23% lost to follow-up). Eighty-seven cancer survivors who indicated no change in response to the two anchor questions were allocated to the stable subgroup and their QWLQ-CS and subscales scores are displayed in Table 5.

The single measures ICC_agreement for the overall QWLQ-CS and subscales ranged between 0.57 and 0.88 (Table 5). Subscales 2 ‘Perception of the work situation’ and 3 ‘Atmosphere in the work environment’ did not have an ICC_agreement ≥ 0.70. The level of agreement, which we assessed by SEM_agreement, ranged between 9.59 and 13.89, except for subscale 5 ‘Problems due to the health situation’, which scored a higher SEM_agreement of 24.17. The mean differences of the overall QWLQ-CS score at baseline and follow-up did not statistically differ from zero (p = 0.694). The Bland and Altman plot (Fig. 1) displays the LoA with the means of baseline and four-week follow-up for the QWLQ-CS and the differences between these two measurements between the 95% confidence interval.

No cancer survivor had the lowest (0) or highest (100) possible overall QWLQ-CS score. The percentages of cancer survivors that scored the lowest or highest possible score on the subscales were <15%, so there were no floor or ceiling effects.

In field study I we adequately performed EFA because the QWLQ-CS and its subscales had good internal consistency. In field study II we concluded that convergent validity was also good, although one hypothesis about correlations between a QWLQ-CS subscale and an existing subscale was not confirmed. The QWLQ-CS subscale 1 ‘Meaning of work’ had a low correlation with the COPSOQ subscale ‘Meaning of work’ (0.34). However, we followed the assumption that convergent validity is adequate when ≥75% of the hypotheses were confirmed [42], furthermore, the other correlations ranged between 0.53 and 0.70, which are moderate to strong correlations (r > 0.40) [29]. A possible explanation for the low correlation is that the COPSOQ subscale probably measured another construct than the QWLQ-CS subscale did. Although the items in both subscales looked similar, the latter subscale had included one different item: ‘Do you feel motivated and involved in your work?’. This might indicates a construct related to motivation and involvement in the organisation as well, whereas QWLQ-CS subscale 1 ‘Meaning of work’ does not. We recommend future research to study the convergent validity of this QWLQ-CS subscale with a different questionnaire that measures the same construct. For instance, the ‘Dedication’ subscale of the Utrecht Work Engagement Scale (UWES) [47].

Discriminative validity was assessed between cancer survivors and employed people without cancer or other physical/mental limitations affecting their job performance. There were statistically significant differences between the scores of the overall QWLQ-CS and subscale 5 ‘Problems due to the health situation’ for the two groups, with cancer survivors having a lower QWL score. This outcome is not surprising, as cancer survivors experience challenges in employment [8], which might influence QWL. However, cancer survivors do not always negatively differ from control groups. For instance, the quality of life of male cancer survivors (e.g. germ cell tumours) was similar to that of age-adjusted men [48]. Perhaps cancer survivors face more health issues at work (which might influence QWL) than in other areas of their lives that are influenced by quality of life. Furthermore, it seems that only health-related problems lower the QWL of cancer survivors. The scores of cancer survivors and employees without cancer did not differ on the subscales that contain generic items, only on the disease-specific items. Therefore, it might be interesting to study if the QWLQ-CS is also a valid QWL questionnaire for healthy employed people.

The reproducibility of the overall QWLQ-CS score was adequate with an ICC_agreement of 0.84, when ≥0.70 is acceptable for use at a group level [42]. However, subscale 2 ‘Perception of the work situation’ yielded a lower ICC_agreement of 0.57. Perhaps, this is caused by the homogeneity of the sample in regard to this subscale. By including cancer survivors with different backgrounds we assumed to have composed a heterogeneous sample. However, most cancer survivors had the same job for a long time, which might have influenced these items about self-efficacy, and made it difficult to distinguish between QWLQ-CS scores. Another parameter that determined the reproducibility of the QWLQ-CS was the SEM_agreement, which ranged between 9.59 and13.89 (range 0–100) for the subscales. This is not uncommon, these SEM values are similar to quality of life outcomes on the SF-36 scale among COPD patients [49], and the SF-36 has been widely used because of its good psychometric properties. However, subscale 5 ‘Problems due to the health situation’ yielded a very high SEM_agreement of 24.17, which suggests that the repeated measures on this subscale for cancer survivors are far apart, and it is more difficult to achieve accuracy on the ‘true’ score and measure clinically important changes. A possible explanation for this high SEM is that the sample of cancer survivors differed in the experience of health-related problems, which may be a consequence of including cancer survivors who were diagnosed between 0 and 10 years ago. To test this assumption we should analyse the reproducibility of the QWLQ-CS only among cancer survivors who are diagnosed <1 year ago.

In sum, the reproducibility of the QWLQ-CS at group level is adequate. However, to use the QWLQ-CS at individual level, the reproducibility should be improved. For instance, by enhancing the true variance, which can be done by improving the scale design [28]. In regards to the QWLQ-CS, this would imply that the descriptions on the scale could be modified [28] or the number of response categories could be enhanced [50].

One strength of field study I is that we reduced the number of items in a systematic manner by performing an EFA. The steps taken in the EFA were statistically grounded, and decisions were made based on pre-set rules and extensive deliberation within the research group. By including cancer survivors with different demographic, health- and work-related backgrounds we developed a questionnaire with good psychometric properties that adequately measures QWL of different cancer survivors at group level.

Unfortunately, the results are less representative for cancer survivors with a different ethnic background, due to the lack of ethnic variety in the samples. Research in the USA revealed that racial or ethnic minorities of women with breast cancer were more likely to stop working compared to white women with breast cancer [51]. Perhaps this was also the reason for the lack of variety in ethnicity in our sample. However, the design of our study might also have led to non-participation: a systematic review on cancer survivorship research among minorities has shown that when working with minority populations, it is advisable to work inside the community and to draw on the help of respected leaders and involve minority members at every stage in the process [52]. This was not the case for our recruitment strategies, and therefore the results are not generalizable to cancer survivors with a different ethnic background. The implication for future research is that it is necessary to extend recruitment beyond hospitals and to recruit within the communities.

A methodological consideration concerns the number of factors in the QWLQ-CS. We used a combination of the scree plot and PA because previous literature suggested that there is no ‘golden standard’ [53]. The scree test provides a clear overview, but PA has proven to be more accurate in determining the number of underlying factors [53, 54]. Therefore, in case of any discrepancy, the final decision should be based on PA [54]. Ultimately, the two methods resulted in a unanimous number of factors: scree plot (N = 4) and PA (N = 8) with varying content and number of items within the factors. For instance, some factors in the PA contained a small number of items (N = 2), which is not desirable; a minimum of three items per factor is considered ideal [29]. Although no consistent evidence exists, it is suggested that PA might lead to overfactoring in case of smaller factor loadings and sample sizes [55]. Our sample size was not expected to be a problem as we had a ratio of 3.5 subjects per item, and a ratio of 4 subjects per item is recommended with a sample minimum of 100 [29]. However, QWL is a multi-dimensional construct and consists of items with very different contents which might lead to more low factor loadings. Eventually, we decided to determine the number of factors based on the outcomes of the scree test and PA, not only on the number of items, content and factor loadings. Therefore, we assume that the EFA did not result in an underfactoring or overfactoring of the number of factors in the QWLQ-CS. The adequate results of the psychometric properties of the QWLQ-CS support this conclusion.

The QWLQ-CS is ready for use in clinical and occupational healthcare and research settings at a group level. As the QWLQ-CS was developed in Dutch we do advise cross-cultural testing if used in other countries. Because the QWLQ-CS is a self-administered questionnaire, it is easy to use in practice and research. The QWLQ-CS can be used in studying differences in QWL between groups of cancer survivors. Previous research found that return to work of head and neck cancer survivors was associated with oral dysfunctions, and problems with social eating and social contacts [56]. It might be possible that different diagnoses lead to difficult problems at work that subsequently influence QWL. These results could be helpful in the development of tailored interventions aimed at return to work or work continuation of cancer survivors. In organisations with many employed cancer survivors, for instance, but also in research settings where rehabilitation interventions are evaluated at a group level and QWL may function as an additional research outcome. Furthermore, as some items such as ‘Because of my health situation I have problems in my work with fatigue and/or lack of energy’ seem to be relevant to other health problems besides cancer, it would be interesting to study if the QWLQ-CS can be applied to employees with other chronic diseases as well. Finally, it would be interesting in future research to measure the responsiveness of the QWLQ-CS evaluating whether it is possible to measure clinical changes in QWL which increases the applicability of the QWLQ-CS in practice.