Development of a new occupational balance-questionnaire: incorporating the perspectives of patients and healthy people in the design of a self-reported occupational balance outcome instrument

We conducted a mixed-methods study that started off with qualitative analyses of the life stories of people with and without chronic autoimmune diseases to identify components of occupational balance. Based on these components, we developed the Occupational Balance-Questionnaire (OB-Quest). A German version was designed first and then forward and back translated into English according to standard translation procedures[14]. Construct validity and internal consistency of the OB-Quest were examined in quantitative data using Rasch analyses and Cronbach’s alpha. This study was part of a larger study, the Gender, Occupational Balance and Immunology (GOBI) study[15]. A flow chart is depicted in Figure 1.

Patients of two outpatient clinics of the Medical University of Vienna, Austria, diagnosed with Crohn’s disease (CD)[16], rheumatoid arthritis (RA)[17], systemic lupus erythematosus (SLE)[18], systemic sclerosis (SSc)[19] or diabetes mellitus type one (T1D)[20] were asked to participate. Only the qualitative data from patients with CD and RA were used from previous studies which employed the same methodological approach[11, 15, 21]. Additionally, healthy people were asked to participate via personal invitation by patients, such as "friends" of a similar age, and announcements in public places such as supermarkets, universities and the general hospital. Sex, age, employment status and health condition, if applicable were recorded. Furthermore, information about disease duration was obtained from patient files where appropriate.

A biographical narrative approach was used to explore the experience of occupational balance. The participants’ own perspectives on their lives, expressed in their life stories, were investigated in relation to current and biographic experiences[22]. The life stories, collected in two interview sessions, were transcribed verbatim and analysed with the biographic narrative interpretative method[22]. With this method, we compered the "lived life" (the biographical data) to the "told story" (the present perspective of the interviewee on her or his life). Hypotheses were developed by a research panel which were then verified or falsified depending on the course of each life story. Through this process, so-called typologies, which were common themes found in more than one life story, were identified. Typologies related to occupational balance were then extracted, defined as components of occupational balance and used as basis for the development of the questionnaire items.

Each component resulted in one item. Items were generated by the first author in collaboration with patients and healthy people[4]. After this initial process a final version of the items was formulated based on the feedback of additional patients and healthy people and on the discussion of the research panel (TAS, BP, AB and MD). Patients from the rheumatology outpatient clinic and internal medicine ward, as well as healthy people ("friends" of a similar age and visitors) were invited to give feedback. Each item included a numerical rating scale consisting of three response categories. "1" indicated a positive score, such as "having a good variety of activities", and "3" indicated a negative score such as "having little or no variety of activities". In addition, a German version was designed first and then the English version was developed by the use of a standard methodology with forward and back translation and proof reading by a total of four English or German native speakers (PG, AJ, LL and VN-D)[14].

Construct validity and internal consistency of the OB-Quest were explored. Therefore, patients with RA or SLE and healthy people completed the OB-Quest, as well as a questionnaire on demographic data. The Statistical Package for the Social Sciences (SPSS)[23] was used for descriptive and RUMM 2030[24] for Rasch analyses. Concerning construct validity we examined overall fit of the items to the Rasch model as suggested by Tennant et al.[25]. Therefore, the mean item log residual test of fit as well as the item-trait interaction chi-square statistics were assessed[25]. Non-significant residuals between -2.5 and +2.5 were interpreted as item fit, non-significant item-trait interaction chi-square values as overall fit[25‐27]. Additionally, we calculated potential differential item functioning (DIF) of sex, age (above and below the median) and health condition (RA, SLE and healthy) concerning construct validity. Furthermore, we used an approach for unidimensionality testing proposed by Smith[28], namely the combination of principal component analysis (PCA) followed by a series of t-tests to assess if subsets of items result in different estimates of person parameters. As prerequisites for the t-tests, "easy" and "hard" subsets of items were selected based on PCA. The sets of items whose residual factors loaded most strongly (positively or negatively) on the first principal component factor were used because these are most likely to violate the assumption of unidimensionality. Easy items were defined as having fit residuals that loaded negatively on the first component[29]. We used Bonferroni adjustment for multiple testing regarding the level of significance of the results of the Rasch analyses. Internal consistency is an estimate of an instruments’ reliability and was assessed with Cronbach’s alpha (α) and Rasch reliability statistics (person separation index, PSI). PSI refers to the reproducibility of relative measure location. A high PSI value (≥ 0.7) is preferred, since this indicates a high probability that people with enhanced performance will achieve high measures (sensitivity) and vice versa[30, 31]. We interpreted Cronbach’s α of ≥ 0.9 as excellent, ≤ 0.89 and ≥ 0.8 as good, ≤ 0.79 and ≥ 0.7 as acceptable and ≤ 0.69 and ≥ 0.6 as questionable[23].

Participants were informed about study procedures, and confirmed their voluntary participation with written and oral informed consents. Furthermore, we guaranteed confidentiality and changed the names in the given quotes. Approval of the Ethics Committee of the Medical University of Vienna, Austria, was obtained.

Ninety people participated in the qualitative part of this study (15 people each with CD, SLE, SSc, RA or T1D and 15 healthy people). The data of 251 people (132 people with RA, 43 with SLE and 76 healthy people) were collected in the quantitative part of this study. Demographic characteristics of the participants are shown in Table 1.

Eight components of occupational balance were identified in the analysis of the qualitative data, as described in Table 2 in the left column. In the following we give one example of how the components relate to the transcribed data. Sabine was diagnosed with systemic sclerosis at the age of 59 years. Sabine maintained occupational balance through a balance of challenging and relaxing activities:

This year I was tandem parachuting four times, out of an airplane of 4000 meters height (first interview, lines 170-171). I love such challenges. I need something like this [parachuting, long distance motor cycling] in addition to my daily routine (first interview, lines 223-224).

The eight components of occupational balance identified in the qualitative analysis were used for the development of the questionnaire items, as shown in Table 2.

The first draft of the OB-Quest was piloted in 20 additional patients and healthy people. Based on their feedback some questions were reworded. The item activities intended to care for oneself and for others was deleted because patients complained that this question would not be relevant for those who did not care for others. The same applied to the example changed family circle in the item on the component adaptation of activities according to changed living conditions. Finally, seven items were formulated as presented in Table 2, second column.

The results of the overall fit, item and fit statistics referring to construct validity are shown in Table 3.

The following sets of items whose residual factors loaded most strongly were used for t-tests: variety of activities, adaptation of activities according to changed living conditions, impact of own health condition on activities, involvement in stressful and fewer stressful activities (negative loading), and challenging and relaxing activities, activities with acknowledgement by the individual and by the sociocultural context and rest and sleep (positive loading). The result of the t-test disproved the equivalence of test scores within the OB-Quest (t = 0.58, non-significant) and thus confounded our hypothesis on unidimensionality of the questionnaire. The item on the component rest and sleep showed DIF regarding the categories health condition and age. Furthermore, PSI was 0.51 and Cronbach’s α 0.38, revealing low internal consistency. Therefore, we decided to remove the items on the components challenging and relaxing activities and adaptation of activities according to changed living conditions.

After deleting these items (components challenging and relaxing activities and adaption of activities to changed living conditions) the t-test and Cronbach’s α changed. The new results of the t-test (t = 1.66, non-significant) were based on the use of the sets of items on the components involvement in stressful and fewer stressful activities and impact of health condition on activities (positive loading), activities with social acknowledgement, rest and sleep and variety of activities (negative loading). Furthermore, Cronbach’s α changed to 0.57 indicating an improved internal consistency.

Because the item on the component challenging and relaxing activities confounded the unidimensionality of the OB-Quest, we decided to split this item into two concepts: too much and too little demand. Consequently, the two new items read now as follows: "1. Do you generally find your activities in your everyday life under-demanding? 2. Do you generally find your activities in your everyday life over-demanding?" Additionally, the item on the component adaptation of activities to changed living conditions misfitted the model. Accordingly this item was split into two concepts also, as shown in Table 2 right column. Furthermore due to DIF (as described above), the item on the component rest and sleep was divided into two items: one on rest and the other on sleep. Additionally, several changes were made based on the feedback of patients, e.g. particular words were reformulated (difficult to understand and/or score; Table 2). The final OB-Quest is proposed as ten items addressing seven components. Subsequently, the revised German version was forward and back translated; the final English and German versions of the OB-Quest are presented in Tables 4 and5, respectively.

The development together with patients and healthy people, as well as the base on experiences of people with and without chronic autoimmune diseases strengthens the questionnaire’s construct validity[5, 13]. Additionally, this process followed common recommendations for the development of self-reported outcome instruments[5, 50] and resulted in the identification of new components of occupational balance. Cronbach’s α or the internal consistency of the questionnaire turned out to improve by the reduction of the items on the components challenging and relaxing activities and adaptation of activities according to changed living conditions. In the current study three of the four native speakers were without a health professional or patient perspective and conducted the forward and back translations of the different questionnaire versions. Another study could include patients of various diagnoses according to the International Classification of Diseases[19]. Moreover, a translation into different languages and an examination of the instrument’s psychometric properties in different countries all over the world, could allow an evaluation of its culture fairness. Additionally, a conceptual definition of occupational balance would be of great value and importance. We extracted components of occupational balance from qualitative data. However, we did not aim to provide a conceptual definition of occupational balance. The conceptualization of a concept requires a complex process of synthesizing the findings of several studies[51]. We suggest further studies on the conceptualization of occupational balance.