Measuring subjective complaints of attention and performance failures - development and psychometric validation in tinnitus of the self-assessment scale APSA

As a first step, 99 items were collected from various sources found in the literature of studies on attention and cognitive performance [4, 5, 8‐10]. The selection of items was done with the intention of generating an inter-culturally equivalent scale. Therefore the items were to address more or less general aspects of a person’s ability to concentrate and to perform daily living activities. Both are aspects of quality of life and health [7].

Items were selected that addressed attention, concentration, disorientation, memory, continuity, distractibility, cognitive control, cognitive-emotional, mistakes, attribution and symptoms. As far as possible, we selected questions related to daily situations and that were neutral in terms of gender and education level. Items were selected that reflected cognitive mishaps and failures that were commonly recognized as such by the subject, rather than items reflecting specific situational or cultural factors. Soon after starting the collection of questions, we had the impression that specific failures might be age-related. After deleting duplicates and items with similar wording we checked the resulting item pool for face validity and further reduced it to 56 German language items, harmonized for better readability and response options. The items of this pool were tested in 44 German-speaking subjects, 4 of whom were tinnitus sufferers. The volunteers were encouraged to comment on the clarity of the questions and response options. Following their feedback some items and response options were changed. After this process, the final item pool comprised 30 items, each with a five-point response scale and a recall period of 4 weeks. The response options were: never, rarely, sometimes, often and always, with never as 0 and always as 4. The 30-item questionnaire was translated and then backward translated into 6 languages including US English and Mexican Spanish. This was followed by cognitive debriefing interviews with tinnitus patients, a clinician review of each language version and a harmonization meeting. All versions, including the German version, underwent final linguistic reconciliation using the US English edition as the master version [11].

This was a prospective, multicultural, non-interventional, observational study in adult subjects with a clinical diagnosis of tinnitus. In the USA there were 5 centers: one private practice, 2 clinical research centers and 2 ear, nose and throat (ENT) centers; in Germany there were 2 ENT specialists and in Mexico one ENT and 1 private practice. All data were collected without altering the patient’s tinnitus clinical management. Adults with a clinical diagnosis of subjective tinnitus who received care from a study physician and met the enrollment criteria were invited to partake in the study. Unfortunately it was not possible to perform stratified selection or to match the subjects from the subgroups for gender, age or educational level. The study was approved by relevant institutional review board, IRBs for the US: Quorum Review IRB; for Germany: Ethik-Kommision der Bayerischen Landesärtzekammer; for Mexico: Comité de Ética e Investigación Christus Muguerza del Parque (Chihuahua), and CimByTa, Centro Investigación Médico Biológica y Terapia Avanzada (Guadalajara) prior to subjects’ enrolment at the site. Investigators explained the study processes and procedures to the potential subjects who met the eligibility criteria. subjects were given the opportunity to ask questions and were provided (either by mail or in-person at the clinic) with a written informed consent form to review. If the patient was not interested in participating, clinical management continued as before. All subjects personally signed and dated the IRB approved informed consent form before enrollment. Quality of data was ensured by regular telephone visits as well as by the monitoring visits at the sites conducted during or after completion of the enrolment at the site. Each subject was asked to self-administer the baseline scales at home and to mail the completed questionnaire to the study center. The study center then mailed the follow-up questionnaire to the subject, to be received 10 to 17 days after completing the baseline assessment. The subject was asked to complete the follow-up assessment and to return the questionnaire by mail to the study center within 7 days. The target window for completion of the follow-up questionnaire was 14–28 days after the completion of the baseline questionnaire.

The inclusion criteria were ≥ 18 and ≤ 75 years of age; clinical diagnosis of persistent subjective, uni- or bilateral tinnitus (i.e. tinnitus could not be absent for > 24 hours at a time); present for at least 3 months; willingness and the ability to comply with the protocol and study procedures, including the ability to understand the written and verbal language of the country. The exclusion criteria were: a clinical diagnosis of intermittent or pulsatile tinnitus; tinnitus as a concomitant symptom of an otological/neurological disease; any treatment, disorder or condition that might cause tinnitus to be unstable.

Subjects were asked to provide self-assessments at baseline and 12 – 30 days later as follows:

Subjects eligible according to the inclusion/exclusion criteria and who completed at least one item on the baseline questionnaire were included in the analyses of the baseline data. Subjects who completed at least one item on the follow-up questionnaire and completed the follow-up questionnaire 12–30 days after the baseline questionnaire were included in the follow-up analyses. All data were summarized using standard descriptive statistics (mean and standard deviation for continuous data, frequencies and percentages for categorical data). To detect differences between the country samples one-way ANOVA for country and Mantel-Haenszel Test for categorical data were performed. For the single APSA items means, standard deviations and the difficulty index (defined as the percentage of subjects giving “never” answer) were calculated. Additionally, to quantify the performance of each single item we calculated probability plots with the underlying model assuming that all items contribute to one dimension.

The analysis of the APSA structure and content was begun with an initial exploratory factor analysis (EFA) [19, 20] consisting of a principal factor analysis with squared multiple correlations for the prior communality estimates, based on a common factor model. The number of factors was limited to Eigenvalue > 1 and PROMAX rotation of the extracted factors was used. This was conducted with the baseline data from 30 items of the APSA in the total population. Items with final communality (h < 0.40) were eliminated and the EFA was repeated with the remaining items. The final result of the EFA was then tested by means of confirmatory factor analysis (CFA) with the follow-up data from the total population as well as for the US, the Mexican and German samples separately using structural equation modeling [21]. A number of fit indices have been proposed [22, 23]. For our purpose (i.e. estimating the goodness of model fit in each country), the chi-squared GFI (Goodness-of-fit index) seemed to be the most appropriate.

The distribution of the resulting factorial scores was characterized by mean, standard deviation (SD), median, minimum and maximum. Internal consistency was evaluated using Cronbach’s coefficient alpha and was considered acceptable if Cronbach’s alpha was 0.70 or greater. Test-retest reliability (r) was evaluated for each scale and sub-scale by computing the intraclass correlation coefficient (ICC) using Shrout-Fleiss reliability with fixed set (3,1) [24]. Test-retest reliability was demonstrated if the ICC coefficient was 0.70 or greater. Minimally detectable change (MDC) is given by MDC = ±1.96 * √2 * SEM, where 1.96 derives from the 95% confidence interval of no change. The standard error of measurement (SEM) is given by SEM = SD * √(1-r) [25]. Convergent/divergent validity was tested with the THI-12, TSS, TRS, and HADS in the total population and each of the countries using Pearson correlation coefficients. To investigate the potential sensitivity to change from baseline to follow-up Pearson correlation coefficients were also calculated between the APSA subscales on the one hand and the THI-12 and TRS total change scores on the other.

We also included a further sample of healthy volunteers in order to estimate sensitivity (true positive/(true positive + false negative)) and specificity (true negative/(true negative + false negative)) of the APSA subscales. We used a linear logistic regression model to calculate a ROC curve to estimate these parameters.

The analyses were conducted using SAS software, Version 9.2 under the Windows operating system (copyright © 2009 SAS Institute Inc). Confirmatory factor analyses were conducted using LISREL software, Version 8.8 under the Windows operating system (copyright © 2006 Scientific Software International, Inc.). Response distribution graphics were generated with jMetrik, version 2.1.

In the majority of cases the APSA was completed without the omission of questions. Only item 17 was not answered by 2 out of 213 subjects. The item response characteristics are summarized in Table 2. The item scores ranged from 0 (minimum of scale) through 4 (maximum of scale), except for items 7 and 14 that had a maximum score of 3.

To further examine the item characteristics we plotted the probability of responses for each item (examples see Figure 1: Response distributions of bad and good performing items) assuming that all items are measuring the same concept. Most of the items showed a good separation of the answering options. Item 17 had good selectivity, but from the wording as well as from the plots it was obvious that the scoring should be reversed if used in a total score. Only the response options of items 9, 11, 18, 20 and 22 did not separate well from each other. Nevertheless we decided to include them in the following factor analyses to see how they would perform.

Further, we wanted to know which of the 30 APSA items were age, gender or education dependent. The results of the general linear model analysis (GLM with factors gender and education level and covariate age using Bonferroni α-correction to control the overall error-type I level of 5%) for the APSA items in the working-age population show that in the USA sample three models are significant. The following effects were observed: Item 15: gender and age; item 16: educational level; item 23: gender and educational level. With all other item we did not observe statistically significant effects of age, gender and/or educational level.

An initial exploratory factor analysis (EFA) with the baseline data of the 30 items in the total analysis set (principal factor analysis, using Eigenvalue >1 criterion for retaining factors) revealed two factors in the inter-item correlation matrix which explained more than 80% of the total variance. Low communality (h < 0.40) was observed for items 8, 9, 18 and 20. The EFA was repeated excluding these 4 items. Again two factors and items with low communality were found. The elimination of items with low communality (h < 0.40) resulted in the exclusion of 10 items in all (items 2, 7, 8, 9, 11, 17, 18, 20, 22 and 25). The last EFA resulted in one common factor explaining 87% of the variance with all remaining 20 items having sufficient high communalities (h ≥ 0.48). This 20-items solution was used to define a score for “attention and performance deficits” which was the sum over these items divided by the number of items answered. For simplicity of communication we call this mean total attention and performance score, APS20. The internal consistency of the APS20 is sufficiently high with a Cronbach’s alpha coefficient of 0.94 (raw score) showing that the items measure the same construct.

Although the results for a one-factor solution and the internal consistency of the APS20 were satisfying we repeated the EFA (principal factor analysis, initial communality: SMC, rotation method: PROMAX) with the baseline data of the 20 items allowing for the extraction of two factors. This was because the second factor’s Eigenvalue had been very close to one and we were curious to know whether information might have been concealed by the single common factor model and the Eigenvalue > 1 criterion. Additional file 1 shows the Rotated Factor Pattern with 2 factors and the variance explained by each factor. The rotated factors were correlated, with r = 0.683. Factor 1, AP-F1, is defined by descriptions of problems with prospective everyday memory including capacity-consuming monitoring; Factor 2, AP-F2, is defined as problems keeping attention focused, necessary to execute tasks in specific situations. For two items (14 and 21) the loadings on the two factors are moderate and quite equal, they will not be used for scaling of the two latent variables in the following analysis.

To confirm the one- and the two-factor models we conducted Confirmatory Factor Analyses (CFA) with follow up visit data using structural equation modeling for each country sample separately and across all countries pooled (see Additional file 2 for the Path diagram of the CFA). The model fit indices GFI for the 1-factor solution were moderately high in all countries (0.58-0.73) and similarly high for the 2-factor solution (0.56-0.71). The lambda value for each item was in an acceptable range; the smallest lambda was 0.53 (item 15: ‘Impatient at work’) in the US sample with the 1-factor solution.

From these results the scaling of the APS20 and the two subscales is justified. The APS20 is calculated from 20 items (1, 3–6, 10, 12–16, 19, 21, 23, 24, 26–30). The 2-factor subscales are computed with 9 items each (AP-F1: 1, 4, 6, 16, 19, 23, 24, 26, and 29; AP-F2: 3, 5, 10, 12, 13, 15, 27, 28, and 30). If 10% or fewer of the items are missing (i.e., 1–2 missing items for the APS20) then the mean score of the non-missing items may be used to estimate the APS20. If more than 10% of the items are missing then the APS20 should be discarded. The scores of the APS20 range from 0 to 4 with higher scores indicating frequent problems with attention and performance.

The APS20 has an approximately symmetrical distribution with a mean of 1.5 (SD = 0.7; skewness = 0.483) at baseline and 1.5 (SD = 0.7; skewness = 0.363) at the follow up visit (see Table 3 for details of AP-F1 and AP-F2).

Dependencies of the APS20 score on several variables were analyzed in a general linear model with factors gender, educational level, and country and their interactions, with age as covariate. There was a significant effect of country (p < 0.001), educational level (p = 0.0233) and two significant interactions (gender * educational level, p =0.0043 and country * gender, p = 0.0063).

In the US the mean APS20 score was 1.9, followed by 1.6 in the German sample, and 1.1 in the Mexican sample. Further, subjects from the lowest educational level had the highest mean score with 1.8 and the highest educational level (“university degree”) had the lowest means scores with 1.3. In the US sample the male subgroup had the highest APS20 mean score with 2.3 and in the German sample, females had the highest score with 1.8.

Many items which we excluded due to low communality (h < 0.40) seem to be rather unspecific and were not contentious. Four of the 10 excluded items were easy to agree on (difficulty index < 10%), one item was very difficult (item 7: 58.7%) for the patients.

Two further items are still of interest for further analysis. Item 17 addresses general satisfaction with one’s concentration and item 25 relates cognitive difficulties to the severity of the tinnitus.

The discriminant and convergent validity of the APS20, AP-F1 and AP-F2, to the other measures at baseline (THI-12, TRS, TSS and HADS) were analyzed computing Pearson correlation coefficients. An overview of the results is provided in Table 4. The APS20, AP-F1 and AP-F2 were moderately correlated with HADS-A and HADS-D but less correlated with the THI-12 total scale. Interestingly, AP-F1 and AP-F2 behaved differently with regard to THI-12 total score and HADS Depression score. The TRS monthly version item “impact on life” and the TRS monthly version total score (only done in the US, resulting in a smaller sample) were moderately correlated (0.23 ≤ r ≤ 0.48). But there was only weak correlation with the items and the total score of the TRS weekly version and the TSS weekly as well as the monthly version (r ≤ 0.31). This picture changes partially if we look at the inter-scale correlations for AP-F2. As can be seen from Table 4, in many cases AP-F2 correlates higher with the scales than AP-F1 and in a few cases even higher than APS20 does (e.g. correlation with HADS-depression).

As already mentioned, two of the ten excluded items (items 17 and 25) were used as further validation criteria. The coefficients for Item 17 (satisfaction with concentration), r = 0.50 and item 25 (concentration problems when tinnitus is severe), r = 0.57 are moderately high.

The box-plot graphs in Figure 2 show that the association between “satisfaction with concentration” (APSA item 17) and APS20 is quite close. Persons who were never or rarely satisfied with their concentration have clearly higher scores on APS20 compared to persons who were often or always satisfied with their concentration. There is a slight drop-down effect of the scores in the category “never” satisfied with concentration, which may be a result of the small number of cases at this end. This association between item 17 scores and APS20, as well as with AP-F1 and AP-F2 is also expressed by the correlation coefficients r = 0.41 (for AP-F1) and r = 0.52 (for AP-F2). This difference can be taken as a further argument for using these two subscales if one wants a differential analysis of cognitive complaints and possible treatment effects.

Not surprisingly, the APSA item 25 (concentration problems when tinnitus is severe) is moderately high correlated with most of the tinnitus scales, whereas item 17 (satisfaction with concentration, reversed scoring) correlates a little less with these scales (see Table 4). Interestingly, item 17 has a low correlation with anxiety (r = 0.22) but a higher correlation with depression (r = 0.42).

For clinical practice the ability of the APSA subscores to discriminate between the severity groups as defined by the THI-12 total score is of considerable importance. For this purpose we used the handicap classification of the THI-12 total score (no/mild/moderate/severe handicap) [12].

As can be seen in Figure 3, the cumulative distribution curves of no and mild handicap overlap, whereas the curves for moderate and severe handicap are separated on the APS20 scale. Hence, the APS20 best separates the moderate and the severe tinnitus groups from the other groups. This implies that patients with severe tinnitus handicap very likely show cognitive failures, whereas patients with no or mild handicap are only associated with occasional mishaps.

The APS20 as well as AP-F1 and AP-F2 show very high stability over time (mean duration of retest interval was 17 days (SD: 3.9) from baseline to follow-up, ranging from 12 to 30 days, scores are listed in Additional file 3). In detail, the ICC (3,1) is high for the APS20 (ICC = 0.9), as well as for AP-F1 (ICC = 0.91) and to a slightly smaller but still acceptable degree for AP-F2 (ICC = 0.87). Internal consistency as measured by Cronbach’s alpha coefficient is high for APS20 as well as for AP-F1 and AP-F2 with alpha ≥ 0.89.

To estimate sensitivity to change we used the change from baseline to follow-up on the APS20, AP-F1 and AP-F2. As there was no systematic intervention or change in intervention during the observational time, any change could only have been due to the consenting process and explanations by the clinician. We used the difference to baseline in tinnitus measures (TRS and THI-12 total scores) as criteria to estimate the sensitivity to change. As can be seen in Additional file 3, the mean change scores from Baseline were low for the tinnitus scales and nearly zero for the APSA subscales with considerable variability in individual change. The change scores in THI-12 correlated significantly with those in APS20 (r = 0.33), AP-F1 (r = 0.21) and AP-F2(r = 0.41). Also, correlations between change in TRS are significant for APS20 (r = 0.31) and AP-F2 (r = 0.40). The MDC, derived using the sample distribution, variability, and test-retest reliability of the APSA subscales, indicate whether a difference between baseline and a follow up visit represents a true difference in cognitive impairment. The values for MDC are ±0.6 for APS20 and AP-F1, and ±0.8 for AP-F2.

Further, we included a sample of healthy volunteers to estimate sensitivity and specificity of the APSA subscales. The data of this survey from a working-age population (N = 35, age range from 30 to 60 years with APS20 mean= 1.14, SD= 0.37) who had no tinnitus complaints showed no difference compared to the tinnitus subjects who according to THI-12 had no or mild tinnitus handicap (THI-12 ≤ 9) on any APSA subscales. There is good discrimination between healthy volunteers and ‘moderate tinnitus handicap’ (10 ≤ THI-12 < 14) on the APS20 (p = 0.0269) and the AP-F2 (p = 0.0014), as well as healthy volunteers versus ‘severe tinnitus handicap’ (THI-12 ≥ 14) in APS20, AP-F1 and AP-F2 (all p < 0.0001). Therefore we estimated sensitivity and specificity on the no/mild handicap group as the “negative” and the moderate/severe handicap group as the “positive” group. For APS20 we found 87% sensitivity, 46% specificity; for AP-F1: 85% sensitivity, 44% specificity; and for AP-F2: sensitivity of 89%, specificity of 54%.