Inter-rater agreement of the Quality of Life-Alzheimer’s Disease (QoL-AD) self-rating and proxy rating scale: secondary analysis of RightTimePlaceCare data

Improving quality of life (QoL) is an important focus of various therapeutic interventions for people with dementia [1]. Therefore, QoL is increasingly gaining importance as an outcome measure to evaluate interventions in dementia care [2]. The QoL of those with dementia is considered as an individual, subjective, dynamic, multidimensional and complex construct, which includes the assessment of and adaptation to the consequences of dementia [3‐6]. QoL can only be understood within the person-environment system, which Lawton has described in four sectors: behavioural competence (evaluated functioning of a person in relation to inner or outer events), perceived QoL (evaluations about major life domains), psychological well-being, and objective environment [6, 7].

The measurement of dementia-specific QoL involves particular features, due to the particular symptoms of the disease. Functional impairment of memory, cognition, time perception, attention, judgement and communication, along with the level of insight into the illness and the severity of the disease can all influence how questions are understood, and the rating and communication of the subjective condition [8‐12]. In addition, the reliability of QoL assessment is affected by limitations of consciousness [13] and behavioural and non-cognitive symptoms such as depression, restlessness and psychosis [8]. Emotional symptoms such as social deprivation can also influence QoL ratings by people with dementia [9]. The progress of dementia is likely to make self-rating (SR) infeasible at a certain point. For people with severe dementia, proxy rating (PR) by informal or professional carers is indispensable if they are not to be excluded from QoL determination.

A recent review [13] has identified 16 QoL measures for people with dementia and has assessed their psychometric properties as well. Many measures were based on proxy assessment, with questionable validity for people with mild to moderate dementia. The best researched measure was the Quality of Life in Alzheimer’s Disease scale (QoL-AD) [14]. It is available as a self-rating version (QoL-AD-SR: Quality of Life in Alzheimer’s Disease Self-Rating scale) and as a proxy rating version (QoL-AD-PR: Quality of Life in Alzheimer’s Disease Proxy Rating scale). The instrument has been translated in various languages and has good psychometric properties overall [14]. Bowling et al. [13] give an overview of the psychometric evidence. However, some discrepancy between the two rating versions has been identified, which indicates further research aimed to clarify the relationship between the assessment through self-rating by the person with dementia and proxy rating [13].

We conducted a systematic literature search in Medline via PubMed (April 2016), which was aimed at identifying the body of knowledge on the agreement between the two QoL-AD rating versions. The search strategy used was: ‘QoL-AD OR (quality of life Alzheimer’s disease) OR (quality of life Alzheimer’s disease scale) OR (quality of life Alzheimer’s disease questionnaire) AND ((agreement OR accordance OR consensus OR conformity OR rapport OR congruence OR match) OR (discrepancy OR gap OR mismatch OR difference OR distinction OR disagreement) OR (caregiver bias)) AND (self OR (proxy OR caregiver OR carer))’. According to the four-phase PRISMA [15] process for selecting articles, all studies in the English or German language published in the last 10 years were included. A total of 28 relevant studies remained [16‐43]. Across these studies, carers assessed QoL lower than those suffering from dementia. Fourteen articles [19, 20, 22, 25, 26, 28, 29, 31, 33, 36, 37, 39‐41] reported the level of agreement between QoL-AD self- and proxy ratings; ranging from a poor [20, 28] to a very good [37] agreement. In almost all of the studies, the level of agreement at the QoL-AD total score was estimated using correlation analyses, such as the calculation of the intraclass correlation coefficient (ICC) or the application of the paired-samples t-test. However, calculation of correlation coefficients is never appropriate for agreement assessment, as correlation coefficients give information about a linear relation of two metric variables and not about the agreement [44, 45]. Two clearly distinct measures that are intended to assess the same construct regularly correlate in a certain way without necessarily reaching good agreement and good agreement can be reached without good correlation. Interpreting correlation analyses as agreement assessment must therefore be questioned and should generally be avoided. Only two studies [25, 28] used the Bland-Altman plots as recommended for metric variables [44, 45]; the presented data indicate an unacceptable range of agreement. Factors associated with the level of agreement between self- and proxy ratings were reported in eleven studies [16‐22, 25, 27, 28, 36]. The most frequently mentioned factors for different ratings were behavioural and psychological symptoms of dementia (BPSD) and severity of cognitive impairment of those with dementia, as well as caregiver burden (see Additional file 1).

We found no studies investigating the level of agreement between QoL-AD-SR and QoL-AD-PR or agreement at the item level in a large European sample.

Therefore, our study aims to explore the inter-rater agreement of the measures QoL-AD-SR and QoL-AD-PR with the following objectives:

In addition, the identified factors (see Additional file 1) were hypothesised to be associated with the level of agreement and explored using regression analysis.

Data for the secondary analysis were obtained from the European RightTimePlaceCare study (RTPC; FP7-Health-F3–2010-242,153) [46]. Cross-sectional data were used.

The aim of our study was to analyse the inter-rater agreement of the measures QoL-AD-SR and QoL-AD-PR, in terms of both the total score and the items, including a setting-specific analysis, and to identify factors associated with this agreement.

Our analysis suggests that proxies, i.e., informal carers and best informed professional carers, rate the QoL of those with dementia on average lower than those with dementia themselves. This is confirmed by previous studies that compare the performance of the two QoL-AD measures at the group level [16‐43].

We found no acceptable inter-rater agreement (predefined range: +/− 3 scale points) of the QoL-AD measures in the whole sample. Comparing both settings and the means of differences, professional carers in ILTC rated the QoL on average one point less than the people with dementia, the informal carers in HC rated on average 4.7 points less. Nonetheless, no acceptable inter-rater agreement could be demonstrated in either HC or in ILTC when considering the Bland-Altman plots.

In almost all former studies [19, 20, 22, 26, 29, 31, 33, 36, 37, 39‐41] the level of agreement at the QoL-AD total score was estimated using correlation analyses, which are not appropriate for agreement assessment. Bland-Altman plots were used in only two studies [25, 28]. Bosboom et al. [25] concluded that the agreement between self-rating and rating from the proxy-proxy perspective and from the person-proxy perspective is acceptable. However, this conclusion stems from a clear misunderstanding of the Bland-Altman plots method, and in particular LoA. The authors argued that only 2.5% (self-rating vs. proxy-proxy rating) and 5% (self-rating vs. person-proxy rating) of their Bland-Altman plots were beyond the LoA. However, LoA are derived from the distribution of the differences between paired observations, and should always include approximately 95% of these observations. This holds true for all Bland-Altman plots and LoA, independent of the actual agreement. For a valid conclusion, a predefined acceptable range for the differences would have been necessary. The actual LoA were − 8 to 6 points (self-rating vs. proxy-proxy rating) and − 9.5 to 15 points (self-rating vs. person-proxy rating). These LoA are far beyond an acceptable range. Zhao et al. [28] considered inter-rater agreement for the total score as fair (ICC = 0.58). They did not discuss their Bland-Altman plot, which even missed LoA. Using the given data from Zhao et al. [28], LoA ranges from − 7.3 to 13.9, indicating an unacceptable range of agreement.

Therefore, the QoL-AD proxy rating is not directly interchangeable with and cannot replace the QoL-AD self-rating of a person with dementia.

At the item level, we did not find any agreement better than poor except for the item ‘Marriage’, which achieved a fair agreement. The agreement on item level is comparable to the results of the studies by Conde-Sala et al. [33], Hoe et al. [40] and León-Salas et al. [31], which demonstrate mostly poor agreement on the item level. Chan et al. [29] and Wolak et al. [37] had a more positive conclusion, stating that agreement on the item level was largely good. Both studies inappropriately used correlation analysis for agreement rating.

We initially identified several factors that were hypothesised to be associated with the level of agreement and fitted two models. However, no factor importantly influenced the difference between self- and proxy ratings, except the setting (HC vs. ILTC).

The most frequently mentioned associated factor of those with dementia was BPSD [18, 20, 25, 28, 36]. In both fitted models we were able to confirm the strongest influence of all continuous variables for BPSD measured with the NPI-Q. Our results are in line with studies [18, 20, 25, 28, 36] that showed similar effect sizes and direction of effect.

The influence of the functional status or independence in activities of daily living is demonstrated in our analyses of Model 1. The level of agreement changed in such a way that the higher the dependence of the person with dementia (lower KATZ ADL score) the lower the QoL-AD rating of the proxy relative to that of the person with dementia. In Model 2, including only people with dementia living in HC, the effect was in the same direction but had a low magnitude and did not reach statistical significance. This confirms the results of Zhao et al. [28] and Zucchella et al. [18]. A possible explanation for the varying ratings could be the “disability paradox” introduced by Albrecht and Devlieger [66]. It means that many people with severe disabilities report a good QoL, although for external observers these people do not seem to be in good health. Another phenomenon in this context is the concept of response shift, defined as changes in the meaning of one’s self-evaluation of a target construct resulting from changes in internal standards, values, or conceptualisation [67]. Thus, to underestimate a person’s QoL compared to his or her own rating might also be a result of response shift.

Huang et al. showed that living in HC led to a lower difference of self- and proxy ratings compared to ILTC [17] (i.e., the proxy rating in HC was not as low as the self-rating in HC compared to ILTC ratings). This is the opposite of our results, but the ILTC proxy raters in the study by Huang et al. were also relatives, i.e., informal carers. In our analysis, we are not able to distinguish setting and proxies (staff ratings in ILTC and informal carer ratings in HC). Therefore, we cannot draw any conclusions as to whether the discrepancy results from differences according to the setting or from the proxy raters.

Unlike Bosboom et al. [25], Conde-Sala et al. [16, 19], Huang et al. [17] and Zhao et al. [28], we did not find any influence of the severity of cognitive impairment or depressive symptoms [18‐20]. The level of education of the person with dementia, as stated by Huang et al. [17] and Tay et al. [20], could not explain the differences between measurements.

Caregiver burden has most often been identified in previous studies as an influencing factor of carers, and our analyses showed similar effects. Both measurements for caregiver burden (ZBI and NPI-Q-D) showed positive gradients on the difference in single regression models and highly correlated with each other (results not shown). The estimator for NPI-Q-D changed signs when combining both measurements in our predefined Model 2, because of multicollinearity. The extent of the standardised effect sizes of both estimators did not reach a relevant level, and are similar to previous results [18, 22, 28, 36]. The results of the studies by Huang et al. [36] and Schulz et al. [22] on the influence of the QoL of carers could not be confirmed in our analysis. No influence of the informal carer’s relationship to the person with dementia, as stated by Huang et al. [17], was shown. Finally, our analysis found no differences in terms of gender of the informal carer, unlike the findings of Conde-Sala et al. [19].

Overall, our fitted regression models covered only 8 and 13% of the observed variance of the difference between self- and proxy ratings. Therefore, it can be assumed that there are further unknown factors influencing the difference between self- and proxy ratings, which were not addressed by the data or our analysis.

Our analysis showed that professional and informal carers appear to generate lower proxy ratings of QoL than those with dementia themselves. The assessment of the inter-rater agreement of the two measurement methods QoL-AD-SR and QoL-AD-PR revealed pronounced differences. Nevertheless, the QoL-AD benefits from good psychometric properties and the applicability to people with a wide range of dementia severity to rate themselves.

Our data indicate that the QoL-AD self- and proxy ratings are not directly interchangeable due to the inter-rater gap. Thus, QoL-AD-PR provides a complementary perspective rather than a substitute for self-rating. In particular, a mix of self-rating and proxy rating may be biased. From a clinical point of view, our study suggests that either only one rating method should be performed or both rating methods parallel with separate analyses grouped in self- and proxy ratings. Self-ratings should be applied whenever possible. It is also required to report transparently who has responded to the QoL-AD.

An implication for future research would be to compare the QoL-AD-SR with the corresponding ratings of several proxy groups such as informal carers and professional carers. This would allow comparison of the levels of inter-rater agreement between the person with dementia and various proxies, and the agreement among the latter. Hereby, the impact of the setting on the level of agreement might be taken into account more specifically.