Cross-sectoral inter-rater reliability of the clinical frailty scale – a Danish translation and validation study

To ensure cultural and conceptual compliance with the source instrument [9, 11], we translated the CFS into the Danish language using the 10-step ISPOR Principles of Good Practice for the Translation and Cultural Adaptation of Patient-Reported Outcomes [17]. The Danish translation process is depicted in Fig. 2.

Health care professionals from the primary and secondary health care sectors were recruited to validate the CFS-DK and to support potential future use in a cross-sectoral context. Forty raters assessed 15 written clinical case vignettes using the CFS-DK. The raters were 10 community nurses, 10 general practitioners, and 20 hospital doctors (10 from internal medicine, 10 from intensive care) and were recruited as a convenience sample from the authors’ professional network.

Cases consisted of a short text and a picture of each case-patient and were selected to collectively represent all nine levels of the CFS. Cases provided essential information on 1) symptoms of diseases, 2) dependency on others, 3) cognitive function, and 4) physical condition. Each case was presented alongside a picture of the CFS-DK for reference (Fig. 1). Cases were built to imitate real-life patients by authors SKN (senior registrar in geriatric medicine) and KAR (consultant and professor in geriatric medicine). Prior to completing the questionnaire, raters were asked to view a five-minute video introducing frailty as a concept, and the CFS and its use in clinical practice, as well as raising awareness of pitfalls in using CFS, e.g. scoring patients with dementia. The raters then assessed each case according to the 9-point CFS-DK. The cases were presented in random order of severity but were rated in the same order by each rater. Finally, raters were asked to assess the relevance of the CFS for their own area of work and for cross-sectoral collaboration.

Study data were collected using an online questionnaire developed in REDCap (version: REDCap 9.1.15 -© 2020 Vanderbilt University) [18, 19], an electronic data capture tool hosted at Open Patient Exploratory Network (OPEN) at Odense University Hospital, the Region of Southern Denmark.

Inter-rater reliability was assessed using the intraclass correlation coefficient (ICC) with 95% confidence intervals (CI) [20, 21] that was calculated i) for all 40 raters, and ii) within each professional group. The ICCs were considered poor (< 0.40), fair (0.40–0.59), good (0.60–0.75), or excellent (> 0.75) according to standard practice [22].

The sensitivity of the inter-rater reliability was examined using a random effect linear regression [23] with CFS-DK scores as outcome and with random effects included for case and rater, and with rater experience and professional group as covariates. The values from the community nurses were used as reference. In this way, we could assess the extent to which the inter-rater reliability was sensitive to the raters’ length of clinical experience and any unobserved variations in raters and cases. Other studies have used a graphical Bland-Altman approach [24], but this was less appropriate for the current study as our raters came from four different professional groups.

Statistical analysis was performed using SAS software (SAS Institute Inc., Cary, NC, USA). The statistical significance threshold for all tests was set to P < 0.05. Figures were made using “R” software (Version 3.6.1) [25] and the ggplot2 package [26].

In accordance with the ISPOR guidelines [17], the final report of the translation process is available in Additional file 1. The translation process is depicted in Fig. 2 and is summarized as follows:

Step 2–3: We observed high agreement in the meaning and wording of the two forward translations. Minor incongruities were mostly related to synonyms for particular words. Differences were discussed and resolved in a reconciliation meeting. Step 4–5: The back translation corresponded well to the reconciled forward translation. The few discrepancies were identified and discussed, leading to minor word changes in items 4, 5, 6, and 9.

Step 6: To ensure concordance between the harmonized translation and the source instrument, the original instrument developer was contacted for revision and feedback. This led to two minor changes in items 5 and 9. Step 7–8: To test for conceptual coherence, interpretation, and cultural relevance, the translated instrument was tested on five respondents from the target population (two general practitioners, one community nurse, and two geriatricians). Three respondents were women and two were men, and occupational experience ranged from 5 to 40 years. The five respondents were asked to score the same three cases, after which a cognitive debriefing was held with each respondent individually. This did not result in changes to the translation. Step 9–10: The final translation was proofread. The original source instrument and the Danish translation are presented in Fig. 1.

All 40 raters assessed the 15 cases, yielding 600 observations with 40 replicate observations per case. Twenty-two of the 40 raters (55%) were women, and mean length of clinical experience in the four professional groups ranged from 14 to 17 years, with hospital doctors being the most experienced.

The overall inter-rater reliability for all 40 raters (based on individual assessments) was 0.85 (0.74; 0.93). As shown in Fig. 3, the inter-rater reliability was similar for the four professional groups, though highest for the hospital doctors specialized in intensive care (0.90, CI 0.82; 0.96). The ratings had narrow interquartile ranges and median ratings within one level for all but one case. The ratings for Case 13 (which described a case at level 9 with terminal illness but ‘not otherwise evidently frail’) had wide interquartile ranges for both primary care physicians and community nurses.

Table 1 shows the results of the random effect linear regression (with CFS-DK scores as outcome and with case and rater as unobserved random effects) aiming at examining the sensitivity of inter-rater reliability to rater experience. Although the hospital doctors specialized in intensive care came close to being significantly different (p = 0.05), the inter-rater reliability did not differ significantly between the professional groups or according to the length of the raters’ clinical experience (p = 0.96). The random effects components showed that variation between cases (4.40) was more important than variation between raters (0.01).

All but one of the 40 raters considered the CFS-DK to be relevant for their own area of work, and all raters considered it to be relevant for cross-sectoral collaboration.

This study used clinical case vignettes, and although the case vignettes were clinically appropriate and built to imitate real-life patients, they could also be regarded as hypothetical. Case vignettes provide the opportunity to let raters assess the exact same information, with the same structure and nearly identical setting at the same time point, allowing equal comparisons between professional groups. The cases were designed to encompass all the CFS levels. Ratings given to the 15 cases clearly reflect these different degrees of severity. Conversely, case vignettes introduce a risk of inflated inter-rater reliability and ICC as raters assess all cases at once, which does not reflect clinical practice.Another limitation to this study is the relatively small number of cases (n = 15). However, the use of case vignettes allowed a high number of observations of 600 (15 cases, 40 raters), a quadrupling compared to previous investigations of inter-rater reliability following translation of the CFS [24].

Finally, raters were recruited as a convenience sample, inherently risking inflated reliability measures and sensitivity analyses.

This study has several strengths. First, the translation process was completed using a rigorous procedure that followed the ISPOR guidelines. Poorly translated instruments threaten the validity of the data, and quality is dependent on methodology [17]. Second, the validation included key actors most often involved in significant transitions between the primary and secondary health care sectors, and the health care professionals were experienced in their clinical fields. Third, raters were informed that inter-rater reliability would be compared between professional groups but that individual rater performance would not be assessed. We expect this to have reduced the risk of a Hawthorne effect (i.e. awareness of being observed).