Background
Methods
Study design and participants
Procedures
Outcomes, data collection, analysis
What is the maximum difference in WC ratings that stakeholders would find acceptable when two experts independently assess the same patient? | Lawyers (n = 81) | Psychiatrists (n = 242) | Experts (n = 114) | Judges (n = 47) | Insurers (n = 108) |
---|---|---|---|---|---|
… in the current situation of performing evaluations, median difference (interquartile range, IQR) | 15% (10–20%) | 20% (10–25%) | 20% (10–25%) | 15% (10–20%) | 10% (10–20%) |
Comparing RELY 1 and RELY 2
Patient and public involvement
Results
RELY 1-study
RELY 1 | RELY 2 | |
---|---|---|
Psychiatrists, RELY 1: n=19a; RELY 2: n=35b | ||
Age | ||
31–40/ 41–50/ 51–60/ > 60 years/ missing | 5/ 42/ 21/ 32/ 0%c | 3/ 40/ 31/ 20/ 6% |
Gender | ||
male | 79% | 83% |
Experience | ||
Years since board certification as psychiatrist, mean (SD) | 15.6 (9.7) | 15.8 (9.0) |
Number of years performing disability evaluations, mean (SD) | 13.8 (9.2) | 12.4 (7.5) |
Number of evaluations in the previous year, | ||
0–4/ 5–20/ 21–50/ > 50/ missing | 0/ 10 / 32 / 58/ 0% | 6/ 17/ 31/ 40/ 6% |
Time span from training to rating in days, mean (range) | 404 days (115–578) | 41 days (5–88) |
Patients, RELY 1: n=30; RELY 2: n=40 | ||
Age, years: mean (SD) | 47.2 (8.6) | 48.6 (10.1) |
Gender | ||
male | 57% | 53% |
Marital status | ||
Unmarried/ married/ divorced/ missing | 20/ 40/ 40/ 0% | 20/ 28/ 45/ 8% |
Nationality | ||
Swiss/ others/ missing | 63/ 23/ 14% | 70/ 28/ 2% |
Country of birth | ||
Switzerland/ others/ missing | 67/ 27/ 6% | 75/ 23/ 2% |
Severity of disorderd | ||
mean (SD) | 5.3 (2.1) | 4.9 (1.8) |
Typicality of study patient compared to other patients seen by the expert | ||
frequent / semifrequent / rare | 36/ 44/ 20% | 27/ 56/ 17% |
Main diagnoses (ICD 10 classification) Number of diagnoses RELY 1: n=36; RELY 2: n=47 | ||
Mood disorders (F3) | 26% | 40% |
Neurotic, stress-related, somatoform disorders (F4) | 19% | 21% |
thereof somatoform disorders (F45) | 6% | 15% |
Organic (F0) | 11% | 9% |
Disorders of adult personality and behaviour (F6) | 11% | 6% |
Psychoactive substance use (F1) | 3% | 0% |
Mental retardation (F7) | 0% | 2% |
Behavioural and emotional disorders with onset in childhood (F9) | 0% | 2% |
Patients without main diagnosis | 19% | 19% |
Reliability and agreement
Reference for WC | WC Mean (95%CI) | Total variance | Variance components Absolute variance (Relative variance) | Reliability | Agreement | |||||
---|---|---|---|---|---|---|---|---|---|---|
Proportion of WC ratings between two psychiatrists whose ratings differed equal or less than the ‘maximum acceptable difference’ of 25 percentage points WC | ‘Standard error of measurement’ (95%CI) | ‘Maximum acceptable difference’ (95%CI) | ||||||||
Psychiatrists | Patients | Residuals | ICCabs.agree (95%CI) | reported in natural units | reported in natural units | |||||
Last job | RELY 1 N = 120 | 43.6% (34.1–53.2) | 1092 | 263 (24%) | 414 (38%) | 415 (38%) | 0.38 (0.19–0.55) | 52.2% (94/180) | 26.0% WC (21.5–31.0) | 72.2% WC (59.5–86.0) |
RELY 2 N = 160 | 46.3% (39.9–52.6) | 1064 | 76 (7%) | 495 (47%) | 493 (46%) | 0.47 (0.29–0.61) | 61.7% (148/240) | 23.9% WC (20.8–27.0) | 66.1% WC (57.7–74.9) | |
Alternative work | RELY 1 N = 119 | 55.0% (47.3–62.8) | 1060 | 88 (8%) | 457 (43%) | 515 (49%) | 0.43 (0.22–0.60) | 61.6% (112/177) | 24.6% WC (20.9–28.4) | 68.1% WC (57.9–78.8) |
RELY 2 N = 155 | 62.9% (57.7–68.0) | 669 | 50 (7%) | 292 (44%) | 328 (49%) | 0.44 (0.25–0.59) | 73.6% (170/231) | 19.4% WC (16.9–22.0) | 53.8% WC (46.8–61.0) |
a) Expected by stakeholders | b) Observed in the RELY studies | ||||
---|---|---|---|---|---|
‘Maximum acceptable difference’a | Corresponding ‘Standard error of measurement’ | ‘Standard error of measurement’ | Corresponding ‘Maximum acceptable difference’ | ||
25% WC | 9.0% WC | Last job | RELY 1 | 26.0% WC | 72.2% WC |
20% WC | 7.2% WC | RELY 2 | 23.9% WC | 66.1% WC | |
15% WC | 5.4% WC | Alternative job | RELY 1 | 24.6% WC | 68.1% WC |
10% WC | 3.6% WC | RELY 2 | 19.4% WC | 53.9% WC |
RELY 2-study
Reliability and agreement
Comparing RELY 1 and 2
Sociodemographics
Variances, reliability, and agreement (Table 3)
Patients’ and psychiatrists’ perception of the functional evaluation
Discussion
Main findings
Strengths, limitations, challenges in design and performance
Illustration of the interaction of various sources of variance and their impact on the reliability measure ICC. | |
General formula for ICCabs.agree [25]: \( \frac{\sigma_{Patients}^2}{\sigma_{Patients}^2+{\sigma}_{Psychiatrists}^2+{\sigma}_{Residuals}^2\ } \) | |
Example 1 - Analogy to the situation observed in RELY 1: the ICC is calculated based on a patient variance of 500, a psychiatrist variance of 100 and a large residual (unexplained) variance of 500. ICC = \( \frac{\mathbf{500}}{\mathbf{500}+100+500} = 0.45 \) which corresponds to a fair discrimination of patients [26] | |
Example 2 - Analogy to the situation observed in RELY 2: The ICC is calculated with a patient variance of 250, a psychiatrist variance of 50 and a large residual (unexplained) variance of 250. ICC = \( \frac{\mathbf{250}}{\mathbf{250}+50+250} = 0.45 \) which corresponds to a fair discrimination of patients (equal to example 1) | |
Despite reduction of total variance, the proportionate reduction of variance across all sources of variance results in an ICC of 0.45 identical to example 1. Despite reduction of variance by half, the ability to discriminate patients in their ability to work did not change. | |
Example 3 - Typical situation for a reliable instrument: Most variance is explained by patient variance, with little psychiatrist variance and residual variance: patient variance of 500, psychiatrist variance of 25, and residual variance of 75. As a result, expert variance and residual variance contribute little to the total variance, indicating low measurement error. This allows excellent discrimination among patients. ICC = \( \frac{\mathbf{500}}{\mathbf{500}+25+75} = 0.83 \) |
Source of variation | Factors that may impact on the variance of overall work capacity |
---|---|
Psychiatrists | • Experience in disability evaluation • Knowledge about previous work • Structuring and prioritizing of information • Psychiatrists’ idiosyncrasies (e.g. leniency/strictness) |
Patients | • Socio-demographic features • Diagnosis, severity of disorder • Compliance, including malingering • Skills in presenting their case • Symptom exaggeration |
Residuals | • Interaction psychiatrists*patients • Interaction patient*last job; patient*‘alternative work’ |
External factors: • Changes in legislation with impact on medical evaluations • Interferences of legal demands with medical judgements • Turn-over of staff involved in the studies • Overall attitude in society towards disability |