Use of a structured functional evaluation process for independent medical evaluations of claimants presenting with disabling mental illness: rationale and design for a multi-center reliability study

RELY will be a multi-center reliability study in which four psychiatric experts will independently assess workers undergoing a disability evaluation to assess restrictions and limitations associated with their mental illness.

Consecutive claimants will be recruited from the Zürich office of the Swiss National Disability Insurance and from the Swiss National Accident Insurance Fund, Suva, while the evaluations and videotaping will be arranged through four independent assessments centers (two in Basel, one in Lucerne, and one in Interlaken).

Medical and administrative staff at the insurance offices will identify and contact eligible claimants who have been scheduled to undergo a polydisciplinary disability evaluation, including a psychiatric evaluation. Eligible claimants will be workers whose first time disability evaluation includes a psychiatric interview, and capable of communicating in German without an interpreter. No other exclusion criteria apply.

After providing claimants with brief information about the study, the administrative staff will request permission to provide their name and telephone number to the research team so that they can be contacted about joining the RELY study. At that point, the insurer’s involvement in the study will end. Claimants who approve being contacted for the study will receive a detailed patient information that explains the study and its implications for the claimant in plain language. Next, a researcher will call to discuss the study with the claimants, to answer any questions that may have come up and to explain the informed consent form for participation. The claimants are then being asked to sign and return the informed consent form to the researcher. (For further details see Declarations, 2. Ethics approval and consent to participate).

Medical experts eligible for our study will be psychiatrists who currently provide disability evaluations for the Swiss National Disability Insurance or the Swiss National Accident Insurance acquired through a convenience sample of 20 psychiatric experts. No further eligibility criteria (such as level of expertise or years of experience) applied to reflect real life and prevent sampling error [2]. According to Swiss practice [36], the vast majority of experts are active in patient care. Participating psychiatrists will undergo a standardized, pre-piloted, training program for functional evaluation (see below [26]).

All psychiatric experts will conduct at least one videotaped disability evaluation and rate up to seven videotaped evaluations completed by their colleagues. They will be reimbursed for their time according to their usual rate. Psychiatrists and claimants will not be blinded to the study. Psychiatrists performing the evaluation will not be visible on the video, however, psychiatrists rating claimants shown in the videos may recognize voices of their colleagues.

Our functional evaluation approach focusses on the claimants’ health related functional limitations that relate to their ability to work, and complements psychiatrists’ conventional evaluation. The functional evaluation is consistent with the framework of the International Classification of Functioning, Disability and Health [27] and consists of the following components: 1) a semi-structured functional interview focusing on the claimants’ self-perceived limitations to work as a starting point for the evaluation; 2) a reporting instrument (IFAP) to document experts’ judgments with regards to the claimants’ mental functions (12 items) and their functional capacities (13 items); and 3) a detailed description of their last job and any suitable alternative work.

The training consists of written material describing the interviewing approach and information about the IFAP-instrument. Three face-to-face meetings lasting for three hours each will be used to discuss the functional evaluation with a methodological and a content expert based on cases from the experts’ own practice. Between training sessions, psychiatrists will be asked to use the functional evaluation with their own claimants and bring forward to the meetings any questions that arise. Additional details of the functional evaluation training program are described elsewhere [26].

Sociodemographic data on all eligible claimants (age, gender, civil status, nationalities and country of origin) and psychiatric experts (age, gender) will be recorded, complemented by psychiatrists’ professional experience (years since specialization; years working as psychiatric expert; number of work capacity evaluations over the last year; working concurrently as treating psychiatrist and psychiatric expert [Yes/No]). For eligible claimants who chose not to participate, insurance staff will record their mental disorder, gender and age.

Following each disability evaluation, the interviewing psychiatrist and three psychiatric raters will record all mental disorders that impact on work capacity using ICD 10 coding. They will fill in the IFAP-instrument, thereby providing estimates and judgments on overall work capacity related to the claimants’ previous job and to suitable alternative work (single item on a scale from 100 to 0 %, higher values indicate better work capacity), and on the following variables: commonly affected mental functions (12 items on a 5-point scale, higher score indicate greater impairment) and functional capacities commonly affected in mental disorders, also related to the claimants’ previous job and to suitable alternative work (13 items on a 5-point scale, higher scores indicate greater impairment).

Each psychiatrist who completes a disability evaluation will rate their certainty regarding the claimant’s overall work capacity (scale from 0 to 10, higher scores indicate greater certainty), the severity of the claimants’ mental disorder(s) compared to that of other claimants with similar mental disorders (scale from 0 to 10, higher scores indicate greater severity), and the extent to which claimants in the study represent ‘typical claimants’ referred for disability evaluation.

Six criteria capture ‘typicality’ in the context of our study: severity of mental disorders, complexity of diagnoses, impact of the claimants’ mental disorders on work capacity, work trajectory, spectrum of jobs and occupational activities, and education and professional training. For each criterion, respondents will relate the current claimant to other claimants assessed in their practice, report the frequency of evaluating such a claimant on a 3-item scale (seldom/neither seldom nor often/often) and generate a global judgment on the frequency of encountering such a claimant (same 3-item scale).

Our primary outcome is psychiatrists’ global rating of work capacity (expressed as percentage work capacity) in the previous job and in suitable alternative work. The intra-class correlation coefficient (ICC) should be at least 0.6 to be useful [42], which seems achievable in the insurance medicine context. [43, 44] With a sample size of 30 claimants, each evaluated by four raters, we expect to estimate an assumed ICC of at least 0.6 with a precision expressed as the width of a two sided 95 % confidence interval (CI) of ± 0.15 [45].

In an exploratory analysis, we will assess which variables – clinical (diagnoses), professional (psychiatrists’ characteristics) and demographic (claimants’ characteristics) – are associated with variation in the reliability of expert judgments, although we anticipate that these analyses will be underpowered to detect significant associations.

We will describe psychiatrist and claimant characteristics with absolute and relative frequencies for categorical variables and with mean and standard deviation, or median and interquartile range, for continuous variables. To explore whether random allocation of psychiatrists to claimants was successful, i.e., producing low rater-group-membership, we will calculate the frequency of the same two, three or four psychiatrists being allocated to a claimant (‘rater-group-membership’) and calculate the Jaccard similarity coefficient of each pair of psychiatrists.

Each rating (rather than a consensus rating) will be used for our analysis. To estimate the means and associated 95 % CIs for the primary and secondary outcomes we will use mixed-effects models that take into account the crossed cluster design (i.e., each psychiatrist will assess several claimants and each claimant will be assessed by several psychiatrists) by including crossed random intercepts for claimants and psychiatrists, if appropriate, in order to obtain adjusted standard errors [46].

Furthermore, we will use mixed-effects models to estimate the variance components needed for the computation of the ICCs of work capacity, IFAP items and IFAP item sums (i.e., the residual variance and between-claimant variance for the ICC_{absolute agreement}, and additionally between-psychiatrist variance for the ICC_consistency) [47]. The associated measures of precision (95 % CIs) will be obtained by model-based parametric bootstrapping. Additionally, we will perform exploratory subgroup analyses on the ICCs for work capacity for specific diagnostic groups (ICD 10 categories of mental disorders, F00-F09; F10-F19; F20-F29; F30-F39; F40-F48; F50-F59; F60-F69). The clinical interpretation of ICC and agreement will follow the criteria by Fleiss [48] and Cicchetti [49]. For the ICC: poor (ICC < 0.4); fair (0.40 to 0.59); good (0.60 to 0.74); excellent (0.75 to 1.00), for the level of percentage agreement: poor (<70 %); fair (70 % to 79); good (80 to 89 %); excellent (>90 %).

To assess the associations of certain claimant traits (age, gender, IFAP 2 scores) and psychiatrist judgments (severity of disease, certainty of rating) with the reliability of work capacity, we will use a likelihood ratio test to compare two mixed-effects models, with one model allowing for heteroscedasticity between the claimant groups as specified by the claimant trait or psychiatrist judgment of interest. We will compute ANOVAs of mixed-effects models with main diagnosis or severity as a fixed effect to assess differences in certainty in work capacity ratings between main diagnoses or severity of disease. To check for a rater effect with regard to work capacity, where some psychiatrists may be systematically more lenient or strict in their judgments, we will use a likelihood ratio test to compare the two corresponding mixed-effects models used for the computation of ICC_{absolute agreement} and ICC_consistency.

To establish trust in the study among patients and their representatives, we will establish a stakeholder group with members from patient organizations (n = 3), patient lawyers (n = 1), the academic legal profession (n = 1), cantonal and federal social courts (n = 2), professional medical societies (n = 3), and insurers (n = 3) whose names will be published on the study website. This group will have observer status and act as a guarantor for the RELY study which implies regular meetings (minimum once a year) for an update about study progress, public acceptance and awareness, interim and final results. Members can ask specific questions about content and course of the study and act as contact within their community although confidentiality about details of the meetings will be agreed upon. This level of transparency should support the study’s credibility to the public.

Furthermore, we will establish a study website to inform claimants and the public about the study purpose, design and organization, involved researchers, the members of the stakeholder group and funders.

Reliability studies are often used to evaluate the measurement properties of human observers [50]. The results of this real-life study will provide an estimate of the measurement properties among expert psychiatrists who use our functional evaluation approach when determining health-related work capacity of workers claiming disabling mental illness.

Insurer’s typically consider medical evaluations as the best evidence for informing whether or not to award disability benefits [14]. However, for any evaluation process to be valid, it must first demonstrate reliability (although reliable evaluations can still be invalid). Inter-rater reliability can inform about the fairness of expert judgments (i.e., whether the claimants‘ evaluators come up with similar levels of functional impairment and similar judgments on remaining work capacity) when an independent reference standard for the diagnosis ‘work incapacity’ is lacking. Reliability reflects the extent to which medical evaluations can discriminate between claimants despite measurement errors inherent in the assessment. Reliable evaluations should consistently distinguish between claimants with high, moderate and low work capacity when performed by knowledgeable experts.

In work capacity evaluation, multiple sources of variation can reduce inter-rater reliability regarding the evaluation of mental functioning (IFAP 1) and functional capacities (IFAP 2) of claimants [51, 52]: Information variance when experts obtain different information from the claimants as a result of asking different questions; observation variance when experts differ in what they pick up and remember when presented with the same information; interpretation variance when experts attach different importance to what they observe and hear; and criterion variance when experts apply different criteria to score the same information. Furthermore, psychiatrists, who typically lack expertise in vocational evaluation, may vary in their interpretation of the consequences of the claimants’ functional limitations on job demands and work ability.

Table 1 summarizes our multi-facetted approach to reduce these potential sources of variability: the semi-structured functional interview on occupational limitations with its five steps, the reporting instrument with 13 items to document the claimants’ functional limitations in reference to a detailed job description, the anchor definitions for grading severity of functional limitations and the calibration with peers in small group case based-learning. Videotaping the interview with the claimant will protect against subject variance.

Studies on inter-rater reliability of expert judgments in disability evaluation are scarce. Considerable indirectness (e.g., the use of hypothetical patients [9, 13, 44, 53, 54] and experimental settings [55‐57]) limits their generalizability to actual claimants. The RELY study will use functional evaluations from the real world to establish the inter-rater reliability of medical experts that apply our functional evaluation approach when judging work capacity. A careful analysis of the most important sources of variance affecting disability evaluation (Table 1 and [51, 52]) directed the development of appropriate techniques and instruments to reduce their impact on expert judgment.

The RELY study includes additional methodological safeguards to guard against bias [35, 58]. Consecutive claimants and a convenience sample of psychiatrists with an expected broad spectrum of expertise in medical evaluations will reduce sampling error [59]. Random allocation of three raters to each claimant will prevent group membership and bias. Four ratings per claimant (interviewer and three raters) will increase precision in estimate of the ICC [60]. Videotaping interviews will protect against subject variance where otherwise the claimant may report different things at different occasions. Raters will be blinded to co-raters to ensure independence of ratings, although complete success in blinding cannot be guaranteed. One outcome will assess the typicality of the study claimants and, if confirmed, will strengthen the generalizability of the findings.

Little is known about the impact of the experts’ personal value framework on the kind of efforts they expect from claimants and the level of endurance of strains they expect from claimants to tolerate at work [14, 52]. This potentially important source of variance is not considered in the RELY study.

In our study, we mandate that certain questions be asked but do not otherwise interfere with specific psychiatrist’s approach to disability evaluation. Experts coming from different schools of thoughts (e.g., systemic, behavioral, analytic approach) have developed individualized routines in performing psychiatric evaluations, which may present challenges to experts reviewing videotapes. Informal consultation with the participating psychiatrists revealed that experts should come to similar functional findings, independent of school of thoughts, and that the IFAP-instrument with its thirteen items on functional capacities would guide interviewers to explicitly probe around these work related items during the interview [52]. While reassuring, it remains to be seen whether the psychiatrists’ expectations about the low impact of school of thoughts on functional findings will hold.

Although the IFAP-instrument has not been validated in the evaluation of work disability, the core component – reporting of the functional capacities – has been validated in related settings, such as occupational rehabilitation [29, 30], or social psychiatry [32]. Furthermore the swift spontaneous uptake of Swiss psychiatric experts indicates a high degree of acceptance and face validity.

A major concern remains that high reliability and expert agreement does not necessarily ensure valid evaluations. More conceptual work is needed on how to determine the validity of expert judgments in disability evaluations [34, 52].

Little prior knowledge exists about the processes of usual psychiatric evaluations: How do experts sort the collected information? What are their implicit criteria to select certain information and discard others? What weights do they attribute to the information selected, which makes some information more relevant than others? An optimal research program on disability evaluation would have started by understanding the processes and their relative impact on the critical outcomes before studying the critical societal and patient related outcomes. This approach, however, would have lasted two decades and longer with uncertain results. We therefore used the experience and opinions from our expert group to fill the many gaps we encountered in the design of the study, fully aware of the importance but also limitations of expert advice.

While we expect the sample size to have enough power for measuring the pre-specified reliability with pre-specified precision, it will lack power to detect significant effects when analyzing the evaluation process and predictors for work capacity. These analyses will therefore only inform about trends, they may, however, help in the planning of future studies. Multiple imputation assumes that data are missing at random, and the complete case analysis assumes missing completely at random. Missing data may not be missing at random, a condition that no statistical analysis can address.

Limiting the study to German speaking claimants will exclude many migrants with insufficient language skills who represent a substantial proportion of people applying for work disability.

The RELY study addresses a narrow but important aspect of work disability: Structured exploration of claimant-reported work limitations and standardised reporting of findings about mental functions (IFAP 1) and functional capacities (IFAP 2). Many other aspects impact on variance and reliability of disability evaluation and therefore qualify as high priority topics for research [14, 61]. For example, we need a better understanding on how experts come to their highly complex judgments of an individual’s work ability, and decision-making in complex situations has evolved as a distinct area of research [62‐64]. What kind of claimant information do experts use in their judgments and in what way? How do experts weigh single cues, and how do they aggregate information into a final judgment? What is the experts’ understanding of the functional demands of jobs available in the free labor market? To what degree are they aware of any mismatch between functional job demands and claimants’ functional limitations and how do they integrate such mismatch in their judgment on work ability? How to make the experts’ personal value framework on what to reasonably expect from claimants more explicit? How to minimize the impact of the experts’ personal expectations on the judgment of work capacity and replace them by explicit expectations that result from a societal consensus [65].