EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

Expert opinion is often sought to complement available information needed to inform model-based economic evaluations in health technology assessments (HTAs). In many cases, experts are invited to provide quantitative judgements on unknown model parameters [1]. In the context of HTA, we define expert elicitation as the process of a subject expert specifying a quantity of interest and associated uncertainty around it, which can be encoded as a probability distribution [2]. This enables the probabilistic handling of quantities to address parameter uncertainty.

Expert elicitation is, however, not without risks: not only is guidance scarce for its use in HTA [3], contributing to increased methodological anxiety, but also an impressive body of literature exists on the biased nature of human judgement of probability. Possible common biases may relate to the training or experience of the expert, or even their current mood [4]. Motivational biases may include group thinking (when the desire for harmony or conformity in the group results in an irrational outcome), misinterpretation, wishful thinking, impression management (experts displaying high confidence to protect their public image despite having little actual knowledge on the topic) [5], and experts with the best exposure to the relevant topic wanting to formulate opinions to influence the outcome, irrespective of their true belief [6].

The dominant theory in the elicitation of probabilities is that of cognitive biases. In the 1970s, Tversky and Kahnemann [7] initiated a biases and heuristics research programme, which explored probability judgements. In their early work, they concluded that the instinctive strategies employed by humans to make probability judgements, referred to as heuristics, lead to systematic and predictable error. They concluded that humans were unlikely to achieve acceptable performance in probability elicitation. Their work was widely cited, discussed and adopted by many disciplines.

However, research interest is now slowly moving beyond the “heuristics and biases” research programme [8]. Evidence suggests that although the heuristics described by Tversky and Kahnemann are reproducible, the bias introduced by such strategies could be drastically reduced by changing the task characteristics [9‐11]. Furthermore, in a study where experts provided estimates to technical questions in both their area of expertise and in areas in which they were not experts, Mullin found that, as experts, participants “were far more serious and cautious about the tasks than as non-experts” [12]. Finally, an accepted way to increase the robustness of elicited estimates against bias is to combine the opinions of multiple experts [13‐16].

Even if the risk of bias associated with probabilistic judgement can be significantly reduced, elicitation amplifies the complexity by requiring multiple summaries from the expert to specify a single distribution [2], increasing the burden of the task. Given the enormous potential for a facilitator to reduce bias in expert elicitation, typical recommendations on conducting expert elicitation support a degree of face-to-face interaction with the expert during the elicitation session [2]. The role of the facilitator is to make sure that the questions are clear to the expert, that the expert’s belief is accurately recorded, and the facilitator may even point out fallacies in the expert’s judgement and allow the expert to revise their opinion. Furthermore, experts may feel more responsibility to complete a session with a facilitator than to complete an anonymous questionnaire [17].

Conducting a face-to-face elicitation requires a significant commitment of resources [17] (trained facilitator, travel costs, time); especially when multiple experts are involved. It can be very time consuming, especially in disease areas where there are few experts, who are geographically dispersed, with busy schedules. Moreover, during face-to-face elicitation, the facilitator needs to be careful not to impose their beliefs, or biases, onto the expert’s belief.

Previous work [18] highlighted the critical impact that expert availability has on elicitation, where low availability for face-to-face sessions led to recruitment taking a very long time. Besides ensuring that the question and task are clear to the expert; and the expert’s response is appropriately encoded to reflect their belief, expert elicitation in the HTA context usually has relatively short timeframes, especially for those HTAs informing national and local decision-making. Thus the time available for face-to-face meetings may be limited and might prevent getting estimates from certain experts.

The use of self-administered internet or email-based (distance) elicitation provides an opportunity to reduce time necessary for scheduling sessions, and reduce costs associated with travelling for the sessions, while reaching experts who are farther away. The expeditious nature of HTAs makes distance elicitation a very attractive solution, and examples of its use in HTA are available [19‐22]. It also ensures that all experts receive exactly the same information on which to base their judgements and there is no bias introduced by a facilitator. The downside is the difficulty of formulating the question in a way that experts understand (without additional prompts), and a likely low response rate as with any survey (due to lack of motivation to respond or the cognitive burden of the exercise). However, in a recent ecological study that used self-administered elicitation together with facilitated elicitation, Baker et al. [23] found little difference between expert estimates that could be explained conclusively by the mode of administration.

Essentially, with distance elicitation, more experts can be reached in a shorter time, however usually at the cost of very low response rates, particularly when the target respondents are clinicians [24].

For any elicitation, it is expected that all experts would have the required substantive expertise (knowledge on the relevant topic), but the level of normative expertise (the ability to make quantitative judgements) would vary greatly [25]. There is a risk that participants using distance elicitation, who have the intended expertise but not the skills to adequately formulate their opinion or even use the provided elicitation tool, do not complete the task. However, if they are assisted by a facilitator, they would be able to provide their opinion as probability distributions. This could lead to a possible bias in any aggregation of the individual distributions.

The aim of the study was to assess the practicality of a self-administered elicitation tool by exploring whether the elicited beliefs of the experts were influenced by the mode of elicitation (face-to-face with a facilitator vs. distance self-administered). Response rates among experts agreeing to participate, and time taken to complete the task, were also compared for the two modes. We hypothesised that, although potentially victim to a lower response rate, the distance approach would produce a higher number of complete responses, due to a larger potential sample than face-to-face elicitation.

The objectives of the study were to:

Using an Excel-based elicitation tool, probability distributions were obtained with the stated intention of informing a hypothetical decision problem.

Recruitment was very slow, taking several months. Even among the experts who explicitly agreed to participate, some dropped out of the study before completing the elicitation session. Recruiting through snowballing was more successful than approaching potential participants through the official research and training organisations. Some participants commented that this might be because official channels were saturated with various requests and many GPs would ignore messages not directly related to their activity. Another possible barrier to recruitment in the early stages was the lack of incentives for the participants. Towards the end of the recruiting period, a new invitation to participate was circulated where the focus was changed from the methods explored to the drinking behaviours topic, which recognised the potential for learning more about representing uncertainty. This might have contributed to more participants joining the study towards the end. Nonetheless, our approach to sampling, as well as the relatively small number of respondents, limit the generalisability of the findings.

It took longer to obtain data from the email participants than it took to schedule meetings in the face-to-face arm contrary to the hypothesis that email would result in more responses more quickly. A possible explanation is that the email participants were less motivated to engage in and complete the elicitation, because the objective and its deadline were perceived as very abstract. In contrast, by scheduling a time for the elicitation session, the objective was clearer for the participants in the face-to-face group. These findings seem to confirm the evidence collected previously [18] regarding practical issues with the duration of expert recruitment.

Our findings show that it took longer for experts to complete the elicitation session by email than in face-to-face sessions. Participants in the email arm spent more time on both the training part and the questionnaire sessions than the face-to-face arm. The differences were, however, only a few minutes. Facilitation may have helped by increasing the confidence of the participants in the face-to-face arm. All of the sessions (whether face-to-face or distance) were completed in a reasonable time. One of the possible reasons for the good rate of successful elicitation sessions was the elicitation tool, which achieved a good balance of instructions and tasks, while also managing the cognitive burden of the exercise.

The individual distributions elicited in the face-to-face and distance arms were comparable. The distributions elicited by email were more uncertain, with wider intervals; however the means were more grouped together than in the face-to-face arm. It is unclear if this was an effect of different approaches, or a difference in the participating experts, for instance, the difference in familiarity with statistics as rated by the experts. When combining individual distributions for each arm, the aggregated distributions were still comparable. These findings were similar to existing studies exploring self-administered elicitation [23]. We did not perform further quantitative comparisons of the distributions; however, given the context of a decision model used for HTA purposes, the ultimate practical comparison should have been a comparison of model outputs when the model is informed by each of the distributions.

Because of the computer-assisted nature of the task, it was possible, in theory, to favour the recruitment of experts who had better computer literacy, but not necessarily “better” expertise. However, to compensate the impact of selection bias, the elicitation question addressed a relatively common phenomenon that most GPs were believed to be able to express in a quantitative way. A ‘Think Aloud’ analysis [43] would have provided deeper insight into the interaction with the tool and respondents’ choices; this was not done as part of the study, but such a protocol is been prepared for the future development of EXPLICIT.

Another potential source of bias is the recording of inconsistent distributions. This is a risk especially with distance elicitation. EXPLICIT limited the possibility of this situation occurring by doing several real-time consistency checks to the summaries introduced by the experts. However, EXPLICIT does not prevent the expert from building a highly skewed distribution, even when common sense might suggest a normal distribution. Arguably, for facilitated sessions, such situations could be discussed between the facilitator and the experts, leading to a more consistent distribution, but this was not possible for email elicitations.

Finally, distance elicitation does not have to be completely devoid of facilitation. This study attempted to isolate the impact of facilitation by only providing it in the distance arm through the elicitation tool. It would be possible, however, to provide facilitation by phone or in a video-conference to an expert. Scheduling phone calls to take the expert through the elicitation exercise might also increase the chances of obtaining the responses in a timely manner.

Outcomes were comparable between the face-to-face and email elicitation groups. Recruiting is, however, a key issue and the potential experts must be vigorously pursued, especially in a time-limited HTA context, regardless of whether the session is conducted face-to-face or remotely; this is an even more critical issue if the experts are scarce. Remote self-administered elicitation can be a viable approach when experts are not immediately available for a face-to-face session and EXPLICIT can be used successfully to achieve this, possibly at the expense of poorer response rates and additional time needed for the elicitation sessions.