Background and rationale
The Delphi method is used widely to achieve consensus among experts on a particular topic [
1]. It was developed first by Dalkey and Helmer [
2] as a way to establish the opinions of experts anonymously on issues related to sensitive military operations. Since then, the Delphi method has transitioned from a pen-to-paper approach to the e-Delphi method, where surveys are carried out online using the Internet with the potential for a global reach and potentially achieving consensus faster than pen-to-paper versions.
The Delphi method, in general, involves multiple stages or rounds of a questionnaire. It is an iterative process. Participants complete an initial survey. When this survey round closes, participants are sent “feedback”. This feedback often includes information such as how the participant answered each question/item on the survey and how each item/question was answered by the participants overall (e.g. the average or median score). Participants are often grouped into subgroups, also referred to as “stakeholder” groups. Each stakeholder group contributes a different expertise based on their connection and experience of the condition being investigated. Responses can be aggregated at the whole group level or at the level of varying stakeholder sub-groups, e.g. patients, healthcare providers and researchers. Participants are given the opportunity to modify how they responded to each question in subsequent rounds with the knowledge of how other participants/participant groups answered the questions. In so doing, it is anticipated that respondents will converge on responses to items within the survey. Each participant remains anonymous to other participants throughout the Delphi process.
The Delphi technique confers several advantages over other methods of reaching consensus such as forums or discussion meetings [
3]. Maintaining anonymity by avoiding personal identification of participants from one another is viewed as a strength in achieving consensus, by enabling stakeholders to provide their opinion without the process becoming dominated by more assertive individuals. This idea, of gathering the expertise of a variety of stakeholders to establish consensus on essential outcomes for a particular disease or condition, provides a strong rationale for its use in the development of core outcome sets [
4].
A core outcome set (COS) is an agreed minimum set of outcomes or outcome measures. It is a recommendation of “what” should be measured and reported in all trials, other studies and potentially routine clinical practice in a specific area. The Delphi method is used commonly to achieve consensus in COS development [
4] where participants rate the importance of outcomes for inclusion in a COS. The process followed is otherwise the same as outlined above.
Concerns have been expressed about the lengthy process of completing multiple Delphi rounds, and time waiting for feedback on consensus results between rounds, as potential causes of lost interest and dissatisfaction among participants [
5]. These issues may contribute to challenges in recruitment and retention of participants in this phase of the COS development process.
A “Real-Time Delphi” process may offer benefits in the development of COS. The Real-Time Delphi method was developed by Gordon and Pease [
6] to improve the speed and efficiency of gathering opinions of experts and making decisions in situations of urgency. As software has developed since then, so too have approaches to using the Real-Time Delphi method.
The Real-Time Delphi approach maintains the benefits of working toward consensus in a survey setting but does so in a potentially more time-efficient manner. Feedback can be given to participants on the web page in “real-time”. This provides feedback to participants more quickly than at the end of a Multi-Round Delphi survey round, which often last up to 3 weeks in COS [
4]. The Real-Time Delphi method comprises a round-less Delphi approach [
6], where participants are encouraged to re-visit the survey and re-rate items throughout the period in which the Delphi survey is live. Essentially, it removes the time taken in the Multi-Round Delphi for the survey administrator to evaluate the results and provide feedback to participants. Participants can engage with the consensus process from the outset by seeing how other participants have answered the questions and potentially modifying their answers.
Studies comparing traditional-style Delphi surveys and a “Real-Time” Delphi survey approach include a comparison by Geist [
7] of a traditional, Multi-Round, pen-to-paper approach with a Real-Time Delphi model, which found that attrition was lower in the Real-Time Delphi arm. The authors suggested this may have been due to a lack of email prompts engaging participants. This study acknowledged that with improved modelling, including increasing the number of reminders to participants to re-visit the Real-Time Delphi, this approach could be a cost-effective and efficient mode of achieving consensus. Gnatzy et al. [
8] conducted a comparison of a traditional, sequential round Delphi and a Real-Time Delphi to see if the survey results were affected by changing the survey method (i.e. using a Real-Time Delphi method). However, although this group describe an in-depth approach of evaluating a Real-Time Delphi model, participants were not randomised, and the surveys took place with participants from different countries and at different times. Any or all of these factors could affect the results of comparing these two Delphi surveys.
Thiebes et al. [
9] discuss how the Real-Time Delphi deviates from the characteristics of a Delphi survey as outlined by Rowe, Wright and Bolger [
10] who distinguish four key features that characterise a Delphi survey, i.e. “anonymity”, “iteration”, “controlled feedback” and “statistical group response”. “Anonymity” is key within the Delphi process, whereby participants remain anonymous from one another throughout the survey, encouraging all participants to give their opinions without being influenced by more influential individuals. This feature is maintained in the Real-Time Delphi design.
“Iteration”, according to Rowe, Wright and Bolger [
10], refers to presenting participants with a questionnaire comprised of multiple rounds and facilitating participants to change how they answer a question. Although the Real-Time Delphi is essentially round-less, it can incorporate iteration by encouraging participants to re-visit and re-answer questions based on changing feedback. This feedback differs from the definition of “Controlled feedback” which Rowe, Wright and Bolger [
10] suggest should be given between rounds. Instead, in the Real-Time Delphi, this feedback, such as how the participant answered the question and how other participants, as a group responded to the question, is provided once the participant has submitted their answer. “Statistical group response” refers to the feedback that is provided to the participants. Instead of the feedback being reflective of all participants who have taken part in the survey after each round, the feedback in the Real-Time Delphi is reflective of the participants who have taken part up to that point in the survey. This is why participants are encouraged to re-visit and re-respond to questions if they wish, to get a reflection of the views of participants at different time points and to capture any changes in consensus. Whilst in a Real-Time Delphi, feedback is not “controlled” by limiting feedback only at the end of distinct rounds, it can be argued that the Real-Time Delphi approach still largely incorporates the key features of the Delphi procedure.
Despite claims that a Real-Time Delphi will improve efficiency of the survey process by removing the strict rounds and the time it takes for participants to complete the survey [
6,
8], there is no strong evidence on how it compares to the Multi-Round Delphi.