Developing outcome, process and balancing measures for an emergency department longitudinal patient monitoring system using a modified Delphi

The methodology to develop the evaluation measures was a two-stage process. Stage One consisted of a workshop with the research programme’s Participatory Action Research Group (PAR) group. All 13 members were invited to attend. Ten members attended (Table 1). The purpose of the workshop was to build on the collective professional experience of the members to gather suggestions regarding the evaluation measures to be used for the implementation of ED-ACE. Participants were informed about and asked to consider the Institute for Healthcare Improvement (IHI) framework for developing outcome, process and balancing measures [8]. The ‘stickies’ method was used to allow each individual to generate suggested outcome, process and balancing measures [9].

In Stage Two, an electronic modified-Delphi study was undertaken to develop a consensus on a suite of measures to be used for the evaluation of ED-ACE [10]. The evaluation workshop was the modification on the Delphi and the rest of the process proceeded as a standard Delphi. A multidisciplinary Delphi panel was created to include the research team (including Health Systems, Epidemiology and Public Health, Patient Q&S, and Human Factors researchers); members of the project’s Scientific Advisory Group; Consultants, Registrars, Advance Nurse Practitioners (ANPs) from the ED at the planned implementation site who had not been involved in the research; all members of the EMP and the EMP’s Emergency Medicine Nursing Interest Group (ENIG); the Lead EM Consultant and EM nursing leads in all similar-sized EDs in Ireland. Fifty-eight professionals in total were invited to participate, 49 of which participated in Round 1 and 39 in Round 2 (Tables 1 and 2).

The Delphi approach was used in this study to reach consensus because other commonly used consensus group methods (for example, focus groups) were not cost-effective and feasible due to the fact that the panel for this study represented diverse geographical locations, making it impractical and costly to meet in person [11]. Other reasons for using the Delphi consensus approach for this study include participant anonymity (to each other, though not to the study’s lead researcher) and the avoidance of groupthink or domination that might arise in a face-to-face discussion [12]. Using the multi-disciplinary expertise of the PAR group to develop and categorise the initial set of potential measures was an innovative and unique element of this study.

The evaluation measures for the Delphi study were entered into SurveyMonkey software (https://​www.​surveymonkey.​com/​) to create an online/web-based electronic questionnaire that was used for both rounds of the Delphi study. A pilot study was then carried out based on three members of the research team assessing the content and flow of the draft questionnaire as well as ensuring the clarity of the measures and their categorisation. The feedback comments from the pilot study were used to create the final study questionnaire and an email containing the web link to the questionnaire was sent to all the Delphi panellists. This email also included a cover letter outlining the overall study objectives, how the initial list of measures was developed and an explanation of the Delphi process. Participation was on a voluntary basis and in keeping with the Delphi process participants were assured that their responses would be anonymous. Completion of the questionnaire was also taken as consent to participate. As the completion of the study questionnaire was anonymous, a background section called ‘Source of Expertise’ (Part A) was included in the questionnaire to capture background information (Please see Additional file 1: Appendix A for the Delphi R1 questionnaire).

Descriptive statistics were used to summarise the characteristics of the panellists who completed the questionnaires in each round. The mean Likert rating, standard deviation and 95% bias-corrected bootstrapped confidence interval (CI) for each variable was calculated. Greatorex and Dexter [13] concluded that, although statistics such as the mean and standard deviation assume an interval scale, the mean can be understood to represent group opinion and the standard deviation the amount of disagreement within the panel. Missing data was due to dropout with some panellists failing to progress through consecutive sections of the questionnaire (see below) although where panellists ranked measures within a section they ranked all measures. All responses and rankings were included in the analyses.

Ensuring participant anonymity meant it was not possible to analyse at the individual (panellist) level change in relation to how measures, which were included in both rounds of the Delphi process, were ranked. However, where a variable reached the ‘high agreement’ threshold in R2 but not in R1, a one-sided test was used to determine whether the proportion of panellists who rated the variable as ‘Important’ or ‘Very Important’ in R2 was significantly higher than in R1. Bonferroni corrections were applied to take account of multiple testing. Data were analysed using Stata 14.0 SE.

An initial list of 90 potential measures was developed at the workshop. After duplicates were removed there were 73 potential measures remaining (Fig. 1). A scoping literature review was conducted to explore evaluation measures used in other studies of longitudinal patient monitoring systems being implemented in the ED setting. Two additional measures were added following the literature review.

Following the workshop, the list of proposed measures was categorised into outcome, process and balancing measures and more specifically into the following categories:

The first round (R1) Delphi questionnaire contained 75 potential measures which participants were asked to rate on a 5-point Likert scale as very unimportant, unimportant, neither unimportant or important, important, very important for the evaluation of ED-ACE. This phase of the research was about determining the perceived importance of each measure by the key stakeholders. Assessing measures for feasibility and determining minimum data sets for measures would come later. At least 70% of Delphi panellists had to rate a potential measure in the ‘high agreement range’ (‘important’ and ‘very important’), for the measure to be selected for inclusion in the final suite of measures [11]. Participants were also invited to add any additional measures they felt should to be included in the Delphi round two (R2) questionnaire. The R1 questionnaire was emailed to 58 participants in August 2015. Personalised emails were sent to all the participants external to the project to improve the response rate. Two separate email reminders were sent after 3 weeks and then after one further week. The survey closed on 15th September 2015. Responses to R1 were analysed and proposed additional measures were collated (Fig. 1 provides a flowchart of the entire process). For R2 all 58 R1 participants were again invited to participate. This time they were asked to rate 27 potential measures. As in R1, personal email invitations to participate and reminders to the Delphi panellists were sent in R2.

Table 2 details the characteristics of the Delphi panellists. The same group of 58 stakeholders were invited to participate in both rounds of the Delphi study. The response rates were 84.5% for R1 and 67.2% for R2. In R1 thirty-nine panellists rated the treatment process measures, thirty-five the implementation process measures, thirty-four the outcome measures and thirty-one the balancing measures. In R2 thirty-five panellists rated the treatment process measures, thirty-two the implementation process measures, and thirty-one both the outcome measures and the balancing measures.

Twenty two variables, which did not meet the threshold for inclusion following R1, were revised if considered necessary for clarification, and panellists were asked to rate these again in R2 (in addition to other variables suggested by the Panellists in R1). 13 of these 22 variables were rated as ‘Important’ or ‘Very Important’ following R2 and these are listed in Table 3. However the proportion of panellists who ranked the variables as ‘Important’ or ‘Very Important’ in R2 was only significantly higher than in R1 for five of the variables. One of the 22 variables, “Number of patients who are in ED waiting for in-patient beds”, had the highest mean Likert score of all balancing measure variables across both rounds (4.33, 95%CI (3.98,4.63)).

A suite of 69 process (treatment and implementation process), outcome and balancing measures for evaluating the effectiveness of ED-ACE were developed (Fig. 1). Table 4 details the three measures in Delphi R1 and R2 with the highest mean Likert ratings per category thus leaving a set of the 12 highest-ranked measures. The purpose of using the Delphi technique was to reach consensus on which process, outcome and balancing measures would be the most important to evaluate ED-ACE. Table 4 therefore includes both those measures that reached consensus in R1, and therefore excluded from R2, as well as those reaching consensus in R2 across the four categories of process (treatment and implementation process), outcome and balancing measures [8]. Thus, while some clinical treatment variables may have rated higher than the top three balancing variables it was considered important to represent the top three variables across the four categories.Additional file 2 Appendix 2 provides a list of the mean Likert ratings for all of the proposed measures, the 69 that reached consensus as the most important and the remaining 34 that did not reach consensus.

Firstly, valid and reliable measures depend on the availability of high quality data [20] and while this study is the first step in developing consensus on a suite of measures, a separate study is required to determine how feasible and cost-effective it will be to collect the minimum data set required for implementing the measures [21]. Secondly, while we considered the views of frontline staff and researchers, there was no patient representation in the consensus development process. Thirdly, while the response rate to R1 was high (84.5%) this did drop to 67.2% in R2. In order to maintain the rigour of this technique, a response rate of 70% is suggested, but as Hasson et al. [22] note to achieve this, the researcher must know the identity of respondents, and non-respondents must be pursued individually. Given that our study was anonymous and we could not pursue participants individually a response rate of 67.2% can be considered acceptable. Also given that our study was anonymous we were unable to examine changes within individual respondents between rounds. Finally, the study investigators assumed that all the Delphi panellists had relevant knowledge regarding the evidence base for all the measures that reached consensus. However, it is conceivable that not all the panellists were aware of the evidence supporting all the measures that reached agreement.