Be careful with triage in emergency departments: interobserver agreement on 1,578 patients in France

A multicentric cross-sectional study was conducted over a 3-day period (a weekday and two weekend days), in April 2007, in a sample of EDs located in the Provence-Alpes-Côte d'Azur (PACA) region, in France. This region has a population of 4.8 million which represent 7.6% of the population of France, and covers 34,400 km² with population densities from 153 persons per km² [27]. The PACA region is served by a total of 53 EDs, which treated between 11,000 and 65,000 ED patients per year. The distribution of 53 EDs allows to 99% of the population access to an ED in less than 45 minutes (and 85% in less than 15 minutes). Private (17%) and public (83%) hospitals were represented.

The 53 EDs were classified according to the following two strata: the number of annual visits to these EDs (high attendance (25 000 or over visits per year) and medium or low attendance (less than 25 000 visits per year)) and, the geographical location of EDs (located urban area characterized by higher population density with at least 2 000 residents and by the urban-type land use, not allowing any gaps of typically more than 200 meters [28] or not). Finally, 17 EDs were randomly selected among the 53 EDs according the two strata. Table 1 describes the characteristics of these 17 EDs.

All patients aged 18 years and older who presented in one of participant EDs between the hours of 8 AM and 12 midnights were included. Study hours were limited because of few patients come after midnight [29]. Patients were excluded if they required immediate medical care and had communication difficulties.

Immediately after the admittance and nurse triage, all patients agreeing to participate were interviewed face to face in the ED by a trained research assistant who was not involved in care. The anonymous standardized questionnaire collected the following variables [Additional file 1]:

Moreover, during their activity, trained triage nurses, after the admittance, and trained ED physicians, immediately at the end of the consultation, were asked to complete the questionnaire for each patient seen [Additional file 2]. They independently gave their expert opinion concerning the urgency of the admission of the patient. All ED health professionals had at least one year' experience of the ED.

Data were analyzed on Spss 16.0 by using proportions or means, and standard deviations of all variables. The main outcome variable was whether the ED visit was urgent or not.

To evaluate the level of agreement on triage categories between nurses and ED physicians, we calculated the chance-adjusted measure of agreement (Kappa-value) from 4 × 4 tables. Qualitative descriptions of agreement were as follows: 0.81-1.0 = "almost perfect", 0.61-0.80 = "substantial", 0.41-0.60 = "moderate", 0.21-0.40 = "fair", 0.0-0.20 = "slight" [31]. Kappa-values are reported with 95% confidence intervals (CIs). Sensitivity, specificity, and positive and negative predictive values (PPV and NPV) of accuracy of categorization into urgent and nonurgent case between triage nurses and ED physicians who were the reference. To assess the discrimination power of this model, a receiver operating characteristic (ROC) curve was constructed. The ROC curve is a graphic method for indicating the trade-off between the true-positive rate (sensitivity) and the false-positive rate (1 - the specificity) of a test or diagnostic manoeuvre. Generally, the most discriminating tests have the largest area under the ROC curve, the maximum being 1.0 [32]. Moreover, sensitivity, specificity, PPV and NPV of decision for hospitalization were calculated. For these analyses, ED patients hospitalized at the end of the consultation were compared with patients categorized into urgent or nonurgent cases by triage nurses and by ED physicians.

Analyses of agreement were performed within subgroups stratified by explicit criteria. Subgroups were defined according to the following criteria:

Kappa-values with 95% CI were analyzed within all these subgroups.

Our study is a non interventional research and does not need to be approved by an ethics committee under the criteria of the bioethics law. So, our study does not require the authorization of the National Commission for Informatics and Freedom due respect for patient anonymity [34].

Of the 1,578 patients included in the study, 52.4% were males and the mean age of ED patients (± standard deviation (SD)) was 45.2 years ± 21.4 (from 18 to 100 years); 14.3% of patients were 75 years old and over. Most patients had primary health insurance with supplementary coverage (86.0%); 10.4% of them were covered by French health insurance specifically for individuals and families with low incomes and resources (named "CMUC"). The majority of included patients were followed by a general practitioner (92.9%). More than one third suffered from chronic disease (36.7%).

Presenting complaints had lasted less than 24 hours for 77.7% of patients. Only 17% had been referred to the ED by a primary care physician. The others were self-referred (63.4%) or referred for medico-legal reasons (19.4%) (Employer, school, police...).

More than half of patients were consulting the ED for non-trauma complaints. Nearly two thirds of ED patients received diagnostic tests; 59.2% received treatment in the ED, and 22.7% were hospitalized.

Of the 1,578 ED visits, the proportion of nonurgent ED patients was 26% according to triage nurses upon the entry, and 34.3% according to ED physicians at the end of the consultation (p < 0.001, Table 4). Overall level of agreement was moderate (kappa = 0.43 ± 0.02; 95% CI, 0.39% to 0.48%). The model showed a high sensitivity of 88.0% (Table 5). The area under the ROC curve was 0.70 with 95% CI 0.68 to 0.73 (Figure 1).

Table 4 shows the distribution between triage categories determined by triage nurses upon the entry to the ED and by ED physicians at the end of the consultation. Of the 1,036 patients categorized as urgent by ED physicians, 124 (12%) of them were categorized as nonurgent by triage nurses. These 124 patients were, for the majority, women (54%), self-referred (68.0%) and suffering from a medical problem for more than 24 hours (29.0%).

Within the 17 EDs, the levels of agreement were variable, ranging from 0.21 to 0.71. The highest kappa value concerned an ED with the smallest number of patients (n = 31).

Table 6 shows results of analyses in subgroups. The levels of agreement within all subgroups based on explicit criteria were low (from moderate to slight) except in 3 subgroups of case mix.

The levels of agreement within the 22 subgroups of complaints were variable, ranging from 0.09 to 1.00. Among the 22 subgroups, 10 showed fair inter-observer agreement (k = 0.21-0.40) and 7 moderate agreement (k = 0.41-0.60). The lowest level of agreement concerned the subgroup of urinary-nephrology (k = 0.09, slight). The highest kappa-values concerned three subgroups of complaints: cranial injury (k = 0.61, substantial), gynecological complaints (k = 0.66, substantial) and toxicology complaints (k = 1.00, almost perfect).

For the other subgroups, levels of agreement were also low (from 0.20 to 0.47) and showed considerable variability. The lowest level of agreement concerned the subgroup of hospitalization (k = 0.20, slight) and the highest concerned the three following subgroups: duration of the presenting complaint (> 24 hours, k = 0.47), suffering from chronic disease (k = 0.47) and self-referral (k = 0.46). These three levels of agreement were moderate.

Hospital admission is not a relevant indicator. The distribution of categorization of urgency relative to hospitalization status is shown in Table 7. Whatever the professional who conducted the categorization (triage nurse or ED physician), most urgent patients were not hospitalized. Among the 409 nonurgent patients identified by triage nurses, 9% were hospitalized. These patients had no specific characteristics. Similarly, among the 536 nonurgent patients identified by ED physicians, 18 were hospitalized (3.4%). The majority of these 18 patients were older (70%, mean age 69.2 years ± 4.7; median 79.5 years), and reported neuropsychological problems (20%) and alteration of clinical status (20%).

When categorization of ED visits into urgent or nourgent cases was compared to hospitalization, ED physicians had higher sensitivity and specificity than nurses (respectively 94.9% versus 89.5%, and 43.1% versus 30.9%). Overall, for ED physicians and triage nurses, positive predictive values were low (32.8% versus 27.5%) and negative predictive values were higher (96.6% versus 90.9%) [Table 8].

Several potential limitations should be addressed. Firstly, while we examined in great detail the different sub-groups based on following explicit criteria: age, medical status, and type of referral to the ED, we did not analyze the impact of the trained ED health professionals themselves. We conducted the study with ED health care professionals present during the inclusion period, in the usual manner, i.e. without the use of written protocols or algorithms. However, in previous studies measuring level of agreement, training, experience, knowledge, and skill of ED health professionals did not influence kappa values [35, 42, 43]. The authors found substantial disagreement even among health care professionals with the same training. Secondly, when designing our test study, sample size calculation should have been performed in order to guarantee the design accuracy. But, we performed a sample size calculation retrospectively based on the methodology of Flack VF et al [44]. Data were analysed on PASS 2008. In a test for agreement between two raters using Kappa statistic, a sample size of 1,986 subjects achieves 80% power to detect a true Kappa value of 0.43 in test of null hypothesis: Kappa = 0.50 versus alternative hypothesis: Kappa <> 0.50 when there are two categories with frequencies equal to 0.70 and 0.30. This power calculation is based on a significance level of 0.050. Thus, we included 1,578 patients in our study. Moreover, we found six similar studies which compared different methods of categorization in the same population [35, 40]. These articles showed considerable variability in levels of agreement between the different methods to categorize ED visits into nonurgent or urgent cases, ranging in κ value from 0.20 to 0.74. These studies did not perform a sample size calculation.