Validity of the Paediatric Canadian Triage Acuity Scale in a Tertiary Hospital: An Analysis of Severity Markers' Variability

Triage systems are inherent to the functioning of Emergency Services, establishing a hierarchy of care based on clinical risk [1, 2]. With the increasing influx of patients and frequent overcrowding [3], the adoption of a valid triage system, capable of distinguishing patients who need urgent care, from those who can wait safely, becomes paramount [4‐9]. In Paediatrics, triage is an even more challenging process, due to the clinical and psychosocial characteristics of these patients and their caregivers [10, 11]. Thus, it is essential that emergency services that serve children and adolescents adopt validated models and are applied by properly trained and certified staff.

Developed in 2001 and reviewed in 2008 and 2012, Paediatric Canadian Triage and Acuity Scale (PaedCTAS) is the paediatric triage scale adopted by the Advanced Paediatric Life Support from the American Academy of Paediatrics, by the American College of Emergency Physicians (ACEP) and Canadian Association of Emergency Physicians among others [12]. It is used in several emergency services in Canada, United States and several European countries [13]. It is used in 3 out of 14 Paediatric Portuguese emergency services.

According to the ACEP and the Emergency Nurses Association (ENA), the ideal triage scale must demonstrate the characteristics of reliability, validity, utility and relevance [10]. The validity of triage systems depends on their ability to discriminate between different levels of urgency, reflecting the patient's true acuity [14].

In the absence of a gold standard for “urgency” to assess the triage’s validity, several parameters have been used, including mortality and hospitalization rates [15‐17], expert opinions, Paediatric Intensive Care Unit (PICU) admissions [15, 17], length of stay [15, 16, 18], number of complementary diagnostic tests performed [17, 19], cost [19] or a combination of several indicators [20].

Several studies have evaluated the validity of the Canadian Triage and Acuity Scale(CTAS) [19, 21]. However, most of the existing studies have been conducted in Canada [22] and only a few in an European context [13]. Moreover, the studies also fail to compare the validity of surrogate outcomes markers of severity for all possible cut-offs in triage levels, needlessly aggregating information from different levels and therefore losing possibly valuable information. It is also important to assess this issue in particular regional contexts, given that the results may vary widely [13]. To the best of our knowledge is also the first PaedCTAS validity study carried out in Portugal.

Thus, we aimed to evaluate the validity of PaedCTAS triage system in a metropolitan, tertiary, university-affiliated Portuguese hospital's Paediatric Emergency Department (PED). Furthermore, we aim to study the performance and appropriateness of the different surrogate severity markers to validate triage.

This is an observational, retrospective study that took place in PED of a metropolitan, university-affiliated hospital with a catchment area of approximately 800 thousand inhabitants, receiving approximately 76,000 visits per year from an estimated population of 137,016 children or adolescents [0–17 years] [23].

At any moment, in the PED, there are always 2 senior physicians, trained in Paediatric Emergency Medicine, 2 to 3 residents (depending on the workload), 8 nurses and 3 auxiliary staff per shift. All these teams work in 12-h shifts providing 24 h per day coverage.

The PED nursing team triages visitors from Level 1 through 5 according to the PaedCTAS [24]. To each level is assigned a different degree of urgency i.e. Level 1- “Resuscitation”, Level 2- “Emergent”, Level 3- “Urgent”, Level 4- “Less Urgent” and Level 5- “Non Urgent”, that classifies the patient based in the 3 steps: (1)the initial general state of the patient, including appearance, capillary perfusion and respiratory effort; (2)the assessment of the main complaint; and (3)the evaluation of vital signs, taking into account age and associated risk factors. Pain also plays a decisive role in this classification. Expected waiting times to be seen by a physician or reassessed are established for each priority level, i.e.level 1 are seen immediately, level 2 can wait up to 15 min, level 3 up to 30 min, level 4 up to 60 min and level 5 up to 120 min. The triage was translated to Portuguese and the translation validated with the National Emergency Nurses Association (NENA) [25]. NENA also provided training to Portuguese nurses on the PaedCTAS.

The triage itself is performed by a specialized triage nurse, and the triage level is assigned by the triage algorithm, implemented in the hospital’s information system i.e.electronic health record.

Besides having an Intensive Care Unit (ICU), this hospital also has an Intermediate Care Unit (IMCU), for patients with conditions that do not require intensive care but are also not appropriate for general admission.

In this study, all visits made by patients who were admitted the hospital’s PED (i.e.from 0 to 17 years old) in a 6-year period (between 01/Jan/2014, and 31/Dec/2019) were considered. Deceased children were excluded due to its small number and children that left without being seen and left against medical advice were excluded due to the inability to measure the surrogate outcome markers. All exclusion were reported.

For the main analysis the predictor variable used was the level assigned by the PaedCTAS. All five triage levels were split in two classes, as there are 5 levels, 4 cut-off points were used. The first cut-off considers level 1 as urgent and all the other levels non urgent i.e., 1.2345. Henceforth, this will be the nomenclature used i.e., the dot will be used to separate urgent and non-urgent levels.

The studied outcome measures, used as severity markers were:· “Admitted to intensive care”, patient’s discharge destination from the PED was the hospital’s ICU;· “Admitted to intermediate care” , patient’s discharge destination from the PED was the hospital’s IMCU;· “Admitted to hospital” patient’s discharge destination from the PED was the hospital’s inpatient care;· “Investigations performed in the PED” , which reflects the situation when a patient is asked to stay in the ED, for the physician to better assess the condition’s evolution e.g. concussion;· “Uses PED resources”, the patient is considered to “use PED resources” if during the visit the patient was medicated or if laboratory or radiologic exams were performed. The variables “Admitted to intensive care”, “Admitted to intermediate care” are subgroups of the patients “Admitted to hospital”.

For every outcome and cut-off, the sensitivity, specificity, positive predictive value(PPV), negative predictive value (NPV), positive likelihood ratio (LR +) and negative likelihood ratio (LR-) were calculated with 95% confidence intervals (CI) [26, 27]. For each outcome, Receiver Operating Characteristic (ROC) curves were plotted and the area under the curve (AUC) were calculated with 95% CI [28].

All the data analysis was performed in R version 3.4.1 (2017–06-30) [29]. The integrated development environment (IDE) used was RStudio Version 1.1.383 [30].

Ethical committee’s approval was granted for this study [FMUP 180/18].

This paper follows the structure presented in the RECORD statement i.e. The REporting of studies Conducted using Observational Routinely-collected health Data [31].

From the total of 467,917 visits to the PED within the study period, 15,119(3.23%) were removed due to the exclusion criteria and missing data, remaining 452,798(96.77%) for analysis. Deceased children(n = 15), patients that left without being seen (n = 10,658,2.28%), patients that left against medical advice (n = 257,0.05%) and missing data on the triage level variable (n = 1,859,0.40%) were removed from the analysis n.b. there is one overlap in the exclusions, therefore the sum of individual exclusions is higher than the total observations excluded.

Only one of the deceased children was not triaged level 1. The child was triaged level 2, had the first contact with the doctor 6 min later and died 17 h later, the child was transferred from another hospital and suffered from several comorbidities.

There were 1859 (0.4%) missing values in the variable “Triage Level”. Regarding the evaluated severity markers there were 4179(0.89%) missing values in the variable “admitted to hospital”, there was no missing data in the other markers.

Population characteristics are described in Table 1. There is a low variability of influx through the study years and the lower attendance during the summer. Regarding the mode of arrival of patients, most are walk-in patients, ranging from 63.1 to 92.7% for different triage levels. Most patients have home as their discharge destination, ranging from 70.1% to 93.3%, being patients triaged level 1 i.e. resuscitation, the only exception (27.5%).

The severity markers by triage level are shown in Table 2. It is important to notice the low frequency of the severity markers Admitted to intensive care and Admitted to intermediate care. It should be pointed out the increase in proportion of hospital admissions and admissions to intermediate care from level 4 to 5.

The results of the triage system’s performance as a predictor by surrogate severity markers are presented in Table 3. It shows similar trend sensitivity in all surrogate markers, lowering their values as the cut-off lowers in the triage levels. Considering the LR + the best surrogate marker was “admitted to ICU”, followed by “admitted to hospital” and “Investigations performed in the PED”. However, between the latter surrogate markers the best performance depends on the cut-off. The same is observed regarding LR-.

Figure 1 shows the ROC curves and AUC with 95% CIs for each surrogate severity marker. It should be pointed out the good performance of the severity marker “Admitted to the ICU” and the similar performance of “admitted to hospital” and “Investigations performed in the PED”, better than “Admitted to IMCU”. Furthermore, it is important to notice the high sensitivity and specificity of the second cut-off(i.e.12.345) regarding “Admitted to the ICU”, and the similarity in sensitivity and specificity of the third cut-off for the, “Admitted to intermediate care”, “Admitted to hospital” and “Investigations performed in the PED” surrogate markers, especially in the third and fourth cut-off points.

The major objective of this study was to evaluate the validity of the PaedCTAS. Additionally, we studied the appropriateness of the different surrogate severity markers to validate triage.

Triage system’s validity refers to the triage system’s ability to predict ‘true’ urgency. However, the concept of ‘true’ urgency is impossible to measure since the study of the impact in delayed treatment to a patient would be unethical [32]. There are two major methodologies for triage validation:(1)those using criterion validity i.e. reference standards developed by expert panels or other triage systems; and (2)those using construct validity i.e. severity proxies [33]. In the context of diagnostics research and using surrogate markers of severity, validity can be expressed in sensitivity and specificity of a triage system, or their ratio i.e. LRs. Sensitivity represents the ability for a triage system to identify high urgent patients. Specificity represents the ability for a triage system to identify patients with low urgent problems [17]. Although there are no recommendations about the safe limits of sensitivity, under-triage or over-triage rates for emergency triage systems, an effective screening tool is expected to prioritize sensitivity, since under-triage (real high urgent patients triaged as low urgent) is unsafe [17]. On the other hand, high sensitivity may result in a low specificity. Therefore, good balance between over and under triage is important, so that “true low urgent” patients are not triaged as high urgent, compromising the flow of patients, delaying the observation of real high urgent patients and compromising their safety [34]. Going further, the balance between under-triage and over-triage i.e. sensitivity and specificity is already captured by the concept of likelihood ratios. Positive likelihood ratio answers the question: How likely is a person to be correctly triaged as urgent when compared to a person wrongly triaged as urgent? The larger the LR + , greater the likelihood of being urgent, a low value, close to one, suggests over-triage. Conversely, a negative likelihood ratio answers the question: How likely is a person be wrongly triaged as non-urgent when compared to a person correctly triaged as non-urgent? The smaller the LR-, the lesser the likelihood of being urgent. A high value, close to 1, suggests under-triage. They have a similar interpretation to PPV and NPV but without being influenced by prevalence [35].

This is the first study comparing surrogate outcomes markers of severity considering all possible triage cut-offs in the triage validation process, questioning the popular dichotomization of triage levels for performance and validity measures, this study also explores the disparity when evaluating different severity markers. It is also the first Paediatric Canadian Triage and Acuity Scale (PaedCTAS) validity study carried out in Portugal, providing groundwork to improve triage’s adequacy to the country’s population and Health Service.

The PaedCTAS performs extremely well predicting ICU admission, especially considering second cut-off(i.e.12.345). Regarding the surrogate markers “Admitted to IMCU”, “Admitted to hospital” and “Investigations performed in the PED” the results were good and very similar, particularly when assessed at the third cut-off point(i.e.123.45).

Nevertheless, given the internal PED procedures this can be explained. Patients under investigations and the IMCU are kept in the same physical space, under the same surveillance, the decision for admission in IMCU usually relates to the need for a longer observation period, and not necessarily to severity of illness.

The PaedCTAS guidelines indicate an estimated percentage for need for hospitalization for each triage level: 70% to 90% for level 1, 40% to 70% for level 2, 20% to 40% for level 3, 10% to 20% for level 4 and 0% to 10% for level 5 [36]. However our hospitalization rates were quite inferior for all triage levels: 68.1%, 23.8%, 6.5%, 1.2%, 1.7%., similar to what has been reported by several authors [16, 37, 38] and comparable to a multicentric study from Canada with 550,940 children (61%, 30%, 10%, 2%, and 0.9% for patients in Canadian Triage and Acuity Scale levels 1, 2, 3, 4, and 5, respectively) [16].

As expected, the triage system performed extremely well detecting ICU admission, these results are better than all those reported by Gravel et al. 2013 and Allon et al., and in line with Gravel et al. 2019 considering PaedCTAS. This study’s results, regarding ICU admission were better than those reported by Zachariasse et al. [22] for all triage scales.

It would be expected that the outcome “admission to IMCU” would have a performance marginally better then hospital admission, however that is not the case. This might be due to the clinical reasoning behind IMCU admission i.e. the physicians’ concern for impending clinical deterioration[, therefore IMCU admissions are less severe.

Zachariasse et al. also reviewed adult’s and children’s triage systems regarding hospital admissions n.b.in the paper the evaluation was made considering patients discharged home i.e., not admitted to hospital, therefore the values of specificity and sensibility are swapped. The results from our study are among the highest regarding sensitivity for all triage scales. The variation regarding specificity is very high among studies and our results show a low specificity, nevertheless they are very similar to other PaedCTAS studies, particularly to those with high sensitivity. These results are promising regarding external validity of this study.

Surprisingly, “investigations” i.e., when a patient is asked to stay in the ED, for the physician to better assess the condition’s evolution, performed extremely well as a triage predictor, better than hospital admissions. To the best of our knowledge, this is the first paper measuring this outcome for triage validation, hence there are no studies to compare.

“Resource use” is used in triage validation mainly using Emergency Severity Index (ESI) criteria as reference standard or having costs as an endpoint. A study showed high validity of the CTAS [21], our study had contradictory results. However, the Lee et al. study was done on elderly patients and the concept of resource utilization was CT scan and specialist consultation which contrasts with our definition i.e.if during the visit the patient was medicated or if laboratory or radiologic exams were performed. The usage of “Resource use” as a severity marker for triage validation is problematic for the lack of consensus on its concept, preventing comparisons.

Some severity markers might not be very useful detecting high urgent patients e.g., resource use and admitted to IMCU. However, changing their cut-off, they might be useful for ruling out low urgency patients.