Validity and reliability of the South African Triage Scale in prehospital providers

The objective of this study is to assess the inter-rater reliability (i.e., consistency among raters) and validity (i.e., triage accuracy) of the SATS when used by WCG EMS prehospital providers for single-patient triage.

The study was designed as a prospective, assessment-based validation study among WCG EMS providers from March to September, 2017 in Cape Town, South Africa.

At the time of this study, foundational education for WCG EMS providers from across the Western Cape Province included a 6-week certificate courses for BLS (recently discontinued), a 12-week course for ILS (soon to be replaced with a 1-year certificate), and a 2-year (diploma) and 4-year (degree-earning) training for ALS providers [14]. WCG EMS providers responded to over 500,000 calls per year, of which approximately 40% are trauma cases [11, 12]. Providers often staff ambulances as a mixed-tier crew (e.g., BLS with ILS, or BLS with ALS). BLS providers’ scope of practice is limited and best described as advanced first aid (e.g., airway suctioning, splints and wound care) plus cardiopulmonary resuscitation with general access to a relatively narrow selection of medications (e.g., oxygen, oral glucose and oral non-opioid analgesics). ALS providers, however, can deliver a wide variety of drugs and may perform advanced cardiac, trauma, and critical care life support and procedures, including endotracheal intubation and ventilator management. ILS providers’ scope of practice lies between BLS and ALS, and ILS providers can perform several invasive interventions and deliver a narrow selection of intravenous drugs [12].

Initial training on the use of SATS occurred “on the job” by educators within the EMS system – however, all cadres of providers received the same SATS training. WCG EMS providers intermittently may participate in various refresher trainings, short courses, and advancement courses which are offered at the Western Cape Province College of Emergency Care several times per year. Topics at the College cover a variety of clinical and non-clinical content (e.g., leadership training). All providers enrolled in courses at the College during the study period were eligible to participate in this study. One BLS, one ILS, and one ALS cohort each at the College was chosen from which study participants would be enrolled.

From March to September 2017, study investigators recruited a convenience sample of 102 participants from the College. Sample size calculations were based on an intraclass correlation coefficient (ICC; a descriptive statistic used when quantitative measurements are made on units that are organized into groups) of 0.2, power of 0.9, and alpha set of 0.05. None of the participants were actively enrolled in courses at the College that pertained to triage nor use of SATS [15]. Participants from classroom cohorts of BLS, ILS and ALS providers were approached for consent. Staff at the College of Emergency Care delivered advanced advertisement of the study to eligible classes. A study staff member verbally reviewed informed consent with all potential participants, and written consent was obtained from each willing participant.

Participants were briefed on the assessment procedures by a study investigator (JP, TB). Participants were individually administered the written SATS assessment under supervision of one study investigator. Participants were each given an examination booklet and a standard SATS adult color reference table (routinely available in their clinical practice) (online Additional file 1). Each assessment was administered in an examination booklet, comprised of 50 clinical vignettes of adult prehospital emergency cases typical for the Western Cape Province (see sample vignette in Fig. 1). The 50 vignettes were retrieved from a larger set of 100 validated vignettes previously used for hospital staff assessment of SATS [2]. Vignettes were purposefully selected to represent a mixture and balance of case types and SATS distributions that are representative of those seen by WCG EMS. The case context in the original hospital-based vignettes were lightly edited by study investigators (NM, TB, JD) to reflect the prehospital context, but the core clinical scenario and gold standard TEWS, discriminator, and final SATS were unchanged [10‐14]. Twomey and co-investigators from the hospital-based study re-scored all 50 vignettes by re-providing gold standard TEWS, discriminators, and SATS answers for each vignette. There were 19 trauma and 31 medical vignettes. Selected vignettes had the following gold standard SATS color (i.e., final triage color): 5 green, 15 yellow, 23 orange, 6 red and 1 blue. By SATS convention, blue is dead on arrival, red denotes the highest triage acuity, followed by orange, yellow and green corresponds to the lowest acute). The correct TEWS responses in vignettes ranged from 0 to 13. See Additional file 2 for all vignettes.

Each participant provided demographic information (age, sex, qualification, years of experience, and current district). For each clinical vignette, participants provided a TEWS value (between 0 and 17), a clinical discriminator (if applicable), and the final SATS color. Responses were manually entered into a password-protected Microsoft Excel spreadsheet Version 15.0 (Microsoft Corporation, Redmond, Washington, USA) by a study staff member. A second research team member manually verified accurate data entry from a random selection of 10% of assessments.

Reliability was assessed by inter-rater reliability, defined as level of agreement in vignette item (i.e., TEWS, discriminators, and SATS) responses among EMS providers. Validity (i.e., triage accuracy) was assessed using mistriage rates, defined as rates of over- or under-triage when comparing EMS raters to gold standard vignette responses.

Cleaned data were exported from the Microsoft Excel spreadsheet into a statistical software program R, version 3.4.0 (R Core Team, Vienna, Austria). Demographic data were descriptively analyzed. Vignette responses were scored in comparison to the gold standard answers to calculate the proportions of TEWS or SATS that were under, over or accurately determined (e.g., a score of Orange is considered under triage if the gold standard score is Red), and the proportion of discriminators that were missing, correct or incorrect.

A novel statistical measure of agreement, Sklar’s Omega, was used. Sklar’s Omega is a Gaussian copula-based framework that permits estimation of the degree of agreement between the EMS providers’ scores and the gold standard physicians’ scores [16]. Confidence intervals were estimated using bootstrapping methods with 1000 iterations. Traditional interpretations of agreement were used: less than 0.2 represented slight agreement, between 0.2 and 0.4 fair agreement, between 0.4 and 0.6 moderate agreement, between 0.6 and 0.8 substantial agreement, 0.8 or greater was considered near-perfect agreement [17]. The absolute difference between the overall expected agreement with gold standard and inter-rater was averaged to find an overall average difference for each vignette.

Based on our findings, there are several key recommendations. First, there exists an opportunity for focused re-training of prehospital providers on clinical discriminators, with emphasis on trauma and neurologic complaints and several other frequently misused discriminators. Second, EMS providers may also benefit from re-training and reminders to help them consider the patient’s trauma and ambulatory status during calculation of the TEWS; however, this may be qualification dependent. Third, EMS providers may need computational assistance with calculation of the TEWS, and/or a re-formulation of the TEWS table to minimize the likelihood of a computational error, especially in critical (‘red’) cases. Last, the clinical discriminators may need re-formulation to be more compatible with prehospital providers’ clinical knowledge, ‘diagnostic’ capabilities and clinical context.

This study utilized a written assessment rather than simulated cases or chart analysis and therefore may not reflect the true performance of SATS in the ‘live’ clinical environment. However, the artificiality of the testing environment should not affect inter-rater reliability, as the effect of the testing setting will be distributed between the group of subjects and within individual subjects. Additionally, the vignettes tested some, but not all, of the clinical discriminators or combinations of TEWS values, so findings are limited to only those tested. Analysis of results according to level of training ALS, ILS or BLS was not done due to the relatively small sample size.