Development of a trigger tool to identify harmful incidents, no harm incidents, and near misses in prehospital emergency care

The ATT was developed by a stepwise approach including the [1] review of existing literature regarding patient safety and areas of risk for AEs in the prehospital field [2] expert panel discussions with adaptation of the ATT through a video link, and [3] clinical evaluation of the ATT through RRR (Fig. 1).

The EMS system in Sweden is funded by taxes across regions, resulting in local variations in guidelines and documentation systems. At a national level, it was specified that in each ambulance, one of the two EMS clinicians must be a registered nurse (RN) often with a degree of master in ambulance/intensive/anaesthesiologic care.

We invited the majority of EMS organizations and three services were recruited across Sweden: an urban service with 85,000 EMS missions each year and a median mission time of 70 min and two urban–rural mixed services with 53,000 and 41,000 EMS missions each year and median mission times of 78 and 66 min, respectively. Two of these services participated in the first two sessions of structured discussions and the other one participated in the third.

We used the World Health Organization (WHO) terminology for incidents including NM, NHI, and HI [15, 16] and categorized the incidents according to the National Coordination Council for Medication Error Reporting and Prevention index (NCC MERP) [17].

An NM incident (e.g. lack of documentation) neither affects nor harms the patient but poses the risk of an error and is categorized as (A) or (B) according to the NCC MERP.

An NHI (e.g. omission of electrocardiogram (ECG) in patient with chest discomfort not diagnosed with acute coronary syndrome) affects the patient but does not cause harm. It is categorized as (C) ‘An incident that affected the patient but did not cause any harm’ or (D) ‘An incident that affected the patient and demanded observation or treatment to assure that no harm occurred’.

NMs and NHIs are traditionally not used when using TTs but could be valuable in terms of evaluating the clinical setting by receiving information about commonly occurring NMs and NHIs [18].

An HI (adverse event; e.g. omission of ECG in patient with chest discomfort and later diagnosed with ST-elevated myocardial infarction, thus delaying time to causal treatment) harms the patient. It is categorized as (E) ‘Contributed to or resulted in temporary harm and required intervention’; (F) ‘Contributed to or resulted in temporary harm requiring outpatient care, readmission, or prolonged hospital care’; (G) ‘Contributed to or caused permanent patient harm’; (H) ‘An event that required lifesaving intervention within 60 min’; or (I) ‘Contributed to the patient’s death’.

Existing TTs for inpatients, homecare, paediatric care, and psychiatric care presented by the Swedish Association of Local Authorities and Regions (SKR) were reviewed for developing the new ATT [19]. To cover a prehospital context and to identify risk areas for patient safety, PubMed, Cinahl and Medline was searched with the following keywords: patient safety, prehospital, ambulance, trigger tool, and adverse event. Headlines and abstracts were read and included with the following criteria (1) a description of incidents regarding patient safety in EMS measured with various methods, (2) ≥ 18 years of age, (3) not published before 1980.

Five experts, including RNs and medical doctors (MDs), were invited via email to contribute during the discussions. The RNs were eligible to participate in the development of the ATT if they had 10 years’ EMS clinical experience, previous experience in patient safety in an EMS organization, and experience regarding the usage of a TT. The MDs were recruited because of their knowledge and experience in the methodology of developing TTs. They were also responsible for patient safety in the healthcare system at a regional or national level within their organization. All members of the panel where confident in other methods used in the patient safety context such as incident reporting or root cause analysis. The experts were recruited on the authors opinions who was suitable but also depending on personal interest from the experts.

Three sessions of structured discussions was employed to reach a consensus among a panel of experts regarding the ATT, where the collective opinion of the group is deemed stronger than that of each individual [20]. An expert is known to be a specialist in their field, an individual who has knowledge on a specific matter [21, 22]. During the three sessions, the discussions was structured as follows: each definition and trigger were analysed using a shared screen view for language, clinical relevance, and user-friendliness. The document was seen by all experts, and the removal, adaptation, or combination of triggers was done instantly. Each correction was approved by the experts before moving on to the next trigger. The session was video recorded so that the session would be viewable again and the corrections were emailed to the experts with the ability to give written feedback. Consensus was considered achieved when all experts had approved the ATT via email, and no further feedback was provided by them in writing.

The measurement of patient safety with TTs requires access to patient records. In this light, review teams consisting of an RN and an MD were formed, where the RN starts with a primary review of the patient record with guidance of a list of triggers. A trigger can be a clue that an incident has occurred. Each trigger comes with a definition containing criteria that need to be fulfilled for the trigger to be considered positive.

The RN searches the records for positive triggers, reviews them for a potential incident, and classifies the incident according to the three first steps (AB, C, D) of the NCC MERP [17]. Records with an incident graded C and D undergo a secondary review by the MD, who assesses them for an HI. If an HI is found, the MD classifies it according to the last five steps (E, F, G, H, I) of the NCC MERP. The MD also classifies the type of HI and whether it is preventable by using the following scale: (1) ‘not preventable’; (2) ‘probably not preventable’; (3) ‘probably preventable’, and (4) ‘certainly preventable’.

The three EMS organizations each formed a review team consisting of a MD and a RN. The MD was in medical charge of their organization, and the RN had extensive experience of working in the EMS within this organization. One organization used two RNs having comparable experience in working in the EMS in the primary review.

In a pilot study before session two of the structured discussions, two review teams from two different organizations randomized 150 records and performed an RRR to gain experience in using the ATT.

Before session three of the structured discussions, three review teams from different organizations randomized 900 records and performed an RRR to further evaluate the ATT and to receive frequencies of positive triggers and incidents from their organizations. Before the RRR the review teams received training in how to extract and randomize data, how to use the ATT within Microsoft Access®, definitions of common terminology regarding a TT (e.g. positive trigger, NM, NHI, and HI), and examples with fictitious records. The reviewers had no previous experience in using a TT.

The criteria for inclusion in both RRR was age ≥ 18 years and a primary mission where a patient assessment takes place. Children were excluded in this study because several studies have shown that different triggers are required to study the paediatric population in EMS [23, 24]. There are plans to develop a set of triggers adapted for children.

The positive triggers, NMs, NHIs, and HIs were presented in a frequency-based manner according to both the WHO and the NCC MERP. Each member of the review teams graded the triggers according to clinical relevance, comprehensibility, and utility with a 4-point Likert scale, where 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant after the RRR. The item-level content validity index (I-CVI) was calculated for each trigger by summing the number of reviewers grading trigger 3 or 4 and divided by the total number of reviewers. I-CVI of 0.80 or higher was considered highly relevant [25]. The positive predictive value (PPV) was calculated for each trigger by how many times the trigger resulted in a near miss, no harmful or harmful incident divided by the total times the trigger was found multiplied by 100 [26]. The Mersenne Twister algorithm was used for randomizing the records [27].

Two sessions of independent inter-rater reliability (IRR) tests between two primary reviewers were conducted, with the triggers serving as the variable for testing. The total outcome of positive triggers was summed up and analysed in a confusion matrix with Cohen’s kappa (Table 3) [28]. Kappa values of 0.21–0.40, 0.41–0.60, 0.61–0.80, and 0.81–1.00 were respectively considered fair, moderate, substantial, and almost perfect [29]. The ATT produces an overrepresentation of negative triggers when no incidents have occurred and creates a prevalence problem which causes Cohen’s kappa to be low [28, 30]. Therefore, Cohen’s kappa was complemented with a prevalence-adjusted and bias-adjusted kappa (PABAK) [30]. The interpretation of PABAK and Cohen’s kappa is the same [31]. All analyses were performed with R studio version 2023.03.0 + 386.

The review of previous TTs from SKR and the literature search using PubMed, Cinahl and Medline revealed 32 triggers (refer to supplement 1 and 2). The structure of the ATT was constructed to be the same as that of previous TTs from SKR, and each of the 32 triggers was defined and received definition for the trigger to be positive.

The 32 triggers and definitions were emailed to the five experts two weeks before the planned session. After session one, a total of ten triggers were removed, reducing the number to 22. This was achieved by removing clinically irrelevant triggers or combining several triggers into one. These excluded triggers concerned areas that were either fully covered by other triggers after modifying their definition or were outside the scope of EMS. The majority of the removed triggers were related to EMS organizational quality aspects and were deemed beyond the scope of the EMS trigger review within the EMS record. Examples include triggers related to resource allocation, dispatch actions, or inactions (refer to supplement 2).

To receive feedback from the clinical professionals regarding the 22 triggers, they were evaluated with RRR in clinical practice. Two EMS organizations formed review teams consisting of an RN as a primary reviewer and an MD as a secondary reviewer. The review teams were asked to use the ATT when reviewing randomized records from their organization. The aim of the review teams was to evaluate the clinical relevance, utility, and comprehensibility of the ATT in a clinical setting but also the usability of the digital database. The teams also documented time in minutes for each record reviewed. The ATT presented the following top three positive triggers from the 150 records, B1 Deviations from treatment guidelines in 47 (31%), A1 Incomplete documentation in 33 (22%) and B6 The patient is non conveyed after EMS assessment in 34 (22.7%). The review time was median 5 min per record with a range from 2 to 18 min. These experiences and result showed that the method was feasible, and the digital database was sufficient which served as a foundation for the next session of discussion.

Session two consisted of members of the review teams from the RRR with 150 records. The review teams used their knowledge, experience, and result from the previously performed RRR to further adapt the definitions of the triggers. This round resulted in changes in trigger definitions; however, no trigger was removed or added for example A1 Incomplete documentation received five criteria to be considered positive and B5 Inconsistency between the EMS clinicians and emergency physicians assessment and triage received criteria to be positive if the patient were taken from the ED to definitive care (thrombolysis, percutaneous coronary intervention, intensive care) directly after the physicians assessment. (refer to supplement 2).

The first session of IRR testing between two primary reviewers was carried out on 90 records (Table 1). Cohen’s kappa k = 0.5 and PAPAK k = 0.89.

The final use of the ATT consisted of three EMS organizations, where each organization formed review teams. The RNs categorized the records meeting the inclusion criteria by each month and randomized 25 per month in subject for review; 300 records per organization were used. Table 2 lists the demographics, and Table 3 lists the frequencies of positive triggers with PPV grouped by incidents, and Table 4 lists the classification of incidents according to the NCC MERP and WHO. Nine records were excluded from one mixed organization for not meeting the inclusion criteria. After the RRR, the members of the review teams received an online form consisting of the 4-point Likert scale in order to calculate I-CVI for each trigger presented in Table 5.

In the second session of IRR testing, the same primary reviewers as those in the previous session conducted an RRR of the same 90 records, producing Cohen’s kappa k = 0.3 and PAPAK k = 0.88 (Table 1).

In session three, both the experts and review teams (in total 12) were invited for a video meeting. Because of difficulties in gathering both the panel of experts and review teams, this round was divided into two groups with 10 participants. Two weeks before the session was scheduled, the experts and review team members received the frequencies from the previous RRR, including the I-CVI and IRR results. In this round, the number of triggers was reduced to 18 (refer to supplement 2), and the process of categorizing incidents was altered as follows. The primary examiner decided whether positive triggers in the patient records contributed to an incident, number of incidents, and whether the incident affected the patient with a risk of harm or not. Incidents that did not affect the patient or entail any risk of harm were classified according to the categories AB and C by the primary reviewer. An incident with a risk of the patient being harmed is secondary reviewed by a MD and no classification of the incident is made by the primary reviewer but left to the secondary reviewer. The secondary reviewer assessed incidents with a risk of harm and decided whether the patient was harmed or not. If the patient was not harmed, categories AB, C, and D were used. If the patient was harmed, the type of harm and degree of severity were assessed according to the categories E to I. Refer to supplement 3 for a translated manual of the final ATT.

To our knowledge, this is the first study of EMS in which positive triggers are evaluated by a primary and secondary reviewer according to the NCC MERP [17]. This enables the categorization of the triggers into NMs, NHIs, and HIs according to the WHO, which could serve as a foundation for improvement within a given EMS organization [18].

One limitation in this study could be the number of participants of 12 which could affect the results in not recognizing possible areas of risk for patient safety in the prehospital environment. One limitation in reviewing records from the own organization thus creates a risk for underestimating the occurrence of positive triggers, NHs, NHIs, and HIs. Another limitation could be the failure to include a rural organization with a longer time of transport to participate. None of Sweden’s rural organizations agreed to participate in this study.