Validation of triggers and development of a pediatric trigger tool to identify adverse events

The challenge for healthcare organizations to learn from adverse events (AEs) remains. In order to provide healthcare teams with an adequate picture of potential AEs within their system, different means of identifying and analyzing AEs have been developed. Structured retrospective record review of admissions is an established method for identifying AEs that often go unnoticed by using incident reporting systems [1]-[5]. Published AE incidences in children range from 1% to 62% of the admissions in pediatric care [6]-[12].

One of the most used retrospective record review methods is the Global Trigger Tool (GTT), developed by the Institute for Healthcare Improvement in the USA, which has been used for diverse adult patient populations [13]. Kirkendall et al. [7] used the GTT developed for adult care on a pediatric sample, and found that 25.8% of the patients had at least one AE. This study showed the potential use for the GTT in pediatric populations, but it also pointed out that the review process needs to be developed since definitions and reference values in existing tools are not adapted to pediatric care. Development of such definitions has been done for some clinical contexts, and trigger tools have been developed, for example, for neonatal intensive care, pediatric intensive care, critically ill children, otolaryngology, and for the detecting of adverse drug events (ADEs) [6],[8],[14]-[17], but there is limited knowledge about overall AEs in pediatric care. A challenge to the practical use of the GTT in patient safety management in pediatric care has been the absence of a single tool suitable for examining AEs across pediatric populations and events. A promising comprehensive trigger tool for hospitalized children in Canada, based on the Harvard medical practice study review methodology, found AEs in 15.1% [10] and 9.2% [9] of the admissions. Another pediatric trigger tool was launched in the United Kingdom in 2010 and a recently published study reported that at least one AE occurred in 14.2% of the patients [18]. Both these studies represent a development of trigger tools for detecting AEs in a pediatric population. However, the findings still point to a need for further development of the this methodology [18] and if the aim is to promote learning from AEs, continuous development of triggers, trigger definitions and the framework for categorizing AEs is needed.

All acute care hospitals for adult patients in Sweden perform monthly record reviews and enter the results into a national database according to a financial incentive provided by the government. The need for the development of a national tool for detection of AEs in pediatric populations was identified.

The aim of the present study was to develop a broadly applicable national pediatric trigger tool (PTT) including a manual with a thorough description of triggers, including definitions, reference values, AE preventability decision support, and a description of the review methodology.

The overall PTT project was initiated from a university hospital and the Swedish Pediatric Society with the aim of developing a tool suitable for manual trigger search in pediatric care. A national expert group of 20 persons (Figure 1) was established from different subspecialties within the Swedish Pediatric Society, clinical advisors from other professions, as well as researchers to serve as a reference group.

The PTT development was based on previous studies using record review tools with triggers or screening criteria. To collect information on existing tools, the research group conducted a literature review and developed a study design. PubMed© was searched using the keywords trigger tool, adverse event, harm, medical record, record review, reporting systems, pediatric, children, adult, safety management, inpatient and/or outpatient. Twenty different record review tools for manual use available in English were identified. Nine of the tools were tested in specific pediatric specialties, pediatric units, or types of AEs; two of the latter included AEs from several pediatric specialties. Two trigger tools, not available via PubMed©, was also included. A preliminary set of triggers and screening criteria from different record review methods and studies were added to a list, with duplicated or similar triggers/screening criteria removed. After this, additional recommended triggers later approved by the reference group were added. The first trigger list, including a proposal of definitions and descriptions, was sent by e-mail to the reference group for discussions at the respective pediatric units before a one-day face-to-face meeting. Using a modified Delphi process [19], the reference group was asked to add, change, and remove triggers while also adding relevant pediatric reference values to the triggers in addition to changing the trigger definitions and descriptions. The triggers were discussed and evaluated based on their clinical relevance (i.e., trigger face validity or pointing at high-risk/high-volume AEs) and utility (i.e., usefulness for quality improvement and local ability to act on the results). This process resulted in a pediatric trigger tool (PTT) consisting of 88 triggers (Figure 1) in seven modules (Care, Surgical, Medication, Intensive care, Infant, Perinatal, and Emergency Department modules). As none of the existing record review tools met our requirements, our aim was to make sure that the final PTT was inclusive for AEs in different types of pediatric care. Therefore, the study deliberately included a broad range of well defined triggers based on clinical and research experiences, as well as on the literature review and record review experiences, with the aim of validating these in a patient sample.

The PTT was tested and later evaluated at an urban children’s hospital in Sweden, one of seven divisions at a university hospital located at three different sites and consisting of 19 units. The children’s hospital generally admits patients from 0 to 18 years of age and has eight departments with different specialties such as surgery, orthopedic, oncology, intensive care, neonatology, and medicine, for example. At the time of the study, the children’s hospital had a capacity of 250 beds, had around 2,000 employees, and provided care for approximately 25% of all children in Sweden. Records of children under 19 years receiving all levels of inpatient care discharged during 2010 with at least a 24-hour length of stay (study population n = 12,765) were eligible for randomization. The final cohort consisted of 600 admissions (4.7% of all admissions), and was selected by randomization into four blocks of 150 admissions from neonatal, surgical/orthopedic, medicine, and emergency medicine units, respectively. One review team for each block consisting of one registered nurse (RN) and one physician was formed. The study sample reflected 5,559 hospital days for index admissions, comprised a majority of male patients (n = 314, 52.3%), and acutely admitted patients (n = 474, 79.0%). The mean age was 4.3 years (SD 5.3 years, median 18 months, range 0–18 years), and mean length of stay was 9.3 days (SD 16.6, median 4, range 1–138). The youngest patient was born in gestational week 24 plus one day.

Ethics approval was provided by the regional Ethics Committee of Stockholm (number 2012/2014-31/5). Permission for data collection for patients, triggers, and AEs through the electronic patient record system was granted by the head of the children’s hospital and by each department chairman.

Categorical data are summarized using frequency counts and percent. Continuous variables are presented as mean with standard deviation (SD) and median with range. Both the number of AEs per record and the number of records with at least one AE are presented.

Positive predictive value (PPV) of the respective trigger was counted as the number of times a specific trigger identified an AE divided by the total number of times the trigger was found. A trigger identified several times in a single patient in Review Stage 1 became additional cases; for instance, five transfusions to the same patient resulted in five triggers and were classified as C1-Transfusion. Positive predictive values (PPV) over 100% due to physicians identifying additional triggers to the AE not documented in the RNs’ reviews are presented as 100%.

The statistical programs used to collate the results were QlikView 11 by Qlik Technologies, Inc. PA, US, and Stata 12.0 by StataCorp, TX, US.

No records were excluded in the study due to missing documentation.

In Review Stage 1, after excluding 121 records with only false positive triggers, 296 (49.3%) of the 600 records were forwarded to physician review. The physicians identified 1,066 potential AEs from Review Stage 1. After Review Stage 2, 563 different AEs were identified in 204 (34.0%) patients, range 1–27. Of these, 442 AEs (78.5%) in 161 (26.8%) patients, range 1–22, were deemed to be preventable.

The RNs verbally evaluated that the PTT was easy to use, but reported some difficulties in applying a few trigger definitions such as for pain, for example. The proportion agreement between the RNs concerning: identification of the exact same triggers in the same records were 46.1%, identification of records containing triggers were 65.0%, and records containing at least one potential AE were 75.0%. Of the 15 records where the RNs did not agree that the records should be reviewed by a physician, eight records did not contain an AE according to the physician. The remaining seven records included one minor AE each. There was a 68.9% agreement between RNs’ and physicians’ regarding the existence of at least one AE in the record. Half (50.5%) of the 1066 potential AEs, identified via triggers in Review Stage 1, were confirmed as AEs by the physicians. In their independent review of the records containing at least one potential AE, the physicians found a total 24 AEs not identified by the RNs.

The mean time for the review process was 23.9 minutes (median 15, range 2–250) in Review Stage 1 and 17.8 minutes (median 7, range 1–298) in Review Stage 2.

Triggers were totally identified 3,598 times in 417 (69.5%) records, resulting in a mean of 6.0 (SD 15.6) triggers per patient (median 1, range 0–176). There were 109 records with one trigger, and 220 records had five to nine triggers. Individual triggers varied widely in their yield of detections of AEs after Review Stage 2 (PPV range 0.0-100.0%), and the overall PPV of the triggers was 22.9% (Table 2). The PPV for records containing at least one trigger identified by the RNs and an AE confirmed by the physicians was 48.9% (204/417). The PPV was low in the most frequently identified triggers, such as Transfusion (4.55%) and Failures in cardiovascular, respiratory or neurological function (4.1%). More patient records were found to be trigger-positive in neonatal, surgery/orthopedic, and medicine units vs. the emergency medicine units corresponding to the lower AE incidence in these units. The PPV were: for the neonatal, 24.1%; surgical/orthopedic, 21.0%; medicine, 26.2%; and emergency medicine, 10.2%. A wide variability in the number of trigger outcome and number of AEs associated with the respective trigger was observed within the seven modules. The Intensive care and Surgical modules were the most predictive ones, and the Infant and Perinatal modules the least predictive. Most AEs were identified in the largest modules; Care and Medication.

After analysis and discussions of the triggers, PPVs, and AE outcomes, a third trigger list was created and followed by a second Delphi process round (Figure 1). During the end of the study period, the national adult GTT manual and method had been revised/further developed in parallel. When deciding about which triggers would be included in the final trigger list, the revised adult triggers were taken into account so the two trigger tools were harmonized, as far as possible, with regard to trigger names and the ability of the tool to also include no-harm incidents [26].

After evaluation of the outcome and the review process, the number of pediatric triggers was reduced in order to be convenient for use in a nationally applied clinical record audit manual. Triggers with low PPV, which were inexplicit or had low clinical relevance in pediatric care, were removed. For example, the triggers Abrupt drop in hemoglobin and Transfusion were removed from the trigger list, as corresponding AEs will be detected by triggers such as Cardiac arrest or failures in vital signs or Reoperation in the final trigger list.

Triggers that could be merged were merged into a wider trigger to reduce the number of triggers, as where eleven medication triggers were merged into one implicit trigger, for example. Some of the trigger definitions and descriptions were refined with the aim of achieving a more valid PTT, thus reducing the false positive trigger outcomes as with, for example, a low glucose value in newborns needing to occur more than once to be included as a positive trigger. Three new triggers were added after the analysis of the results: Loss of weight, Extreme hyperbilirubinemia and Severe retinopathy of prematurity. The set of triggers and definitions was edited and refined until consensus was reached (Figure 1). The Emergency Department module was removed, a Laboratory module was added and the Perinatal and Infant modules were merged into one.

The resulting national PTT consists of 29 triggers in six modules (Table 3), and includes a manual with thorough description of the respective trigger, AE preventability decision support, and the review process. A further mapping on trigger level was conducted to assure that the old list of triggers was consistent and covered by the new set of triggers in the final trigger list (Table 2).

To our knowledge, this is the largest record review study testing a manual trigger tool method for inpatients in a single children’s hospital. A total of 563 AEs in hospitalized children of all ages and in several specialties constituted rich material for validation of the triggers and led to the design of a PTT with 29 triggers that will be used as a national tool for manual record review checking for quality and safety management.

There are several methodological considerations when continuing the development of trigger tool methodology that we want to raise.

The strengths in our study are the sample size in one single children’s hospital with several specialties and that the reviews followed the patients both in inpatient and outpatient care. We have also made an effort to transparently describe the development of the triggers and the review process. There are several limitations in the present study. The most known limitations belong to the method itself, requiring documentation quality in the records, otherwise leading to a risk of underestimation of the numbers of AEs, in addition to the lack of a golden standard for AE identification with which we can compare our results. However, several studies have shown that retrospective record review is a superior method of identifying AEs compared to other methods, such as clinical incident reports [1]-[5], for example.

The review was performed by newly trained team members, with the exception of two physicians who had previous experience identifying AEs using record review methodology. This was a given constraint in the study design since the method has not yet been used in pediatric care in Sweden, and pediatric knowledge was an inclusion criterion for the reviewers. It was also a request from the hospital that the study would help develop a group with methodology knowledge that could continue working with record review as one part of the hospital’s patient safety management.

Regardless which retrospective record review is used, the methodology is subjective and can be affected by certain biases such as with the hindsight bias, which may provide an overestimation, especially of the determination of healthcare causation, preventability, and severity of an AE. We tried to standardize the application and interpretation of the PTT by having teams involved in the project phase. This included a detailed study manual with specific instructions, definitions and trigger descriptions, regular meetings during the study period, access to the investigators for questions, and a review expert who monitored the study database and gave the reviewers regular feedback.

In our study design the physicians did not review records negative for potential AEs leading to that Cohen’s kappa between physician and RN review could not be calculated. We therefore only reported percent agreement for both review stages.

Another limitation is that we did not validate the final trigger list on an additional sample of records or reanalyzed the reviewed records. This is because the final list of triggers was an outcome of the study and we chose in the design of the study to include a broad range of triggers thus primarily aiming to validate individual triggers. However, a comprehensive analysis was made to assure that the new trigger list did indeed cover the individual triggers validated in the 600 records. A further validation of the final trigger list will be performed in clinical practice when the PTT is in use in pediatric care nationally.

The study was limited to one children’s hospital; although the hospital provides approximate 25% of all pediatric care in Sweden, this may affect the ability to generalize the outcome.

This study develops the trigger tool methodology for pediatric care in a study conducted in a large children’s hospital setting covering a wide range of care. The main contributions of this study were to further develop and adapt trigger definitions, AE preventability decision support, introduce new triggers in pediatric care, and to apply pediatric triggers in different clinical specialties. Retrospective review methodology is an important source of safety information, and further development and adaptation of the method is important in order to provide clinicians and health care providers with adequate information about the gaps and risks in healthcare systems, as well as allow for further research on the epidemiology of pediatric iatrogenic harm. Our findings resulted in a national PTT which might also be adapted internationally.