Implementation study of a 5-component pediatric early warning system (PEWS) in an emergency department in British Columbia, Canada, to inform provincial scale up

The study setting was the ED of Richmond Hospital – an urban public general hospital located in British Columbia, Canada, with an average of 6800 pediatric visits per year; approximately 12% of total ED visits. No PEWS was in place before this study and documentation occurred on a non-pediatric specific emergency nursing assessment record. Richmond Hospital uses the Canadian Triage and Acuity Scale (CTAS) [14] to triage patients by severity and type of presenting symptoms and signs. The scale consists of five levels from 1 (resuscitation) to 5 (non-urgent).

The design of PEWS for inpatients involved both an evidence review and in-depth consultation with numerous stakeholders, including Health Authority leads, quality and safety leads, expert clinicians and frontline care representatives. The result was a 5-component PEWS, consisting of the following: pediatric assessment flowsheet, Brighton PEWS score [15], situational awareness [2], escalation aid, and communication framework [16]. More detail about each component is provided in Table 1. For the pilot study of PEWS in ED, we reviewed the literature related to the ED context, studied the implementation of PEWS in the in patient setting and consulted with ED experts from around the province. The Richmond Hospital team agreed to pilot this inpatient system in the ED, understanding that tailoring to ED would be completed post pilot if the decision was made to move to scale up.

PEWS scoring occurred at triage and then with each vital signs assessment throughout the stay. PEWS scoring was to be completed at a minimum of every 2 h for children with a PEWS score of 0–3 and more frequently for higher PEWS scores [4‐13] (Fig. 1). We hypothesized that this system would enhance providers’ ability to recognize and communicate risk and accelerate mitigating actions.

A team consisting of staff from nursing, medicine, and administration assumed active site leadership roles in implementation and support to health practitioners. Educational resources that had been developed for the provincial inpatient implementation were used for training the nurses in this ED pilot, including instructions for using the flowsheet, e-learning training modules, on-site training workshops, and case studies. Training reinforced the concept that PEWS was a tool to aid clinical judgment, not replace it.

We monitored implementation fidelity through quality audits of medical records at five points during the one-year implementation. These audits were based on selected quality indicators: utilization rates of flowsheets, completion and accuracy of PEWS scoring, documentation of situational awareness factors, and escalation of care. Results were used to provide further education where needed and to promote optimal uptake.

We took a mixed-methods approach, triangulating qualitative and quantitative data, to evaluate this intervention across the three dimensions of fidelity, effectiveness, and utility, following Campbell et al.’s framework for evaluation of complex health interventions [18]. These dimensions and their indicators were selected based on current literature on evaluation of PEWS initiatives, input from provincial stakeholders regarding relevance, and feasibility (Table 2).

We used three types of data collection techniques: medical record review, online survey of providers and semi-structured interviews with key informants to allow us to explore this system from multiple angles and offset the limitations or biases of any one method [19]. The medical record reviews allowed us to explore fidelity and feasibility of implementation, and the documented changes in practice. The qualitative methods allowed us to round out this information by answering questions related to what, why, how the system was working from the perspective of the ED providers [19].

We used descriptive statistics to calculate the overall characteristics of patient medical records and Fisher’s exact test to determine if they were significantly different in any aspect. We used two-samples tests for equality of proportions to determine whether vital signs documentations significantly increased post-implementation compared to baseline. We used Spearman’s correlation to evaluate the relationship between PEWS scores and CTAS scores. Statistical analyses were carried out using Microsoft Excel and RStudio [21].

Response options of the online survey with 5-level Likert scales were collapsed into a measure consisting of 3 levels. Frequencies of responses were calculated for the different categories. In reporting survey responses, the denominators for the calculated percentages represent number of persons who responded to that particular question, not the whole survey.

Qualitative data from open-ended survey questions and interviews with key informants were analyzed using thematic content analysis to identify the key elements of the responses [19]. Responses were reviewed, coded and categorized into commonly occurring themes by two separate members of the research team to enhance trustworthiness of the data. These researchers (TM and Research Associate JM-N) had experience and training with qualitative analysis in health research. The recurrent themes as identified by the researchers underwent member checking to ensure they accurately captured the perceptions of practitioners. This analysis method was chosen to address the primary aim of the qualitative methods: to explore and report on participant’s key perceptions as users of the new system. Saturation was obtained in the themes noted across the survey and interview responses.

In total, 192 medical records were selected for review; 96 at baseline and 96 at final evaluation. The two samples were not significantly different from each other (adopting a statistical significance of 5%), with the only exception being discharge disposition from ED, which can be explained by the opening of a pediatric short stay unit in the ED during the study period (Table 3).

As for provider surveys, the response rate was 38% (n = 37) for registered nurses and 72% (n = 13) for physicians. Half of the practitioners had less than 5 years of clinical work experience. 39% (n = 5) of physicians and 21% (n = 8) of nurses reported having more than 15 years of work experience. Most estimated that 10–25% of their practice was pediatric care.

Three key-informant interviews were conducted.

Medical record reviews show that the intervention was implemented with high fidelity. At triage, 80.2% (n = 77) of the medical records had a PEWS score completed and of those 87% (n = 67) had been accurately calculated. At first bedside assessment, 81.2% (n = 78) of the medical records had a PEWS score present and of those 88.5% (n = 69) were accurate. Medical record review found the most common reason for inaccuracy in PEWS score to be adding the sub-scores instead of taking the highest score for the section, a matter that key informants suggested was improved through practice and more education. At both triage and first bedside assessment, highly urgent or non-urgent patients (CTAS scores 1 and 4) were more likely to be missing a PEWS score than others (p-value = 0.0375 for triage and 0.0354 for bedside assessment, Fisher’s exact test). In 52.1% (n = 25) of patients with an initial PEWS score of 0–3, the time in between assessments was 2 h or less, as per the implementation plan. In 61.4% (n = 59) of the records, a PEWS score was completed at all vital signs assessments throughout the ED visit. In an additional 29.2% (n = 28), PEWS score was completed for over half of the vital signs assessments.

Overall, majority of survey respondents were satisfied or very satisfied with PEWS scoring system (71.8% of nurses, n = 23; 81.8% of physicians, n = 9), PEWS flowsheet (56.2% of nurses, n = 18; 81.8% of physicians, n = 9), escalation guide (68.8% of nurses, n = 22; 81.8% of physicians, n = 9), and reference cards (75% of nurses, n = 24; 70% of physicians, n = 7). Satisfaction was relatively lower for situational awareness tools (41.9% of nurses, n = 13; 36.4% of physicians, n = 4) and the communication framework (54.5% of nurses, n = 17; 45.5% of physicians, n = 5).

Comparison of medical records from before and after implementation showed that adding PEWS scoring to the assessment of pediatric patients significantly increased the rates of documentation of seven parameters embedded in the score at first bedside assessment and throughout the ED stay (Table 4).

Using PEWS in ED further promoted documentation of two situational awareness factors throughout the patient stay: recording of “caregiver concern” increased from 10.4 to 67.7%, and “watcher patient” from 2.1 to 40.6%. “Watcher patient” was noted by the chart reviewer to be particularly important for documenting risks that PEWS score is not designed to capture, including: pain, surgical risk, abnormal lab values, neurologic status, and mental health risk.

We also observed a 51% increase in documentation of notification of most responsible physician and documentation of response by physician.

The majority of practitioners responding to the post-implementation survey reported that the implementation of PEWS in ED changed their knowledge of and confidence in providing pediatric care to a “great” or “very great” extent. The biggest improvement in knowledge was in identification of abnormal clinical signs (58.1%, n = 25), change to a great or a very great extent (Table 5).

Lastly, the survey respondents reported enhanced communication between practitioners with the implementation of PEWS, particularly regarding timing of verbal communication but also frequency and clarity (Table 6). One nurse highlighted: “If the PEWS score is rising or falling, it gives us a reason to contact the MD in clear, concise language”.

Overall, the majority of survey respondents (78.9%, n = 30) agreed that implementation of PEWS in the ED was valuable for pediatric patient care. One nurse highlighted: “It increased our response to children with abnormal vital signs that are compensating or appear relatively well.” The leadership also noted: “PEWS helped validate their clinical decision making and offered a standardized approach to care.”

The PEWS score, assessment flowsheet, and escalation were overall seen as highly useful (average 78.3% indicated useful to moderate extent or higher). While the situational awareness and communication framework were not as favorable, they were still seen as moderately useful or higher by over 65% of the respondents (Table 7).

Some participants perceived little value in using PEWS (5.3%, n = 2). Their reasoning related to the need to further tailor the flowsheets to the ED environment: “PEWS form is good, but frustrating to have to use old form too. It would be great if there was an ED specific PEWS form” (Nurse).

To study alignment of the intervention with current practice, we explored how the PEWS score relates to CTAS score, the well-established triage scoring system. Seventy-seven charts included both a CTAS score and PEWS score at triage. Collapsing PEWS scores 5–13 into a single score (since they would trigger the same escalation guide), we found PEWS scores (0–5) and CTAS scores [1‐4] to be inversely correlated (Spearman’s rho = − 0.574, p-value < 0.001).

The majority of surveyed nurses (89.6%, n = 25) felt that it was “valuable” or “possibly valuable” to complete a PEWS score at triage alongside CTAS (physicians are not involved in triage, therefore were not asked these questions). Practitioners noted that the PEWS score provided a baseline for trending across the visit and gave more objective, thorough results by promoting assessment of all parameters consistently with all patients, which helped to correctly assign a CTAS score. “It (PEWS score) helps me a lot with assessment and priority settings in triage especially for the pediatric population who came in really sick and unable to obtain complete information/data from parents, guardians or significant other” (Nurse).

The impressions of staff regarding the impact of PEWS in ED on pediatric practice are summarized in Table 8. Matters such as earlier identification of risk, more comprehensive assessment, and standardized approaches to communication and mitigation were seen as positive changes. Meanwhile, scores not representing the degree of risk were a reported weakness, particularly at triage e.g. false positive scores when the child is upset or crying, has fever or has been given medication for symptom relief: “Many children find triage to be overwhelming and often are crying, increasing their PEWS score despite looking well” (Nurse).

In this pilot intervention, PEWS scoring was completed at triage alongside CTAS scoring [14]. These two scores have been designed for different purposes and their scoring is done differently: PEWS identifies early signs of deterioration and is scored through assessment of vital signs [4], while CTAS assigns acuity and is scored via assessment and clinical judgement [14]. We were able to show that the scores correlate and that practitioners found them synergistic; we therefore conclude that there is benefit in using both systems with pediatric patients. The PEWS score calculation reinforces a more comprehensive, objective assessment informed by vital signs norms based on age. In our study, implementation of PEWS significantly increased documentation of physiologic parameters; a benefit reported by others [1]. This systematically addresses issues of variance across providers and as highlighted by Roland et al. [7], may reduce cognitive bias, errors or missed assessment components that could otherwise occur because of the pressures of the triage environment [22].

Study participants noted a few limitations when using PEWS at triage. An increase in time at triage was highlighted, an issue which improved with practice and was largely acceptable to practitioners who indicated that the value of scoring outweighed time increase. There were also instances of false positive scores owing to factors such as irritability, which altered vital signs and skewed PEWS scores. In our study, we aimed to counteract this issue through reinforcing the importance of clinical judgement and repetition of assessment when score accuracy was in question. Limited specificity has also been reported elsewhere [9, 10, 12]. As parameters included in a PEWS score assessment are important for determining risk, this substantive increase in thorough assessment and documentation at triage is a very positive change. The limitations of PEWS were acceptable to practitioners who overall found value in using PEWS at triage alongside the CTAS triage tool.

It has been demonstrated that nurses and physicians may fail to recognize deteriorating children due to lack of consistency or accuracy in recording physiological observations [23, 24]. Consistent and comprehensive documentation has been shown to be particularly important in the ED environment where health care providers need to document improvement and appropriateness for discharge [7]. Moreover, with pediatric patients, this documentation is even more critical because providers often rely on general appearance and vital signs as opposed to patients’ descriptions of improvement. This study reviewed documentation of PEWS components across the patient stay. The introduction of PEWS was associated with a substantive increase in documentation across all physiologic parameters. We conclude that the introduction of PEWS in ED can have a positive influence on rates and thoroughness of documented assessment parameters. This not only provides a more consistent “in the moment” picture of physiologic abnormality, but also allows for documentation of trending from the point of presentation.

BC PEWS was designed as a multifaceted safety system as advocated by Lambert et al. [4]. In trying to understand its utility as a system, we asked practitioners to assess the usefulness of each of the components. While all of the components of BC PEWS were ranked useful by the majority of respondents, the PEWS escalation aid and score were perceived as most useful. Practitioners reported that the norm-based scoring enhanced or validated identification of sick children [7]. The escalation aid justified escalating to senior review and encouraged repeated assessment or more thorough monitoring throughout the patient stay.

The systematic documentation and reporting of situational awareness factors were ranked as the least useful part of the system. Although there was a substantive rise in the documentation of “caregiver concern” and “watcher patient”, the medical records reviewer indicated instances where “watcher patient” was missed as an opportunity to elevate a risk profile. For instance, as has been shown in the literature, PEWS scores are more accurate in capturing significant medical illness (particularly respiratory related) than surgical risk [11]. Medical record reviews noted instances such as surgical risk, mental health risk, abnormal labs or neurovitals, where “watcher patient” would have been an appropriate and systematic means of elevating risk profile and guiding appropriate action. Thus, despite lower usefulness rankings, we believe that situational awareness factors highlight risk and create awareness beyond scoring, which has value as part of a safety system in the ED. Moving forward, we believe more tailoring to ED context and further education of staff may improve uptake. Context and staff perceptions play crucial roles in PEWS implementation success as we and others have observed [25, 26]. In addition, as current research focuses on the inpatient setting, further investigation of the impact of situational awareness in the ED on care and patient outcomes will be important.

Enhanced communication is a demonstrated factor in improving quality healthcare [16]. Communication breakdown at handovers are a known point of risk that require consideration when planning an effective system [7]. Because BC PEWS was implemented across the province in inpatient settings, our team chose to implement the same tools to promote a common language, understanding and process across sites and departments, and within the ED. Our findings support that this implementation strategy was effective in enhancing communication as our survey found positive influence on verbal communication and our medical records review found significant gains in documentation. Further, staff reported increased comfort with escalation and communication of risk for senior review, which is an important characteristic of an optimal PEWS tool. As noted above, the staff attributed this gain to having objective tools to validate the need and process for escalation [25]. While we will be taking feedback from this research to revise an assessment flowsheet for ED, we found value in a consistent scoring tool to promote internal and external communication and will retain the Brighton [15] tool in scale up.

As described by Oldroyd and Day, there is a lack of exposure to seriously ill children among ED nurses in general hospitals [8]. Low practice volumes present challenges with maintenance of competency, thus guidance provided by PEWS can assist with identification of serious illness [8]. Our study showed that nurses reported improvements in knowledge and confidence post-PEWS implementation and were completing more comprehensive assessment of vital signs that improved their ability to trend the patient stay. Physician respondents indicated similar benefits in knowledge and confidence and site leaders expressed an overall perception that the healthcare team showed greater awareness of risk or abnormality in the pediatric population. Importantly, improvement in staff knowledge and confidence for providing provision of pediatric care has been noted to be one of the key outcomes of PEWS implementation, which unlike patient outcomes is robustly measurable [27].

Given the decrease in variability of assessment practices using BC PEWS, we conclude that the implementation of PEWS influenced the equalization of patient care between providers. Additionally, having vital signs reference ranges visually evident in PEWS decreased knowledge deficit and cognitive load that may be present for staff who are less familiar or experienced with the pediatric population.

A minority of survey respondents perceived that systems such as PEWS take away from autonomy of experienced nurses. While overall findings of the study support positive associations with the use of such standardized systems, this finding suggests a need to acknowledge and reinforce the role of such systems in supporting clinical judgement rather than replacing or usurping it [26, 28].

Bonafide et al. demonstrated that beyond the marginal performance of PEWS when applied to data sets, clinicians who recently experienced PEWS score failures (false positives) still considered it valuable [29]. In a survey of 254 general ED practitioners in the UK, Griffiths and Kidney found that the majority support the use of early warning systems in the ED, despite the evidence that such scores are low in sensitivity [30]. As with other studies, we found limitations with PEWS in the ED; however these limitations did not appear to alter the overall value assigned to PEWS by staff.

Roland et al. outlined that early warning scores in ED should support earlier diagnosis, more accurate estimation of illness severity and improved communication [7]. Our participants reported that BC PEWS fulfilled all of these functions.

An effective clinical tool is one that practitioners use successfully, and one they want to use [25, 26]. Notably, the hospital team continued to use BC PEWS after the conclusion of the study and have provided ongoing support to the re-design of the system and scale up for the province.

This study demonstrated that BC PEWS is useful and beneficial to pediatric patient care when used with clinical judgment and alongside the currently accepted tools (CTAS) in the ED.

Based on these results and the growing body of evidence in the literature, provincial stakeholders made the unanimous decision to move to provincial implementation of PEWS in all ED settings in British Columbia. This pilot study’s results were used to redesign the 5-component BC PEWS specifically to address the needs of EDs (BC PEWS ED). This involved the creation of a provincial emergency nursing assessment record (short and long form versions) with a compatible PEWS scoring sheet for recording vital signs, and a provincial escalation guide for the ED, which corresponds to PEWS scores. Training materials were developed for ED staff including online training modules, in-person workshops, and a series of quick educational sessions that can be offered as refreshers. Health Authorities from across the province reviewed and accepted the new system to ensure buy-in and regional support. Beginning in February 2018, the province launched a phased implementation of BC PEWS ED, which included the full range of EDs serving children from the generalist rural/remote facilities to the provincial subspecialty pediatric facility. The implementation is being monitored through quarterly audits and planning is underway for researching the scale up of implementation.

The study has a number of limitations. First, there were no accurate measures of deterioration available when identifying the dataset for review so proxy measures of deterioration or severity (i.e. transfer to higher level of care, admission, triage score) were used to select medical records for review. However, we believe that the selected proxy measures captured the population of interest. Second, response rate for the health providers survey was limited to 55%, even after sending out reminders in an attempt to increase the rate. However, our response rate is high compared to a study looking at response rates to online surveys among Canadian physician specialists [31]. Nevertheless, a non-response bias cannot be ruled out. The study did not account for an acclimatization phase in the implementation and did not look at impact on patient outcomes, matters that will be addressed during the research of the provincial scale up.