Grading and assessment of clinical predictive tools for paediatric head injury: a new evidence-based approach

Clinical decision support (CDS) systems proved to enhance evidence-based clinical practice and improve healthcare cost-effectiveness [1‐6]. Based on Shortliffe’s three levels classification, clinical predictive tools, here referred to simply as predictive tools, belong to the highest CDS level; providing patient-specific recommendations based on clinical scenarios, which usually follow clinical rules and algorithms, cost benefit analysis, or clinical pathways [7, 8]. These research-based applications quantify the contributions of relevant patient characteristics to derive the likelihood of diseases, predict their courses and possible outcomes, or support the decision making on their management [9‐11]. Among the healthcare areas that are increasingly utilising predictive tools is the emergency department (ED) [11, 12]. Some of these tools have been demonstrated to support EDs to overcome many of the encountered challenges, such as overcrowding of patients, lack of resources, variable acuity and diversity of clinical conditions [13, 14]. They also have the potential to help clinicians to improve effectiveness through achieving better clinical outcomes, improve efficiency by reducing costs, and improve patient safety by minimising complications and unintended consequences [15‐17].

Traumatic brain injury (TBI) is one of the most commonly presenting emergency conditions and is the leading cause of death and disability among trauma patients [18, 19]. In 2017, the centers for disease control and prevention (CDC) reported that the annual TBI related ED visits were estimated at 2.5 million incidents in the United States (US) [20]. Approximately, one third of these incidents occurred among children aged 0 to 14 years [21]. Many predictive tools have been developed, over the last 25 years, to support the diagnosis of TBI among children and guide the use of computed tomography (CT) in the ED [22, 23]. Through predicting TBI and identifying children who are at low risk of clinically important incidents, these tools are designed to decrease CT scan over-utilisation, to save time and money, and to minimise the exposure of children to the harmful ionising radiation, without compromising their safety or missing clinically significant events [24‐28].

When selecting a predictive tool, for implementation at their clinical practice or for recommendation in clinical practice guidelines, clinicians involved in the decision making are challenged with an overwhelming and ever-growing number of tools. Many of these tools have never been implemented or assessed for comparative effectiveness or post-implementation impact [29‐31]. Currently, clinicians rely on their previous experience, subjective evaluation or recent exposure to predictive tools in making selection decisions. Objective methods and evidence based approaches are rarely used in such decisions [32, 33]. Some clinicians, especially those developing clinical guidelines, search the literature for the best available published evidence. Commonly they look for studies that describe the development, implementation or evaluation of predictive tools. More specifically, some clinicians look for systematic reviews on predictive tools, comparing their development processes or predictive performances. However, there are no available methods to objectively and comprehensively summarise and interpret such evidence [34, 35].

While there are many predictive tools that have been developed, to help clinicians rule out TBI among children at the ED, only a few were considered for use in clinical practice [22‐24]. Therefore, we need to understand what makes certain tools more widely accepted and successfully implemented than the others. This will help national and institutional guideline developer clinicians to make better decisions in selecting and incorporating effective predictive tools in their clinical guidelines to help other clinicians through the decision-making process. Furthermore, this will also help expert clinicians develop better predictive tools for the clinical practice in the future. In addition to the predictive performance measures, such as the sensitivities and specificities of predictive tools, many other quantitative and qualitative measures can be considered for the analysis. The country and year of tools’ development could have an influence on the tools’ acceptance and success. In addition, the number of citations and studies that report the tools’ validation, evaluation or implementation could indicate some sort of attention and acceptance. Furthermore, the quality of the tools’ development studies, and the efforts invested in their development, reflected in the sample size of patients or records used in the development and the number of authors and their experiences, could support tools’ wide acceptance and successful implementation.

The primary objective of this study is to grade and assess paediatric head injury predictive tools using a new evidence-based framework for grading and assessment of predictive tools (The GRASP Framework). The secondary objective is to provide emergency clinicians with standardised objective information on clinical predictive tools to support their search for and selection of effective tools.

Our study is composed of three parts. The first includes identifying paediatric head injury predictive tools, proposed in the literature, and their related published evidence. The second part includes grading these predictive tools using our new evidence-based approach and eligible published evidence. The third part includes conducting a comprehensive and objective analysis to answer the research question.

This study presents a new evidence-based approach to grade and assess predictive tools. Based on the critical appraisal of the published evidence on predictive tools, the GRASP framework uses three dimensions to grade the tools: 1) phase of evaluation; before implementation, during planning for implementation and after implementation, 2) level of evidence; adding a numerical score within each phase, and 3) direction of evidence; positive, negative or mixed. The final grade is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports a positive conclusion. Among the 14 paediatric head injury predictive tools, the PECARN rule stands out clearly, since it is the only tool evaluated in post-implementation impact studies, which needs some explanation.

The 14 predictive tools targeted variable paediatric age groups. Most of the tools focused on children less than 16 years of age. However, some tools extended their cover to less than 21 years, such as Atabaki, while others limited their population to children less than 2 or 3 years, such as Buchanich and Greenes. The tools used different development methodologies and their prediction models used different mix of clinical variables. Furthermore, the predictive performances of the tools, such as their sensitivities and specificities, were different. However, the predictive performances of the tools were not correlated with their assigned grades. This indicates that the technical specifications of the predictive tools did not, in the first place, influence their successful validation, acceptance, or implementation. The country and year of tools’ development were also non-significantly influential on their successful path from validation into implementation. On the other hand, the number of citations of the studies, describing the development of the tools, and the number of studies reporting them are clearly correlated with tools’ success. These two indicators are secondary to the main quality indicators of the tools’ development studies, such as the sample size of patients used in the development of the tools and the number of researchers developing these tools.

In addition, the experiences of the researchers have an important role in leading better-quality studies. Three of the researchers who developed the PECARN rule have already contributed to older but less successful tools. Before leading the team to develop the PECARN rule in 2009, Dr. Kuppermann contributed to developing the Quayle rule in 1997 and the Palchak rule in 2003. Dr. Quayle and Dr. Atabaki each developed her own rule in 1997 and 2008, before joining the team in developing the PECARN rule in 2009. The affiliations of the researchers, to highly ranked institutes, and the support of the studies by dedicated and well-funded research networks, programs, and professional groups, added to the credibility of the tools among clinicians and organisations. As a result of the better quality and higher credibility, the PECARN rule development study was published in a top ranked journal with a high impact factor; the Lancet. In addition, the three main tools; the PECARN, the CHALICE and the CATCH rules were endorsed by professional organisations and recommended in clinical practice guidelines, such as the paediatric head trauma clinical guidelines developed by the Royal Australian and New Zealand College of Radiologists [90].

Many studies compared paediatric head injury predictive tools. Among these, nine compared the three main tools; the PECARN, the CHALICE and the CATCH rules. Despite the fact that most of the studies reported PECARN as the highest quality tool, they reported that all three predictive tools had excellent sensitivities and performed well in assessing the outcome of clinically important TBI, suggesting that all were appropriate for use in assessing mild head injury in the ED [58, 91]. However, each tool is applicable to a different proportion of children with head injury. This makes the direct comparison of the three tools difficult [72]. The CHALICE rule applies to a broad population of head injuries of any severity, the PECARN rule was developed for minor head injuries only and the CATCH rule focused on a group of patients with specific signs or symptoms [59]. The PECARN rule is the most validated [37], and has the best sensitivity while the CHALICE rule has the best specificity [66, 91, 92]. Compared to senior, experienced, and high accuracy emergency physicians, the implementation of PECARN, CATCH or CHALICE rules have a potential to increase the CT rates with limited potential to increase the accuracy of detecting clinically important TBI [93]. In addition, the three tools were not more cost-effective than usual care in some ED settings [94]. Despite that CT is the imaging modality of choice in the ED, because of availability and speed, however, magnetic resonance imaging is recently becoming the preferred modality in children. This would change predictive tools’ comparability and priority for recommendation, where further research is required [92].

Some predictive tools, in other clinical areas, gained their widespread acceptance and successful implementation by providing simplicity and feasibility. The Ottawa ankle and the Ottawa knee rules are good examples of simple paper based five items check lists, designed to exclude the need for an X-ray for possible bone fracture in adult patients at the ED [95, 96]. The resources needed to implement such tools are minimal; no technical requirements, special training or financial support are needed. Both tools were implemented, within 2 years of their development, and demonstrated positive post-implementation impact on the efficiency of ED healthcare services through wide scale high quality experimental studies [97‐100].

Accordingly, selecting effective predictive tools remains a major challenge for most clinicians who usually lack the time and experience required to evaluate such tools; assessing their quality or grading their level of evidence, especially as their number and complexity have increased tremendously over the recent years. This is made worse by the complex nature of the evaluation process itself and the variability in the quality of published evidence. Furthermore, it is not always feasible to compare tools, even those designed for the same predictive tasks, due to the diversity of their development methodologies, clinical variables, target populations, conditioned applications, and predictive performances. Therefore, we chose not to look at the details of every single validation or implementation study. Alternatively, the GRASP framework provides users with a higher level and evidence-based approach to grade predictive tools through the critical appraisal of published evidence on their development and validation before implementation, usability and potential effect during planning for implementation, and post-implementation impact on clinical effectiveness, patient safety and healthcare efficiency. Based on the available evidence, the framework identifies tools that are more trusted by clinicians and researchers and consequently can be more successful. Using the GRASP framework might need some training for expert healthcare professionals and researchers, who are going to grade predictive tools and some awareness for end user clinicians who are going to use GRASP output to select predictive tools.

The main limitations of this study include the possibility of missing some predictive tools which could have been developed by clinicians but not yet published, because the GRASP framework depends on grading predictive tools based on their published evidence. Similarly, some of the published predictive tools could have been implemented in clinical practice but no studies, reporting their implementation or evaluating their post-implementation impact, have been published yet. Furthermore, while this study is in press or soon after it is published, an evidence on some tools may become available and could have an influence on the assigned grade.

Comparing the three main tools, which were assigned the highest GRASP grades PECARN, CHALICE and CATCH, to the remaining 11, we find that three main factors are highly crucial and indicate better tools. Firstly, the quality of the predictive tools, which is indicated by the development methodology of the tools, the patient sample size used for development, and the number of contributing authors. The quality is also reflected through the number of citations and number of studies discussing each tool. Secondly, the experience and credibility of the tools’ authors, reflected in their clinical specialty and affiliated organisations. Thirdly, the support by dedicated and well-funded research programs. These three factors were more significantly influential, than the sole high predictive performance, on the wide acceptance and successful implementation of the tools. In addition, tools’ simplicity and feasibility, in terms of resources needed, financial support, technical requirements, complexity and number of predictors, and training, are crucial factors of their success. It is important to select tools which best fit the intended tasks, the clinical conditions, the healthcare settings and the patient populations. Based on detailed specifications, a group of best predictive tools can be recommended for use in clinical practice. Through evidence-based grading of predictive tools, the GRASP framework confirmed the PECARN rule as the highest quality tool, compared to the other tools, which have variable levels of supporting evidence. The online availability of the GRASP framework will enable clinicians and clinical guideline developers to access detailed information, reported evidence and assigned grades of predictive tools. However, keeping such information up-to-date requires continuous updating of tools’ reports when new evidence becomes available.