Written evaluation is not a predictor for skills performance in an Advanced Cardiovascular Life Support course

Abstract

Objective

Both a written cognitive knowledge evaluation and a practical evaluation that tests psychomotor skills, cognitive knowledge, and affective behaviors such as leadership and team skills are required for successful completion of American Heart Association (AHA) Advanced Cardiovascular Life Support (ACLS) course. The 2005 International Liaison Committee on Resuscitation (ILCOR) Consensus on Science and Treatment Recommendations noted that in Basic Life Support (BLS) there is little to no correlation between written and practical skills. The current study was conducted to determine if there is a correlation between written and practical evaluations in an ACLS course.

Methods

34 senior nursing students from four nursing programs participated in two separate ACLS classes, completing both the written and practical evaluations. Immediately following the courses, all participants served as team leader for a video recorded simulated cardiac arrest event. A panel of expert ACLS instructors who did not participate as instructors in the courses reviewed each video and independently scored team leaders’ performances.

Results

Spearman's rho correlation coefficient between the written test scores and practical skills performance was 0.194 (2-tailed significance = 0.272).

Conclusion

The ACLS written evaluation was not a predictor of participant skills in managing a simulated cardiac arrest event immediately following an ACLS course. The single case simulations used in ACLS skills evaluation test a narrow portion of ACLS content while written evaluation tests can more practically test a broader spectrum of content. Both work in concert to define participant knowledge and neither should be used exclusively to determine participant competence.

Introduction

The American Heart Association (AHA) Advanced Cardiovascular Life Support (ACLS) course was created in 1973 and introduced as an AHA sponsored program in 1974.1, 2 The ACLS course has served as the model for other short-course format resuscitation courses.³ Since its inception, passing both a written evaluation and a practical skills assessment were required for successful completion of the ACLS course. The validity of both the written and practical testing mechanisms currently in use have been demonstrated in previous studies.4, 5 There is however no published research that directly addresses the question of whether student performance on written tests in advanced life support courses correlates to objectively measured performance of resuscitation as a whole or its individual required subcomponents.

The current study proposed the following research question: is a written evaluation in an Advanced Cardiovascular Life Support course a predictor of students’ skills performance in a simulated cardiac arrest?

A single study of paediatric residents suggests that the correlation of written tests and objective performance measures may be poor as the mean score for the standardized Pediatric Advanced Life Support (PALS) written examination was 93.2% (±5.5%), but none of the 28 third-year paediatric residents were able to successfully perform both basic and advanced airway maneuvers.⁶ Only 11% of these same trainees were able to adequately perform both vascular access procedures (intraosseous and femoral central line placement using the Seldinger technique). Unfortunately, correlation data was not published between written and performance based tests as it was not the focus on the study.

An earlier study involving 45 paediatric residents described the residents as performing well on a cognitive written test with 80% also being able to achieve the end-point of four predefined resuscitation skills (bag-valve-mask ventilation, intubation, defibrillation, and intraosseous line placement).⁷ Again, this was not designed as a correlational study and although learners achieved the end-point of each skill, they performed quite poorly at many of the essential subtasks. For instance only 27% of residents verified suction was functioning and only 34% verified the bag-valve-mask was available and functioning prior to intubation. The equivocal findings and lack of true correlations make it difficult to make any generalizations from this paper relevant to the research question we have undertaken.

A systematic review of the literature addressing the correlation between written tests and performance based measures in basic life support (BLS) courses concluded that although the majority of papers (14 out of 17) did suggest a correlation, much of the data for healthcare professionals was inconsistent and contradictory requiring further research.⁸ The question of whether a written BLS test reflects BLS skills competence was felt to be “indeterminate.” Extrapolating the systematic review to advanced life support courses is not appropriate as the knowledge and skills to be performed in advanced life support are different than BLS and the relative performances of students in the written and performance measures are likely to be different.

The literature from the broader medical education community, although not conclusive, does suggest a positive correlation between written tests and more objective measures of performance such as Observed Structured Clinical Encounters (OSCEs).9, 10, 11, 12, 13, 14 The correlations are however quite variable and there is occasional published literature that contradict the above positive correlations.15, 16 Of particular interest from the above studies was the study by Ram et al.¹⁰ suggesting that the written test could actually correlate as well with performance in the clinical environment as an OSCE.

The lack of evidence directly related to the research question and inconclusive data in closely related fields emphasizes the need for well conducted trials to help guide the development of learner assessments within advanced resuscitation courses. Miller's classic pyramid outlines four successive levels of clinical assessment—Knows, Knows How, Shows, and Does.¹⁷ ‘Ideal’ evaluation of a learning should theoretically focus on learners’ actions in the clinical environment (i.e. what one ‘does’) but is generally not feasible for most short courses. Performance based tests, such as the resuscitation case scenario, assess the learner's ability to ‘show’ what they are capable of while written tests may be limited to lower levels assessing what the learner ‘knows’ (knowledge) or what they ‘know how’ to do (comprehension). Knowledge of a correlation between written and performance based tests (or lack thereof) is valuable, if not imperative, to the development of a well designed and practical assessment strategy for advanced resuscitation courses.