EducationThe Use of Cognitive Task Analysis to Improve Instructional Descriptions of Procedures
Introduction
It is well known that surgeons must master highly complex and rapidly changing knowledge and skills. However, due to new demands in surgical education over the past decade, specifically the regulation of duty hours, the increasing complexity of cases, and the decreased availability of operating room time, it has become increasingly challenging to transmit this knowledge and teach essential skills to surgeons-in-training. It is even more challenging for trauma surgeons who must perform and teach in high stress, time-pressured, and uncontrolled environments. In a trauma setting, there is limited time for instruction during a procedure and, due to the nature of the specialty, there is little time to pre-plan and discuss patients prior to arrival. For these reasons, many programs have moved a significant portion of training out of the clinical arena and into a simulated environment. For this instruction to be successful, educators must develop a sound curriculum that supports the simulated modality and teach the essential action and decisions steps that accompany each task.
Traditionally, the creation of procedural skills curricula has relied on the ability of experts to describe specific procedures. This is problematic because recent research has shown that when experts describe how they perform a difficult task, they may unintentionally omit up to 70% of the critical information novices need to learn to successfully perform the procedure [1]. This has been termed the 70% rule and is a serious problem because it forces novices to “fill in the blanks” using less efficient and error-prone trial-and-error methods. Moreover, as these errors are practiced over time, they become more difficult to “unlearn” and correct [2]. There appear to be two reasons for this problem. First, as physicians gain expertise, their skills become automated and the steps of the procedure blend together. Experts perform tasks largely without conscious knowledge as a result of years of practice and experience [3]. This causes experts to omit specific steps when trying to describe a procedure because this information is no longer accessible to conscious processes 3, 4. Secondly, many physicians are not able to share the complex thought processes that accompany the behavioral execution of technical skills. Even physicians who make an attempt to “think out loud” during a procedure often omit essential information because their knowledge is automated. Consequently, it is difficult to identify the points during a procedure where an expert makes decisions [5].
When asked to describe a procedure, many surgeons will rely on self-recall of the specific skill. Studies from the field of cognitive psychology suggest that the use of standard self-report or interview protocols to extract descriptions of events, decision making, and problem solving strategies can lead to inaccurate or incomplete reports 6, 7. These errors are not often recognized by experts because of the automated and unconscious nature of the knowledge described [8]. Moreover, errors are likely to increase in number and impact under stressful situations [9].
A potential solution to the dilemma described above is the use of cognitive task analysis (CTA) to extract implicit and explicit knowledge from experts to better inform surgical instruction. CTA uses a variety of interview and observation methods to capture the knowledge, goals, strategies, and decisions that underlie observable task performance in complex situations [10]. This interview strategy was first developed by the military to increase the learning curve and acquisition of expertise of complex technical skills and is gaining attention and application in surgical education. The common goal among CTA methods is to assist a subject matter expert in the retrieval and recounting of a procedure that may be highly automated and, therefore, not generally available for conscious inspection.
The purpose of this study was to compare the percentage of knowledge that experts omitted while describing a femoral artery shunt procedure and to validate the 70% rule, which is that experts unintentionally leave out approximately 70% of the information that novices need to perform a task successfully. More specifically, we examined the accuracy of experts’ traditional unaided free-recall of the procedure compared with a gold standard protocol derived from the results of cognitive task analysis. In addition, we investigated whether task simulation (use of equipment and visual aids) increased the accuracy of the traditional free-recall descriptions. We hypothesized that (1) surgeons who give unaided free-recall descriptions of the femoral shunt procedure will omit ±70% of the critical steps compared with the CTA gold standard protocol; (2) the completeness of unaided, self-reported surgical protocol information will vary with the use of simulation during the description of the shunt procedure; (3) the use of CTA methods will increase the accuracy and completeness of surgical protocols by between 12% and 40%.
Section snippets
Subjects
A total of 11 surgeons were recruited for this study. Nine trauma surgeons with experience in placing Argyle type femoral shunts in emergency trauma environments were recruited from the medical school of a large urban research university. Four out of nine of these surgeons were faculty in the Navy Trauma Training Program, and all had battlefield experience as part of a Navy Surgical Team deployed to Iraq. A 10th trauma surgeon, who had experience performing shunt placements as part of an Army
Hypothesis 1
Surgeons who give unaided free-recall descriptions of the femoral shunt procedure will omit ±70% of the critical steps compared with a gold standard CTA protocol. The total percentage of agreement between surgeons’ free-recall description of the shunt procedural steps and the gold standard protocol in the unaided interview condition were 25.00% in Round 1 with a 6.25% improvement in Round 2 for a total of 31.25% agreement (Fig. 2). Thus, surgeons omitted an average of 68.75 % of the standard
Discussion
This study suggests that consideration should be given to adopting CTA methods to increase accuracy, decrease recall errors, and capture the underlying judgments, analyses, decisions, and other cognitive processes that experts use to perform complex medical procedures. By applying CTA methodology, we were able to capture more action and decision steps from surgical experts than the use of unaided free-recall methods alone. This study supports previous research, which shows that most forms of
References (18)
- et al.
Cognitive task analysis
Int J Educ Res
(1996) - et al.
Expert error in troubleshooting: An exploratory study in electronics
International Journal of Human-Computer Interaction
(1999) - et al.
Automaticity of action
- et al.
Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory
Am J Surg
(2004) - et al.
The effectiveness of a cognitive task analysis informed curriculum to increase self-efficacyand improve performance for an open cricothyrotomy
J Surg Educ
(2011) - et al.
The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement
Am J Surg
(2007) - et al.
The use of a cognitive task analysis-based multimedia program to teach surgical decision making in flexor tendon repair
Am J Surg
(2008) - et al.
Instructional implications of cognitive task analysis as a method for improving the accuracy of experts’ self-report
Resistance to change: Unconscious knowledge and the challenge of unlearning
Cited by (52)
How Surgeons Think to Avoid Error: A Case Study of the Neurovascular Bundle Sparing During a Robotic Prostatectomy
2024, Journal of Surgical EducationHow I Do It: Structured Narration for Cognitive Simulation-based Training in Robotic Surgery
2023, Journal of Surgical EducationAssessing antibiotic prescribing in nurse practitioners: Applied cognitive task analysis
2022, International Journal of Nursing Studies AdvancesCognitive task analysis and workload classification
2021, MethodsXCitation Excerpt :Approaches are typically classified by the knowledge elicitation approach, and generally includes 1) Observation and interviews; 2) Process training; and 3) Conceptual techniques [23]. CTA has been used in a variety of domains, including autonomous vehicle display design [24], electronic health record design [25], instructional design [26,27], and design of medical training programs [28]. In this paper, a new approach to CTA is discussed that includes several improvements over existing approaches.
Use of sensors to quantify procedural idle time: Validity evidence for a new mastery metric
2020, Surgery (United States)