Education
The Use of Cognitive Task Analysis to Improve Instructional Descriptions of Procedures

https://doi.org/10.1016/j.jss.2011.09.003Get rights and content

Background

Surgical training relies heavily on the ability of expert surgeons to provide complete and accurate descriptions of a complex procedure. However, research from a variety of domains suggests that experts often omit critical information about the judgments, analysis, and decisions they make when solving a difficult problem or performing a complex task. In this study, we compared three methods for capturing surgeons’ descriptions of how to perform the procedure for inserting a femoral artery shunt (unaided free-recall, unaided free-recall with simulation, and cognitive task analysis methods) to determine which method produced more accurate and complete results. Cognitive task analysis was approximately 70% more complete and accurate than free-recall and or free-recall during a simulation of the procedure.

Methods

Ten expert trauma surgeons at a major urban trauma center were interviewed separately and asked to describe how to perform an emergency shunt procedure. Four surgeons provided an unaided free-recall description of the shunt procedure, five surgeons provided an unaided free-recall description of the procedure using visual aids and surgical instruments (simulation), and one (chosen randomly) was interviewed using cognitive task analysis (CTA) methods. An 11th vascular surgeon approved the final CTA protocol.

Results

The CTA interview with only one expert surgeon resulted in significantly greater accuracy and completeness of the descriptions compared with the unaided free-recall interviews with multiple expert surgeons. Surgeons in the unaided group omitted nearly 70% of necessary decision steps. In the free-recall group, heavy use of simulation improved surgeons’ completeness when describing the steps of the procedure.

Conclusion

CTA significantly increases the completeness and accuracy of surgeons’ instructional descriptions of surgical procedures. In addition, simulation during unaided free-recall interviews may improve the completeness of interview data.

Introduction

It is well known that surgeons must master highly complex and rapidly changing knowledge and skills. However, due to new demands in surgical education over the past decade, specifically the regulation of duty hours, the increasing complexity of cases, and the decreased availability of operating room time, it has become increasingly challenging to transmit this knowledge and teach essential skills to surgeons-in-training. It is even more challenging for trauma surgeons who must perform and teach in high stress, time-pressured, and uncontrolled environments. In a trauma setting, there is limited time for instruction during a procedure and, due to the nature of the specialty, there is little time to pre-plan and discuss patients prior to arrival. For these reasons, many programs have moved a significant portion of training out of the clinical arena and into a simulated environment. For this instruction to be successful, educators must develop a sound curriculum that supports the simulated modality and teach the essential action and decisions steps that accompany each task.

Traditionally, the creation of procedural skills curricula has relied on the ability of experts to describe specific procedures. This is problematic because recent research has shown that when experts describe how they perform a difficult task, they may unintentionally omit up to 70% of the critical information novices need to learn to successfully perform the procedure [1]. This has been termed the 70% rule and is a serious problem because it forces novices to “fill in the blanks” using less efficient and error-prone trial-and-error methods. Moreover, as these errors are practiced over time, they become more difficult to “unlearn” and correct [2]. There appear to be two reasons for this problem. First, as physicians gain expertise, their skills become automated and the steps of the procedure blend together. Experts perform tasks largely without conscious knowledge as a result of years of practice and experience [3]. This causes experts to omit specific steps when trying to describe a procedure because this information is no longer accessible to conscious processes 3, 4. Secondly, many physicians are not able to share the complex thought processes that accompany the behavioral execution of technical skills. Even physicians who make an attempt to “think out loud” during a procedure often omit essential information because their knowledge is automated. Consequently, it is difficult to identify the points during a procedure where an expert makes decisions [5].

When asked to describe a procedure, many surgeons will rely on self-recall of the specific skill. Studies from the field of cognitive psychology suggest that the use of standard self-report or interview protocols to extract descriptions of events, decision making, and problem solving strategies can lead to inaccurate or incomplete reports 6, 7. These errors are not often recognized by experts because of the automated and unconscious nature of the knowledge described [8]. Moreover, errors are likely to increase in number and impact under stressful situations [9].

A potential solution to the dilemma described above is the use of cognitive task analysis (CTA) to extract implicit and explicit knowledge from experts to better inform surgical instruction. CTA uses a variety of interview and observation methods to capture the knowledge, goals, strategies, and decisions that underlie observable task performance in complex situations [10]. This interview strategy was first developed by the military to increase the learning curve and acquisition of expertise of complex technical skills and is gaining attention and application in surgical education. The common goal among CTA methods is to assist a subject matter expert in the retrieval and recounting of a procedure that may be highly automated and, therefore, not generally available for conscious inspection.

The purpose of this study was to compare the percentage of knowledge that experts omitted while describing a femoral artery shunt procedure and to validate the 70% rule, which is that experts unintentionally leave out approximately 70% of the information that novices need to perform a task successfully. More specifically, we examined the accuracy of experts’ traditional unaided free-recall of the procedure compared with a gold standard protocol derived from the results of cognitive task analysis. In addition, we investigated whether task simulation (use of equipment and visual aids) increased the accuracy of the traditional free-recall descriptions. We hypothesized that (1) surgeons who give unaided free-recall descriptions of the femoral shunt procedure will omit ±70% of the critical steps compared with the CTA gold standard protocol; (2) the completeness of unaided, self-reported surgical protocol information will vary with the use of simulation during the description of the shunt procedure; (3) the use of CTA methods will increase the accuracy and completeness of surgical protocols by between 12% and 40%.

Section snippets

Subjects

A total of 11 surgeons were recruited for this study. Nine trauma surgeons with experience in placing Argyle type femoral shunts in emergency trauma environments were recruited from the medical school of a large urban research university. Four out of nine of these surgeons were faculty in the Navy Trauma Training Program, and all had battlefield experience as part of a Navy Surgical Team deployed to Iraq. A 10th trauma surgeon, who had experience performing shunt placements as part of an Army

Hypothesis 1

Surgeons who give unaided free-recall descriptions of the femoral shunt procedure will omit ±70% of the critical steps compared with a gold standard CTA protocol. The total percentage of agreement between surgeons’ free-recall description of the shunt procedural steps and the gold standard protocol in the unaided interview condition were 25.00% in Round 1 with a 6.25% improvement in Round 2 for a total of 31.25% agreement (Fig. 2). Thus, surgeons omitted an average of 68.75 % of the standard

Discussion

This study suggests that consideration should be given to adopting CTA methods to increase accuracy, decrease recall errors, and capture the underlying judgments, analyses, decisions, and other cognitive processes that experts use to perform complex medical procedures. By applying CTA methodology, we were able to capture more action and decision steps from surgical experts than the use of unaided free-recall methods alone. This study supports previous research, which shows that most forms of

References (18)

There are more references available in the full text version of this article.

Cited by (52)

  • Cognitive task analysis and workload classification

    2021, MethodsX
    Citation Excerpt :

    Approaches are typically classified by the knowledge elicitation approach, and generally includes 1) Observation and interviews; 2) Process training; and 3) Conceptual techniques [23]. CTA has been used in a variety of domains, including autonomous vehicle display design [24], electronic health record design [25], instructional design [26,27], and design of medical training programs [28]. In this paper, a new approach to CTA is discussed that includes several improvements over existing approaches.

View all citing articles on Scopus
View full text