Association for Surgical Education
Quantitative and qualitative analysis of performance during advanced laparoscopic fellowship: a curriculum based on structured assessment and feedback

https://doi.org/10.1016/j.amjsurg.2014.08.029Get rights and content

Abstract

Background

The objective of this study was to examine the acquisition of advanced laparoscopic technical and cognitive skills during a fellowship.

Methods

During a yearlong fellowship, consecutive assessments were completed by a fellow and 1 attending for 3 advanced procedures. The Global Operative Assessment of Laparoscopic Skills, Objective Structured Assessment of Technical Skills, and procedure-specific rating tools and free-text feedback were used. Descriptive statistics, the t test, linear mixed-effects regression, and qualitative analysis of feedback were performed.

Results

Seventy-six cases were included. Average ratings increased for each assessment area every month (P < .001). There were significant differences between ratings by assessors with more stringent ratings by the fellow. While the attending focused on efficiency and safety, the fellow focused on technical issues, with later expanded attention to advanced cognitive aspects.

Conclusions

These assessment tools can be used as a quantitative index to monitor fellows' learning curve. In combination with narrative feedback, such data can provide measures to direct improvement during self-directed learning.

Section snippets

Materials and Methods

During a minimally invasive surgery fellowship at the University of North Carolina at Chapel Hill, a fellow completed daily self-assessments and reviewed video recordings of the operations in which he participated. The attending also completed assessments of the fellow's performance. Three commonly performed advanced laparoscopic procedures in the fellowship were selected: Roux-en-Y gastric bypass (RYGB), paraesophageal hernia repair with Nissen fundoplication (PEH/NF), and Heller myotomy (HM).

Quantitative data

A total of 76 procedures (HM: n = 16; PEH/NF: n = 19; RYGB: n = 41) were included (152 total assessments). The attending's response rate was 92%. The reliability scores of the assessment tools (items, not raters) were high, with estimates above .85 (range: .85 to .91). There were significant differences between ratings by the trainee and attending. Only the GOALS had moderately good inter-rater agreement, .63 and .64 for RYGB and HM, respectively. The other tools had moderately low agreement,

Comment

In this study, we prospectively collected yearlong data on assessment and instruction during a fellowship with a structured curriculum. We used a single-subject design and measured the performance of fellow over an extended period of time after introduction of an intervention (ie, fellowship training).6, 11 In this design, the subject serves as his or her own control. Our assessments were based on existing, extensively studied measures and used early, frequently, and consistently to capture

Conclusions

Our study demonstrates the feasibility of using operative assessment tools for the measurement of surgical performance in a structured curriculum based on a reflective-practice instructional model. These tools can be used as a quantitative index to monitor a fellow's learning curve. In combination with immediate and specific feedback, these tools can provide a valuable source of instruction and guide performance improvement through self-directed assessments. The successful delivery of such a

References (18)

There are more references available in the full text version of this article.

Cited by (10)

  • Toward effective feedback: From concept to practice

    2020, Surgery (United States)
    Citation Excerpt :

    Educators should be flexible and adopt new trends into their curricular design,24 as long as they do not compromise the quality of training. Video-based training can be effective and allows trainees to learn in a low-stress environment.25 This, however, does not replace the reality that surgical trainees need to progress through a comprehensive training program and to achieve competency in performing surgical operations.

  • Targeted surgical coaching can improve operative self-assessment ability: A single-blinded nonrandomized trial

    2020, Surgery (United States)
    Citation Excerpt :

    The identified improvements in accuracy and correlation for self-assessment may be identified as “training to the scale” rather than augmented real-world ability. At worst, familiarity with the OSATS scale allows for deliberate structured reflection, which can be a valuable process.3,35,36 Self-assessment can be compared to the gold standard expert ratings using group-level analysis or direct comparisons of scores.27

  • Faculty feedback versus residents’ self-assessment of operative performance: Different but complementary

    2018, American Journal of Surgery
    Citation Excerpt :

    While we are aware of the limitations of self-assessment,10 we aimed to examine the utility of self-assessment when used in combination with expert feedback. This study builds on our previous investigation of qualitative and quantitative analysis of an advanced laparoscopic fellow's performance in the operating theater11 by comparing surgical residents' self-assessment of performance to the evaluation of surgical skills by experienced faculty. The goal of the study is to identify the cognitive focus of residents at various levels of training (junior and senior residents) and evaluate the differences between their thought process and that of the faculty by analyzing the narrative feedback portion of standard operative performance rating tools.

View all citing articles on Scopus

Timothy M. Farrell receives consultancy fees from NuWave ($500 to $1000 per year for 2 years) and grant support from the Fellowship Council ($35,000 to $50,000 per year). The other authors declare no conflicts of interest.

View full text