The Knee Arthroscopy Learning Curve: Quantitative Assessment of Surgical Skills

doi:10.1016/j.arthro.2014.02.021

Arthroscopy: The Journal of Arthroscopic & Related Surgery

Volume 30, Issue 5, May 2014, Pages 613-621

https://doi.org/10.1016/j.arthro.2014.02.021 Get rights and content

Purpose

To assess orthopaedic trainees performing diagnostic knee arthroscopies and evaluate procedural competence using a novel statistical method, the Cumulative Summation Test for Learning Curve (LC-CUSUM).

Methods

Twenty orthopaedic trainees in postgraduate year (PGY) 1 through 5 performed diagnostic knee arthroscopy and were evaluated intraoperatively with a validated 10-point knee task-specific checklist (TSCL) and 50-point global rating scale (GRS). A score of 40 points or greater (of 50 points) for the GRS and 8 points or greater (of 10 points) for the knee TSCL was considered to indicate a successful procedure. For the LC-CUSUM analysis, adequate performance was defined as a 10% failure rate, inadequate performance was defined as a 30% failure rate, and an acceptable deviation from adequate performance was defined as 10%. A limit h equal to 1.6 was selected to give a true-discovery rate of 90% and a false-discovery rate of 10% over 50 procedures.

Results

A total of 340 consecutive procedures were performed by 20 trainees during the 12-month study period. The cumulative number of arthroscopic procedures performed by trainees before study start increased with increasing PGY. The median number of arthroscopic procedures performed per trainee was 16.5 (interquartile range, 14 to 21.75). Competency in knee arthroscopy for the TSCL was achieved by 8 trainees (40%), after a median of 16 procedures (interquartile range, 13 to 20), and for the GRS by only a single trainee (5%), after 14 procedures. Threshold-adjusted curves stratified by PGY level enabled multiple trainees to achieve competency for both the TSCL and GRS.

Conclusions

The LC-CUSUM can be successfully applied to knee arthroscopy to provide an individualized assessment of performance and quantitatively demonstrate competency for basic arthroscopic tasks.

Clinical Relevance

The LC-CUSUM is an effective method to evaluate procedure competence in arthroscopic training and can provide objective feedback and benchmarks in the learning phase.

Section snippets

Subjects

Research ethics board approval for the study was obtained from 3 university-affiliated hospitals. The subjects comprised 20 orthopaedic trainees in postgraduate year (PGY) 1 through 5 performing diagnostic knee arthroscopy and partial meniscectomy during 3-month orthopaedic sports medicine rotations in the academic year of July 1, 2011, to June 30, 2012. As expected, the cumulative number of arthroscopic procedures performed before study start increased with increasing PGY (Table 1). Each

Results

A total of 340 arthroscopic procedures performed by the 20 orthopaedic trainees were evaluated during the 12-month study period. The number of arthroscopic procedures was asymmetrically distributed among the trainees, with a median of 16.5 (interquartile range, 14 to 21.75). The LC-CUSUM scores were calculated, and competencies for the TSCL and GRS for each trainee were determined (Table 2). Competency for the TSCL was achieved by 8 trainees (40%), after a median of 16 procedures (interquartile

Discussion

Although procedural competence was demonstrated by a number of trainees, overall competency, especially for the GRS instrument, was poorer than predicted. The capacity to demonstrate competency is influenced by prior arthroscopic experience, and considerable variation existed among individual trainees.

Most of the technical learning curve data in arthroscopic training have been limited to the demonstration of construct validity of simulated environments or the introduction of new intraoperative

Conclusions

The LC-CUSUM can be successfully applied to knee arthroscopy to provide an individualized assessment of performance and quantitatively demonstrate competency for basic arthroscopic tasks.

References (35)

J.L. Irani et al.
Surgical residents' perceptions of the effects of the ACGME duty hour requirements 1 year after implementation
Surgery
(2005)
P.J. O'Neill et al.
Arthroscopic proficiency: A survey of orthopaedic sports medicine fellowship directors and orthopaedic surgery department chairs
Arthroscopy
(2002)
M.J. Elliott et al.
Diagnostic knee arthroscopy: A pilot study to evaluate surgical skills
Arthroscopy
(2012)
R.R. Sonnadara et al.
Orthopedic boot camp: Examining the effectiveness of an intensive surgical skills course
Surgery
(2011)
J.A. Slade Shantz et al.
Validation of a global assessment of arthroscopic skills in a cadaveric knee model
Arthroscopy
(2013)
S. Srivastava et al.
Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity
J Shoulder Elbow Surg
(2004)
A. Alvand et al.
Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair
Arthroscopy
(2013)
L.R. Farnworth et al.
A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: The financial impact of orthopaedic surgical training in the operating room
Iowa Orthop J
(2001)
J.D. Mabrey et al.
Virtual reality simulation of arthroscopy of the knee
Arthroscopy
(2002)
R.A. Pedowitz et al.
Evaluation of a virtual reality simulator for arthroscopy skills development
Arthroscopy
(2002)

J.D. Zuckerman et al.

The early effects of code 405 work rules on attitudes of orthopaedic residents and attending surgeons

J Bone Joint Surg Am

(2005)

P.D. van Hove et al.

Objective assessment of technical surgical skills

Br J Surg

(2010)

A. Darzi et al.

Assessing operative skill. Needs to become more objective

BMJ

(1999)

A. Insel et al.

The development of an objective model to assess arthroscopic performance

J Bone Joint Surg Am

(2009)

L. Dessolle et al.

How soon can I be proficient in embryo transfer? Lessons from the cumulative summation test for learning curve (LC-CUSUM)

Hum Reprod

(2010)

P.S. Jowell et al.

Quantitative assessment of procedural competence. A prospective study of training in endoscopic retrograde cholangiopancreatography

Ann Intern Med

(1996)

P. Ramesha et al.

Use of the Learning Curve–Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: Fetoscopic laser ablation as a model

Am J Obstet Gynecol

(2011)

Cited by (40)

An Overview of Pediatric Knee Arthroscopy With Use of the NeedleScope
2024, Arthroscopy Techniques
As sports-related injuries are becoming more common among children, there has been an increased need for knee arthroscopies in the pediatric population. Nevertheless, pediatric knee arthroscopy exposure is varied among orthopaedic surgeons, especially during residency training. There is a considerable difference in knee arthroscopy case volume between the adult and the pediatric population among orthopaedic residents. Although the fundamentals of adult knee arthroscopy have been well delineated, there is a paucity of literature surrounding basic pediatric knee arthroscopy. Our goal is to further explain the fundamentals needed to perform successful arthroscopic surgery in the pediatric knee. Through discussing proper patient positioning, instrumentation, including the utility of the 1.9-mm small-bore needle arthroscope (NanoScope; Arthrex, Naples, FL), subtle tips and tricks, as well as advantages and disadvantages of arthroscopy in children, this Technical Note along with the corresponding video are designed to help elucidate the intricacies of the pediatric knee. It will serve as educational material targeted to the resident trainee or any orthopaedic practitioner interested in gaining an introductory foundation to pediatric knee arthroscopy.
Can artificial intelligence help decision-making in arthroscopy? Part 2: The IA-RTRHO model – a decision-making aid for long head of the biceps diagnoses in small rotator cuff tears
2023, Orthopaedics and Traumatology: Surgery and Research
The possible applications of artificial intelligence (AI) in orthopedic surgery are promising. Deep learning can be utilized in arthroscopic surgery due to the video signal used by computer vision. The intraoperative management of the long head of biceps (LHB) tendon is the subject of a long-standing controversy. The main objective of this study was to model a diagnostic AI capable of determining the healthy or pathological state of the LHB on arthroscopic images. The secondary objective was to create a second diagnostic AI model based on arthroscopic images and the medical, clinical and imaging data of each patient, to determine the healthy or pathological state of the LHB.
The hypothesis of this study was that it was possible to construct an AI model from operative arthroscopic images to aid in the diagnosis of the healthy or pathological state of the LHB, and its analysis would be superior to a human analysis.
Prospective clinical and imaging data from 199 patients were collected and associated with images from a validated protocoled arthroscopic video analysis, called “ground truth”, made by the operating surgeon. A model based on a convolutional neural network (CNN) modeled via transfer learning on the Inception V3 model was built for the analysis of arthroscopic images. This model was then coupled to MultiLayer Perceptron (MLP), integrating clinical and imaging data. Each model was trained and tested using supervised learning.
The accuracy of the CNN in diagnosing the healthy or pathological state of the LHB was 93.7% in learning and 80.66% in generalization. Coupled with the clinical data of each patient, the accuracy of the model assembling the CNN and MLP were respectively 77% and 58% in learning and in generalization.
The AI model built from a CNN manages to determine the healthy or pathological state of the LHB with an accuracy rate of 80.66%. An increase in input data to limit overfitting, and the automation of the detection phase by a Mask-R-CNN are ways of improving the model. This study is the first to assess the ability of an AI to analyze arthroscopic images, and its results need to be confirmed by further studies on this subject.
III Diagnostic study.
Can artificial intelligence help decision-making in arthroscopy? Part 2: The IA-RTRHO Model – a decision-making aid for long head of the biceps diagnoses in small rotator cuff tears
2023, Revue de Chirurgie Orthopedique et Traumatologique
Les applications possibles de l’intelligence artificielle (IA) en chirurgie orthopédique sont prometteuses. Le deep learning est utilisable en chirurgie arthroscopique grâce au signal vidéo exploitable par computer vision. La gestion peropératoire du tendon du long biceps (TLB) est l’objet d’une controverse ancienne. L’objectif principal de cette étude était de modéliser une IA diagnostique capable de statuer sur le caractère sain ou pathologique du TLB sur des images arthroscopiques. L’objectif secondaire était de créer un deuxième modèle IA diagnostique en fonction d’images arthroscopiques et des données médicales de chaque patient, cliniques et imagerie, pour statuer sur le caractère sain ou pathologique du TLB.
L’hypothèse de l’étude était qu’il était possible de construire un modèle IA à partir d’images opératoires d’arthroscopie d’aide au diagnostic du caractère sain ou pathologique du TLB qui ferait mieux qu’une analyse humaine.
Les données prospectives cliniques et d’imagerie de 199 patients ont été collectées et associées aux images issues d’une analyse vidéo arthroscopique protocolisée validée, dite « vérité terrain », faite par le chirurgien opérateur. Un modèle basé sur un réseau neuronal convolutif (RNC) modélisé via un transfer learning sur le modèle Inception V3 a été construit pour l’analyse des images arthroscopiques. Ce modèle a ensuite été couplé à un perceptron multicouche (MultiLayer Perceptron, MLP) intégrant les données cliniques et imageries. Chaque modèle a été entraîné et testé selon apprentissage supervisé.
La précision du RNC sur le diagnostic du statut sain ou pathologique du TLB était de 93,7 % en apprentissage et 80,66 % en généralisation. Couplé aux données cliniques de chaque patient, la précision du modèle assemblant le RNC et un MLP étaient respectivement de 77 % et 58 % en apprentissage et en généralisation.
Le modèle IA construit à partir d’un RNC parvient à déterminer le caractère sain ou pathologique du TLB avec un taux de précision de 80,66 %. Une augmentation des données d’entrée pour limiter l’overfitting, et l’automatisation de la phase de détection par un Mask-R-CNN sont des voies d’amélioration du modèle. Cette étude est la première à évaluer la capacité d’une IA à analyser des images arthroscopiques. Ses résultats doivent être confirmés par d’autres études sur le sujet.
III ; étude diagnostique.
Development and validation of an objective assessment of surgical skill in arthroscopic management of meniscal tear: A pilot study
2023, Surgery in Practice and Science
As resident evaluation moves to a competency-based system, validated tools for assessment of surgical skill are increasingly important. We created and validated a checklist to measure resident surgical skill for arthroscopic management of meniscal tear.
Using a Delphi survey method, we created an objective, structured assessment of surgical skill for treatment of meniscal tears. The Meniscus Treatment Task List (MTTL) comprises 5 domains: diagnostic arthroscopy, medial meniscectomy, lateral meniscectomy, medial meniscal repair, and lateral meniscal repair. Orthopaedic surgery residents were recruited to perform diagnostic arthroscopy, partial meniscectomies, and all-inside meniscal repairs with cadaveric models. Arthroscopic videos were graded by fellowship-trained surgeons using the MTTL and the validated Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale (GRS). Postgraduate year (PGY), operative time, and case logs were recorded for each resident. Data were analysed using bivariate correlation, analysis of variance, pairwise comparison, Pearson's correlation coefficient, and intraclass correlation coefficient. α=0.05.
Twenty-two orthopaedic surgery residents (PGY1–PGY4) participated. MTTL scores were higher in the PGY4 class than in the PGY1 class (mean difference, 11 points, p = 0.04). Operative time was inversely correlated with number of cases logged (r = –0.53, p = 0.01), number of arthroscopic cases logged (r = –0.50, p = 0.02), and MTTL score (r = –0.46, p = 0.03). MTTL score was positively correlated with number of cases (r = 0.44, p = 0.04) and number of arthroscopic cases logged (r = 0.50, p = 0.02). MTTL scores were positively correlated with the ASSET GRS (r = 0.71, p<0.001). Intraclass correlation coefficient of 0.89 and Pearson's correlation coefficient of 0.89 demonstrated strong interrater reliability of MTTL scores (p<0.01).
This pilot study demonstrates the validity and reliability of the MTTL for assessing resident proficiency in arthroscopic management of meniscal tears in cadaveric specimens. Expansion of this model to other orthopaedic procedures for objective assessment of surgical skill may be useful.
Defining operative experience targets in surgical training: A systematic review
2022, Surgery (United States)
Citation Excerpt :
In Zhang et al’s 2009 study of 9 urology trainees, expert-level operative times were observed after 25 to 30 cases.43 Thirteen further studies used intraoperative assessment tools to determine a relationship between the case volume and development of competence.17,20–22,26,31,35–39,41,44 Four studies in general and orthopedic surgery from the United Kingdom tracked the number of cases required before trainees are awarded ‘level 4’ scores (corresponding to a competent performance) using the validated PBA tool.17,20–22
The surgical learning curve is an observable and measurable phenomenon. Operative experience targets are well established as a proxy measure for operative competence in surgical training across jurisdictions. The aim of this study was to critique the available evidence regarding the relationship between operative experience in surgical training and trainee competence.
A systematic review of the PubMed, Embase, Web of Science, and Cochrane library databases was conducted in accordance with the Preferred Items for Systematic Reviews and Meta-Analyses guidelines. Articles were sought that defined the relationship between procedural volume in surgical training and trainee competence, proficiency, or mastery. The educational impact of included studies was evaluated using a modified Kirkpatrick model.
Of 3,672 records identified on database searching, 30 papers were ultimately included. Fourteen studies defined operative experience thresholds using operative time as a surrogate measure of competence, whereas another 8 used trainer assessments of operative performance (Kirkpatrick level 3). A further 5 studies were able to determine the relationship between trainee case volumes and subsequent patient outcomes (Kirkpatrick level 4b).
Many studies have recorded competent trainee performance in key index procedures after reaching experience threshold numbers in excess of currently mandated targets across jurisdictions. The evidence relating current operative experience targets to patient outcomes across a range of surgical subspecialties of surgical subspecialties is lacking. This review supports a move toward criterion-based referencing of operative performance targets in surgical training.
ACGME Case Volume Minimums Decrease the Number of Shoulder and Knee Arthroscopies Performed by Residents
2021, Arthroscopy, Sports Medicine, and Rehabilitation
The purpose of this study was to examine how the implementation of Accreditation Council of Graduate Medical Education (ACGME) case minimums in 2013 has affected the number of shoulder and knee arthroscopies performed by orthopaedic surgery residents during their clinical training.
The ACGME orthopaedic surgery case log data from graduation years 2007 to 2019 was used to evaluate the number of shoulder and knee arthroscopies performed. The mean and median number of cases performed per resident were compared for the years prior to implementation of the ACGME case minimum (2007-2012) and after (2013-2019).
The ACGME orthopaedic surgery case minimums resulted in a significant decrease in the mean number of shoulder and knee arthroscopies performed. The mean number of shoulder arthroscopies performed in the years before and after the case minimum requirement were 109.8 and 82.0 (P = .025), respectively. The mean number of knee arthroscopies performed in the years before and after the case minimum requirement were 178.6 and 124.8 (P = .006), respectively. Residents in the tenth percentile of cases performed still met the required ACGME case minimums each year. The mean total of all cases performed in the years before and after the case minimum requirement were 2045.5 and 1699.3 (P = .038), respectively.
The number of shoulder and knee arthroscopies performed by orthopaedic surgery residents significantly decreased after the implementation of ACGME case minimums, which may be due to underreporting of cases.
This study may demonstrate the effect of the implementation of the ACGME case minimums on the number of shoulder and knee arthroscopies performed by orthopaedic surgery residents.

View all citing articles on Scopus

: Investigation performed within the Division of Orthopaedics, University of Toronto Sports Medicine Program, Women's College Hospital, and University Health Network, Toronto, Ontario, Canada.

: The authors report the following potential conflict of interest or source of funding: C.V. receives support from Smith & Nephew, Biomet, and Stryker.

View full text

Original ArticleThe Knee Arthroscopy Learning Curve: Quantitative Assessment of Surgical Skills

Purpose

Methods

Results

Conclusions

Clinical Relevance

Section snippets

Subjects

Results

Discussion

Conclusions

Surgery

Arthroscopy

Arthroscopy

Surgery

Arthroscopy

J Shoulder Elbow Surg

Arthroscopy

A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: The financial impact of orthopaedic surgical training in the operating room

Iowa Orthop J

Virtual reality simulation of arthroscopy of the knee

Arthroscopy

Evaluation of a virtual reality simulator for arthroscopy skills development

Arthroscopy

The early effects of code 405 work rules on attitudes of orthopaedic residents and attending surgeons

J Bone Joint Surg Am

Objective assessment of technical surgical skills

Br J Surg

Assessing operative skill. Needs to become more objective

BMJ

The development of an objective model to assess arthroscopic performance

J Bone Joint Surg Am

How soon can I be proficient in embryo transfer? Lessons from the cumulative summation test for learning curve (LC-CUSUM)

Hum Reprod

Quantitative assessment of procedural competence. A prospective study of training in endoscopic retrograde cholangiopancreatography

Ann Intern Med

Use of the Learning Curve–Cumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: Fetoscopic laser ablation as a model

Am J Obstet Gynecol

Original Article
The Knee Arthroscopy Learning Curve: Quantitative Assessment of Surgical Skills