The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial

Purpose

Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs.

Methods

Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail.

Results

A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures.

Conclusions

SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

Introduction

In recent years, there has been increasing interest in competency-based performance assessment in surgical education.1, 2 This trend occurs at a time of growing concern among surgeons and surgical educators that residents may no longer be reaching the levels of confidence and competence expected of them by the end of training.3, 4, 5 There are many challenges to the goal of training safe and independent surgeons, including the erosion of intraoperative trainee autonomy, duty-hours restrictions, patient safety and malpractice concerns, and pressures to increase productivity in an increasingly complex health care environment.⁶ Recent national debate surrounding the risks of concurrent surgery has highlighted yet another related factor, challenging our ability to promote trainee autonomy in the surgical workplace.7, 8, 9 To address this, surgical residency programs are looking to create the systems and to induce the culture changes needed to provide trainees with specific, timely, and consistent performance feedback that would enhance their progression toward operative competence and independence.

In this context, operative performance assessment and feedback is an area that would particularly benefit from new methodologies. Many assessment methods are limited in their ability to ensure that trainees receive personalized feedback in a consistent and timely manner, even while residents are increasingly being asked to provide operative performance assessments as a prerequisite for graduation. Furthermore, little objective data exist to guide competency-based decisions related to resident progression toward independent practice. Emerging technologies offer the potential to address these issues by making it possible to collect “real-time” assessments for resident operative performances.10, 11 If widely adopted, benefits might accrue not just at the individual and program levels, but also at the national level by providing an evidence base to set benchmarks for resident progression.

Over the past 4 years, our group has studied methods for collecting more frequent operative performance assessments, culminating in the development of an innovative mobile technology called SIMPL (System for Improving and Measuring Procedural Learning). This software platform enables the collection of intraoperative feedback and performance assessments for every surgical procedure in which residents participate. Residents, faculty, programs, and accreditors can use these data to better understand the progression of intraoperative resident autonomy and performance. In this article, we describe initial feasibility data from the first 6 months of a multi-institutional trial using SIMPL. We hypothesized that SIMPL could be feasibly used by surgical training programs to increase the frequency of operative performance assessments.