Implementation and validation of a competency assessment tool for laparoscopic cholecystectomy

We included all patients with cholecystectomies fully performed by trainees from our hospital’s visceral surgery training program between June 2013 and January 2016. The interventions were performed either in an emergency or an elective setting for uncomplicated symptomatic gallstone disease or acute cholecystitis. Surgery was performed under the supervision of a senior surgeon with board certification. Procedures which were not performed entirely by the trainee or for which the video recording was not available were excluded. If the senior surgeon felt that significant oral guidance was given during the surgery, this was mentioned at the bottom of the assessment sheet and the procedure was excluded from the analysis. We collected the different scores prospectively. Patient’s demographic data (age, sex, body mass index [BMI], American Society of Anesthesiologists [ASA] score) and the level of case complexity as rated by the senior surgeon were reported. The senior surgeon who had the role of the live reviewer completed the assessment tool form immediately after the surgery and gave feedback to the trainee. Each procedure was recorded, and the video material was collected at the end of the surgery. Each video received a randomized number and was anonymized for the patient, the live reviewer, and the trainee. One investigator kept a list of matches between the video number and the live assessment form. Two blinded senior surgeons designated as A and B separately reviewed the video recordings of each surgery and filled out the LCAT (except for the cholangiography item which was not shown in the video). The video reviewers did not participate in the live evaluation.

All procedures performed in this study met the ethical standards of the institutional research committee, and the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Institutional Review Board was consulted and informed consent was waived. For this type of study, formal consent was not required.

We recorded 42 laparoscopic cholecystectomies performed by 15 trainees. Eight trainees had from 1 to 3 years of post-graduate experience and seven trainees from 4 to 6 years of post-graduate experience. 30 procedures were performed by ten trainees (3 laparoscopic cholecystectomy per trainee), 8 procedures by four trainees (2 procedures per trainee) and 4 were performed by one trainee.

All interventions were supervised by one of 15 senior surgeons. The demographic data of the 42 patients are shown in Table 1. No conversion to laparotomy was observed.

Surgical education is subject to many constraints such as reduction of trainees’ surgical volumes because of the legal limitation of working hours. Every operation performed by a young surgeon is precious and a maximum amount of teaching should be extracted from it. Objective and reliable assessment tools allow to achieve this goal by guiding and, if necessary, correcting each aspect of the intervention. We demonstrated that the implementation of a new assessment tool for laparoscopic cholecystectomy is feasible and the tool appears to be objective.

Over the last 20 years, several tools have been developed to assess general surgical skills [6], skills specific to an intervention or a combination of both [7, 8]. The creation of the LCAT was inspired by a competency assessment tool used in the United Kingdom in the residency training program for right or left laparoscopic colectomies [4]. The assessment is completed at the end of this program to confirm that the surgeon is able to perform the procedure safely. The authors of the tool demonstrated that the higher the score, the better the patient’s outcome in terms of postoperative morbidity and the number of harvested lymph nodes. We used this validated assessment tool for a more common procedure, more suitable for the assessment of non-specialized surgeons.

Feedback from a supervisor after an intervention can be infrequent. A national survey was performed in the residency programs in the United States with 998 participants. Concerning postoperative feedback, only a minority of residents reported that senior surgeons reinforced the educational points of a case with them (34%), gave positive feedback (41%), or discussed areas for improvement (37%) upon completion of an operation [9]. Formal written assessment ensures that feedback is given to the trainee immediately after the surgery. Immediate rating is more accurate than one completed later after the intervention [10]. In our study, all the assessment forms were filled directly after the procedure with full compliance. The form is quick and easy to complete.

Several general scores have been developed and applied to the laparoscopic cholecystectomy. The global operative assessment of laparoscopic surgery (GOALS) consists of general items rated from 1 to 5 with anchor explanations at the first, third and fifth points. GOALS was first applied for a single step of laparoscopic cholecystectomy, the dissection of the gallbladder from the hepatic bed, and then to the entire procedure [11, 12]. The OSATS is a validated score for general surgical skills that has been used to assess laparoscopic cholecystectomy [13]. Another general and specific assessment for laparoscopic cholecystectomy is the operative performance rating system (OPRS) [6]. It is used by the ABS [1] for the assessment of operative performance during laparoscopic cholecystectomy. The assessment is divided into both general and specific for the procedure items and the VAS ranging from 1 to 7 with anchor explanations for points 1, 4 and 7. A common flaw of these scores is the lack of procedure-specific items. A VAS can be misleading and open to interpretation by the rater. In our assessment tool, we divided the laparoscopic cholecystectomy into 16 steps with anchor explanations for each rating. This allows to standardize the procedure and identify specific areas for improvement. Each step can be corrected to maximize the safety of the patient. An additional element compared to other scores is the inclusion of the peroperative cholangiography. This step may not be systematically performed at all institutions but can be valuable in detecting common bile duct stones or delineating the anatomy of the bile ducts to avoid injuries. This skill should be acquired by any independent surgeon. It therefore seems essential to us to include cholangiography in any assessment tool for laparoscopic cholecystectomy. Objectivity and reliability of the LCAT have been demonstrated in our study as the scoring results were comparable between non-blinded live reviewers and two blinded video reviewers.

Autonomy seems to be the most challenging part to assess. In an American study, all fellowship program directors completed a survey about the residents entering their programs. They reported that 30% of the surgeons could not independently perform a laparoscopic cholecystectomy [14]. In our study, there was no evident consensus about the overall autonomy of trainees. This item showed the widest score range between all the reviewers. The autonomy level assessed by VAS should be used as information for the trainee rather than a threshold to be obtained for independent practice. Overall scores based on all the items and/or successive evaluation by different raters could better assess a trainee’s autonomy.

In our study, we compared live assessment scores with the scores given by two blinded video reviewers. Inter-rater reliability was moderate to good for most of the items in the assessment tool. No statistically significant differences in the LCAT part were noted between all three reviewers.

Video assessment simplifies the process as the evaluation is not limited by the rater’s availability. It also eliminates sources of bias as the rater is blinded to the trainee’s identity.

In our study, there were two steps the raters disagreed on: the OSATS and the VAS. In the OSATS, the mean score was different between the reviewers and there were two items which showed poor reliability: knowledge of the procedure and use of the instruments. Watching a video recording without intraoperative sound, reviewers could have difficulties to evaluate the flow of the procedure and the accuracy of the trainees’ requests for instruments. This could explain the discrepancy between the raters and suggests these items should be omitted during video rating. In the VAS part, the mean score was different between the reviewers but the individual items showed good inter-rater reliability. The VAS and the OSATS are different compared to the LCAT: not all grades on the scale have an anchor explanation. The VAS has a scale of 1 to 7 but explanations are available only for grades 1, 4 and 7. The choice of a score of 2 or 5 is therefore left to the rater’s subjective interpretation. The subjectivity of scoring in the OSATS and the VAS part could explain the differences in score between the raters. The LCAT has a clear explanation for each item leaving less room for interpretation.

The LCAT scoring shows consistency between live and video rating whereas the OSATS and VAS scores differed between reviewers.

Evaluation of laparoscopic cholecystectomy with the LCAT seems to be feasible, more objective and producing results consistent between live and video reviewers.

The operative assessment could improve the patient safety and outcome. Trainees who received feedback from an experienced surgeon after performing a cholecystectomy showed significant improvements in their operative time and efficiency of movement [15]. Giger et al. [16] demonstrated that intraoperative complications were more frequent for surgeons with experience of less than 100 laparoscopic cholecystectomies. These surgeons were classified as at risk because their patients had the highest likelihood of suffering from surgery-related complications (odds ratio [OR] 1.36; p = 0.0002). Murphy et al. [17] have shown that postoperative complications after laparoscopic cholecystectomy were more frequent in a non-teaching hospital. These results reinforce the need for control of surgical safety during and at the end of a training program. A successive assessment with the LCAT during the surgeon’s training could help identify any unsafe aspect of the procedure.

The laparoscopic cholecystectomy learning curve is not well defined. In their meta-analysis, Reitano et al. [18] found that thresholds for proficiency ranging from 13 to 200 procedures have been proposed. Despite laparoscopic cholecystectomy being a common procedure, there is no consensus on the minimal number of surgeries required for a trainee to become independent. In addition, reaching a certain number of surgeries does not guarantee high quality performance. De Siqueira et al. [19] reported that out of 30 trainees who had attained 50 procedures required for cholecystectomy certification, only 47% had an assessment score valid for independent practice. The LCAT could represent an alternative to evaluate surgical competence.

There are several limitations of this study. We could not exclude a selection bias, as the most challenging cases were performed entirely or at least partially by the senior surgeon and were excluded from the assessment process. In other scores, the level of guidance is a part of the assessment. The criterion of exclusion of not fully independent surgeries also explains the low number of interventions which is another study limitation. However, we found that an assessment score on a procedure done entirely by the trainee better reflects their surgical skills. It forces the trainee to perform the entire surgery under standardized guidance of the mentor and allows safe evaluation of steps with which the trainee may feel less comfortable. The assessment tool helps identify the steps in which there is room for improvement. Another limitation is exclusion of cholangiography assessment from analysis as cholangiography was not performed in every case. Finally, we could not compare assessment scores with patient outcomes as the study was not designed to include patients’ postoperative data. A further study comparing assessment scores and patient outcomes is planned.

Surgical education faces new challenges posed by the legal hour restrictions and the resulting reduction of the caseload for specific surgeries. In order to improve surgical training, assessment tools are mandatory to help young surgeons progress with each procedure. We propose the LCAT as a new assessment tool specific for laparoscopic cholecystectomy.

Its implementation was feasible in our residency program. The LCAT seems to be suitable for either live or video review as both resulted in comparable scores in our study. With successive assessments using the same tool, a training program can track the improvement of each resident. An external expert could judge the adequacy of the procedure by analyzing a record of a resident at the end of their training. Further studies are required to assess the capacity of the LCAT to differentiate novice and expert (construct validity) and investigate the correlation between the LCAT score and patients outcomes.