Background
Flexible cystoscopy (FC) is a vital diagnostic and therapeutic urological procedure, which enables immediate visual assessment of both the urethra and bladder.
FC comprises multiple steps, each of which require varying degrees of endoscopic skill. It is often assumed that competence in performing FC is achieved in the early years of urological training or within a limited number of procedures. Guidance informing assessment of competency in performing FC is limited. In 2000, a working party of the British Association of Urological Surgeons (BAUS) recommended that a minimum of 60 FCs should be performed under supervision to achieve technical competence [
1]. This number has not been validated. It is now recognised that individuals training to perform a procedure, acquire skills at different rates [
2]. Indicative numbers are a weak method of assessing competence and fail to identify or aid struggling trainees. There is a need for a more objective method to assess competency and guide training.
Cumulative Sum (CUSUM) analysis is a statistical tool that can be used to evaluate the development of competence in defined tasks [
3]. CUSUM analysis has previously been used to chart learning curves and maintaining competency in surgical techniques, but not in FC, in vivo [
3].
The aim of this study was to prospectively define the learning curve in FC of a surgical trainee with no previous FC experience and to evaluate the role of CUSUM as an objective measure of achieving and maintaining competency.
Discussion
This study describes the learning curve to competency in FC of a trainee with no previous experience in endoscopy under supervision. In an era of quality assurance and credentialing the results provide further evidence that CUSUM analysis is an objective technique, which can be used to evaluate progression to competence.
The prospective data in the current study demonstrate that CUSUM is a relatively simple and sensitive method to apply practically to self assessment in surgical training. CUSUM was able to highlight areas for improvement, guiding further training in addition to defining competence. In this study, the trainee took longer to achieve, and maintain, competency than suggested indicative numbers [
1].
The utility of CUSUM was that it was able to define the specific aspects of the procedure which the trainee found most difficult. For three out of five of the components, acceptable performance was maintained following the 122nd procedure. Competence in examination of the ureteric orifices and trigone was a FC step which took substantially longer to acquire, being achieved by the 280th procedure. CUSUM highlighted this as an area for targeted tuition in this trainee.
Even if it is assumed that performing a minimum number of procedures will result in competence the number of FCs required to attain competence has never been validated. BAUS recommended a minimum of 60 procedures in 2000 [
1]. In 2014 the Speciality Advisory committee in Urology (SAC) stipulated that the indicative number of FCs which must be performed for the award of a Certificate of Completed Training (CCT) in Urology should be 300. Prior to this, a review of logbook data from trainees applying for CCT between 2010 and 2012 revealed that only 42 % had recorded flexible cystoscopy activity [
11]. This wide range of recommendations and trainee activity highlights the need for an alternative method, such as CUSUM, to be introduced as a more robust modality for determining competence.
CUSUM analysis has not previously been used to assess skill acquisition in FC. Studies, using virtual reality simulators to assess skill acquisition for FC, have developed a five-point Global Rating Score (GRS). Although this has been of value in evaluating technical and nontechnical skills, it may be limited by inter assessor variability [
12,
13]. Such variability does not occur with CUSUM analysis due to each defined task having a binary outcome. In addition CUSUM has the advantage of being suitable for both self-assessment and supervisor assessment. As a result, CUSUM analysis has the potential to be incorporated into the trainee’s curriculum with the plotted graphs providing readily visualised, accurate, comparable evidence of progression and competence rather than the current implicit logbook approach.
While ideally CUSUM could be used as a tool for assessing skill acquisition, and its maintenance, in trainees and consultants a significant issue is that the statistical calculations are detailed and time intensive. This may be one of the reasons why, to date, CUSUM has not been widely adopted. A possible development would be creation of software to facilitate the data entry and analysis.
The present study has limitations; CUSUM was demonstrated to be an objective technique however the authors acknowledge that only one trainee’s performance was assessed. Maguire et al. used CUSUM analysis for the evaluation of a group of trainees performing retropubic mid-urethral sling procedures [
14]. In keeping with that study, our study found the number of procedures required to acquire, and maintain, competence in performing the procedure was significantly more than expected [
14]. Furthermore, Maguire at al. identified considerable variability in the number of procedures each trainee needed to achieve competence [
14]. For CUSUM to be used for FC assessment routinely its applicability would have to be evaluated in a larger trainee cohort where similar inter-trainee variability is likely be identified. The very least that such a study would achieve would be a more accurate estimate of the range of indicative numbers which a trainee requires to achieve, and maintain, competence.
A key element in CUSUM analysis is the determination of the acceptable and unacceptable failure rates of the FC procedure. Another possible limitation of this study is that these rates were based on a relatively small sample of thirteen consultants. Despite the sample being representative of the centres involved in training in the South West of England it would be desirable to increase this sample size in future studies.
Patient experience and complication rates are important outcome measures, which are integral to true competence. Currently these measures are not incorporated into UK urology trainees’ assessments. These factors were not assessed in this study as part of the CUSUM analysis because the focus was on FC performance. It would be appropriate in future studies to incorporate patient experience into CUSUM and a parallel audit to accurately capture complications.
Conclusion
This study demonstrates the successful use of CUSUM analysis in the assessment of surgical competence for FC. The method is one, which could be used to assess, and monitor competence, in surgical trainees, however, validation of the process, using a larger trainee cohort, is required.
Abbreviations
BAUS, British Association of Urological Surgeons; CCT, certificate of completed training; CUSUM, cumulative sum; FC, flexible cystoscopy; GRS, global rating score; ISCP, Intercollegiate Surgical Curriculum Programme; MRC, medical research council; NHS, National Health Service; SAC, Surgical Advisory Committee
Acknowledgements
We would like to thank Professor Kenneth MacKenzie for his assistance and guidance in this research.