Assessing the learning curve in off-pump coronary artery surgery via CUSUM failure analysis

The methods to critically analyze surgical techniques as they develop can be challenging. The surgical literature has been criticized by a preponderance of small case series with relative paucity of randomized control trials. Surgeons often rely on Level V evidence from more experienced surgeons. Procedural learning is often through direct observation or videos without sufficient scientific analysis. Through individual interpretation of the described or observed techniques, surgeons may adapt procedures often not yet knowing the surgical indications, procedural nuances, length of the learning curve, and eventual outcomes. This chapter discusses research study designs for procedural technique development, phases of study development, and the problems intrinsic to each phase.

Objective measures of cardiac surgery trainee progress are limited despite a push for competency-based assessments. We hypothesized that the cumulative sum failure technique could provide a risk-adjusted, quantitative measure of resident learning curves and competence.

Records of all coronary artery bypass grafting and valve operations performed by cardiac-track residents from 2007 to 2017 at a single institution were stratified by operative resident. Multivariable regression evaluated the association among resident, case number, and postoperative outcomes. To evaluate performance over time, risk-adjusted cumulative sum failure analysis was performed, taking into account institutional expected values and comparing residents with study-defined “early alert” and “concern” boundaries.

A total of 3937 Society of Thoracic Surgeons Predicted Risk of Mortality cases were evaluated from 19 residents. Observed-to-expected ratios for mortality and combined morbidity-mortality were 0.66 and 0.72, respectively, and each individual resident exhibited better than predicted outcomes (all observed:expected ratios <1). When evaluating cumulative sum failure learning curves, residents exhibited an initial slight increase in complications, followed by improvement and better than expected performance. The “early alert” boundary was crossed by 36.8% of residents at any point in training, with 94.7% of residents under this boundary at the end of training. The higher “concern” boundary was crossed by 2 residents (10.5%), although all residents ended their training below this boundary.

Outcomes for trainee-performed cardiac surgery procedures were excellent, with no association between individual trainees and adverse events. Cumulative sum failure analysis based on postoperative outcomes is a potential tool for objective evaluation of resident proficiency.

Las gráficas cumulative sum (CUSUM) se engloban dentro de gráficas de control de calidad, son posiblemente las que mejor se adaptan y las más utilizadas en la monitorización de los procesos clínico-asistenciales.

Una de sus principales ventajas es su empleo en casos raros y en eventos con baja incidencia, en los cuales sería necesario obtener una muestra y un tiempo de seguimiento amplio, imposible en determinados casos, mediante métodos estadísticos convencionales. También por ello, son útiles para estudiar curvas de aprendizaje, la introducción de nuevas tecnologías y, en general, para valorar la calidad de los propios resultados asistenciales, porque su perfil es sensible a cambios de tendencias (positivas o negativas) muy sutiles en los resultados, que no se observarían con otros métodos.

Por otra parte, su uso se puede expandir más allá de lo que es el control o la monitorización de calidad, aspecto que resulta novedoso en investigación clínica.

Cumulative sum graphs are quality control charts that are possibly the most frequently used for monitoring clinical-care processes.

One of their main advantages is the use in rare cases and in events with low incidence, where it would be necessary to obtain a large sample and a long follow-up time with conventional statistical methods, which is impossible in certain cases. This is also why they are useful for studying learning curves, the introduction of new technologies and, in general, for assessing the quality of care outcomes themselves, because their profile is sensitive to very subtle changes in trends (positive or negative), which would not be observed with other methods.

On the other hand, their use can be expanded beyond quality control or monitoring, which is a new aspect in clinical research.

The aim of this study was to evaluate the learning curve in performing transfemoral TAVI (TF-TAVI).

Between October 2006 and October 2013, 312 consecutive TF-TAVI cases performed by 6 interventional cardiologists, using the Edwards Sapien valve and 104 using the CoreValve, were included in the present analysis. Cumulative sum (CUSUM) failure analysis of combined 30-day safety endpoint was used to evaluate learning curves.

The CUSUM analysis revealed a learning curve regarding the occurrence of 30-day adverse events with an improvement after the initial 86 cases using the Edwards valve and 40 cases using the CoreValve. We divided the Edwards valve cases into two groups (early experience: Cases 1 to 86; late experience: Cases 87 to 312). The rate of 30-day mortality and 1-year mortality significantly decreased in the late experience group (17% to 7%, p = 0.019; 34% to 21%, p = 0.035, respectively). We divided the CoreValve cases into two groups (early experience: Cases 1 to 40; late experience: Cases 41 to 104). The rate of 30-day mortality and 1-year mortality significantly decreased in the late experience group (20% to 6%, p = 0.033; 38% to 15%, p = 0.040, respectively). The groups including both valves were also analyzed after propensity-matching (early [n = 52] vs late [n = 52]). This model also showed that 30-day and 1-year mortality rates were significantly lower in the late experience group (13% to 1%, p = 0.028; 34% to 20%, p = 0.042, respectively).

An appropriate level of experience is needed to reduce the complication rate and mortality in TF-TAVI.

Our objective was to establish baseline data and develop a tool to allow for systematic evaluation of pediatric cardiac surgical complications. As a first step, we examined the incidence and distribution of complications, risk stratified by case complexity in a single institution. With improving mortality rates for congenital heart surgery, the next frontier for improving patient outcomes is characterizing and reducing complications. Currently, no standardized approach is available to monitor the incidence and severity of all complications associated with a congenital cardiac surgery program.

Complications occurring in pediatric cardiac surgical patients (January 2006 to March 2009) were collected by database review applying standardized definitions. The surgical procedures were stratified by complexity to analyze the distribution of complications over the risk spectrum. Each complication was assigned a severity coefficient (1–3) used to calculate the combined effect of frequency and severity. The cumulative sum method was used to determined the trend of the adverse outcomes.

Of 292 procedures, 84 (28.8%) were associated with a total of 150 complications. Of the 150 complications, 37 occurred in patients who died. The most common complications were arrhythmias (14.5%), cardiac (12.6%), and operative (12.6%). There was a linear relationship between the frequency and severity of complications and surgical complexity, as stratified using the Risk Adjustment for Congenital Heart Surgery category or Aristotle basic complexity levels (Spearman’s coefficient = 1).

When examined in a systematic fashion, the risk of complications in pediatric cardiac surgical patients is considerable. Our data illustrate that it is possible to track complications over time in a consistent manner. The effect of complication monitoring on patient outcomes remains to be proved.

Surgery and other invasive therapies are complex interventions, the assessment of which is challenged by factors that depend on operator, team, and setting, such as learning curves, quality variations, and perception of equipoise. We propose recommendations for the assessment of surgery based on a five-stage description of the surgical development process. We also encourage the widespread use of prospective databases and registries. Reports of new techniques should be registered as a professional duty, anonymously if necessary when outcomes are adverse. Case series studies should be replaced by prospective development studies for early technical modifications and by prospective research databases for later pre-trial evaluation. Protocols for these studies should be registered publicly. Statistical process control techniques can be useful in both early and late assessment. Randomised trials should be used whenever possible to investigate efficacy, but adequate pre-trial data are essential to allow power calculations, clarify the definition and indications of the intervention, and develop quality measures. Difficulties in doing randomised clinical trials should be addressed by measures to evaluate learning curves and alleviate equipoise problems. Alternative prospective designs, such as interrupted time series studies, should be used when randomised trials are not feasible. Established procedures should be monitored with prospective databases to analyse outcome variations and to identify late and rare events. Achievement of improved design, conduct, and reporting of surgical research will need concerted action by editors, funders of health care and research, regulatory bodies, and professional societies.