Figures
Abstract
Background
The American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) is the first nationally validated, risk-adjusted, outcomes-based program to measure and compare the quality of surgical care across North America. Participation in this program may provide an opportunity to reduce the incidence of adverse events related to surgery.
Study Design
A systematic review of the literature was performed. MedLine, EMBASE and PubMed were searched for studies relevant to NSQIP. Patient characteristics, intervention, and primary outcome measures were abstracted. The intervention was participation in NSQIP and monitoring of Individual Site Summary Reports with or without implementation of a quality improvement program. The outcomes of interest were change in peri-operative adverse events and mortality represented by pooled risk ratios (pRR) and 95% confidence intervals (CI).
Results
Eleven articles reporting on 35 health care institutions were included. Nine (82%) of the eleven studies implemented a quality improvement program. Minimal improvements in superficial (pRR 0.81; 95% CI 0.72–0.91), deep (pRR 0.82; 95% CI0.64–1.05) and organ space (pRR 1.15; 95% CI 0.96–1.37) infections were observed at centers that did not institute a quality improvement program. However, centers that reported formal interventions for the prevention and treatment of infections observed substantial improvements (superficial pRR 0.55, 95% CI 0.39–0.77; deep pRR 0.61, 95% CI 0.50–0.75, and organ space pRR 0.60, 95% CI 0.50–0.71). Studies evaluating other adverse events noted decreased incidence following NSQIP participation and implementation of a formal quality improvement program.
Citation: Montroy J, Breau RH, Cnossen S, Witiuk K, Binette A, Ferrier T, et al. (2016) Change in Adverse Events After Enrollment in the National Surgical Quality Improvement Program: A Systematic Review and Meta-Analysis. PLoS ONE 11(1): e0146254. https://doi.org/10.1371/journal.pone.0146254
Editor: Mohammed Shamji, Toronto Western Hospital, CANADA
Received: September 9, 2015; Accepted: December 15, 2015; Published: January 26, 2016
Copyright: © 2016 Montroy et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The Veterans Affairs Surgical Quality Improvement Program (VASQIP), was a program developed in the 1990s to evaluate risk-adjusted surgical quality [1].The program was successful and eventually expanded to non-VA hospitals. The American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) evolved from VASQIP in 2005, and is an outcomes-based initiative to measure and compare risk-adjusted adverse events between hospitals [2]. As of 2013, 492 hospitals contributed peri-operative data to NSQIP [2]. NSQIP data are collected by trained Surgical Clinical Reviewers through a standardized and validated process. This process includes specific educational requirements and training, inter-rater reliability checks, and regular audits to ensure accuracy of data entered by Surgical Clinical Reviewers. In 2008, the overall NSQIP surgical data collection agreement rate was 98.4%, indicating highly reliable and accurate information [3].
For each patient captured in NSQIP, up to 135 variables are collected and entered into the database, including demographic data, baseline comorbidities, operative information, and 30-day post-operative adverse events and mortality. While there is some variability in case sampling, most hospitals with greater than 1,680 surgical cases per year enter the first 40 eligible cases in each eight-day period. In contrast, hospitals with less than 1,680 cases per year usually submit data from all surgical cases[4].
The information provided to NSQIP is used to identify differences in the incidence of peri-operative adverse events across specialties and between similar-procedure-volume institutions. Institutions are informed of their performance through Individual Site Summary Reports (ISSR) that are generated semi-annually by NSQIP. NSQIP reports include: total cases, number of adverse events, deaths, expected event rate based on pre-operative risk, logistic model observed/expected ratios with 95% confidence intervals (or hierarchical model odds ratios, as of July 2011), outlier status, and hospital decile rank. Relative deficiencies may be targeted by hospitals for quality improvement and the impact of quality improvement initiatives may be tracked over time using NSQIP reports. Institutions may or may not implement formal quality improvement programs to address sub-standard outcomes[5].
We performed a systematic review and meta-analysis to determine whether participation in NSQIP is associated with a reduction in peri-operative adverse events. We also examined whether simple monitoring of outcomes is sufficient to induce improvements in hospital-specific surgical outcomes, or whether quality improvement initiatives are necessary. We hypothesized that a directed quality improvement program would result in greater adverse event reductions.
Methods
Information Source and Search
A comprehensive search of Medline, Embase, and Pub Med was performed (S1 File). Two medical librarians at The Ottawa Hospital assisted in the planning and execution of the search. Since NSQIP is primarily a North American initiative, we searched for English language articles only, published between January 1947 and May 24th 2013 using the search terms NSQIP and National Surgical Quality Improvement Program. All duplicate studies were removed and two reviewers independently screened article titles and abstracts. The reviewers reconciled any disagreements on which full text articles would be obtained for further screening.
Eligibility criteria & study selection
To be eligible for this review, articles had to report data collected from NSQIP-participating hospitals, and had to present evidence of monitoring of the Individual Site Summary Report or the implementation of a quality improvement program. Included studies compared surgical data before and after commencement of either 1) monitoring the Individual Site Summary Reports (ISSRs) only, or 2) the implementation of a formal quality improvement program in addition to ISSRs. Studies which implemented a formal quality improvement program first identified potential areas for improvement using data provided by their Individual Site Summary Reports. Our primary objective was to determine the degree of surgical quality, defined as the incidence of 30-day morbidities and mortality associated with surgical procedures. We defined formal quality improvement programs as the implementation of protocols or practice guidelines within the hospital with the specific aim of reducing surgical morbidity and/or mortality.
The following categories of articles were excluded from this review:(1) Articles not available in English;(2) Studies that did not compare surgical quality data before and after either the monitoring of the NSIQP Individual Site Summary Reports or the implementation of a quality improvement program;(3) Case studies/case series;(4) Conference abstracts;(5) Commentaries, editorials and review articles;(6) Systematic reviews; and (7) Studies which included results from VASQIP patients. Although systematic reviews were not eligible, they were used to identify additional relevant articles.
Data collection process & data items of interest
For each article, two reviewers abstracted data using a standardized electronic form. Any disagreements were reconciled by consensus, and if needed, a third party. For studies that reported surgical outcome data over multiple years before and after joining NSQIP, we considered the first year of data presented as pre-quality improvement and the most recent year of data as post-quality improvement implementation. When the number of adverse events was reported as a proportion of the total number of patients, the number of events was calculated by multiplying the total number of patients in the study by the proportion of patients with the adverse event. Publication date, country of study origin, duration of follow-up, and study population size were also collected. Patient characteristics and the type of formal quality improvement program utilized were recorded when available. Outcome data collected included any individually reported adverse event, overall 30-day morbidity, and mortality before and after implementation of the quality improvement initiatives.
Quality of Studies
There are no validated methods for assessing the quality of observational studies included in a systematic review. We therefore conducted an assessment of study quality based on STROBE guidelines for reporting of observational studies[6].
Analysis
Overall pooled estimates of the risk ratios (pRR) with 95% confidence interval (CI) for each outcome (e.g. superficial surgical site infections) were calculated using random effects models. In addition, studies were stratified by use of a specific quality intervention or not. No statistical testing was performed to determine the association between quality intervention and adverse events. Funnel plots were generated to assess potential publication bias. All analysis was done using the Comprehensive Meta Analysis Software, version 2.2.064.
Results
Of the 1024 titles and abstracts identified by the electronic search, 940 were excluded for being irrelevant or for not meeting our inclusion criteria (Fig 1). A total of 84 full text articles were independently reviewed and 73 were excluded. Eleven full-text articles were abstracted and included in the analysis. Two of the included articles reported data from state-wide cohorts, one within 10 Tennessee hospitals, and one within 16 Michigan hospitals. Therefore, data from 35 institutions, representing 43,010 patients were included in the analysis.
NSQIP = National Surgical Quality Improvement Program; VASQIP = Veteran’s Affairs Surgical Quality Improvement Program.
Characteristics of Included Trials
Included articles were published between January 2010 and January 2013 from hospitals in the United States. Surgical quality data were collected between 2005 and 2011, with specific years varying between studies. Colorectal procedures were the most commonly analyzed surgeries (n = 4), followed by vascular (n = 2), non-specific general surgery (n = 1), otolaryngology (n = 1), non-cardiac (n = 1), intra-abdominal (n = 1) and hepatopancreatobiliary (n = 1) (Table 1).
Nine (82%) of the 11 studies implemented a quality improvement program. Of these, one study used Surgical Care Improvement Project (SCIP) guidelines[7], while one study used both SCIP and Comprehensive Unit-based Safety Program (CUSP) guidelines[8]. Two studies used modifications of the SCIP guidelines. Five studies created their own formalized quality improvement program that was not based on any specific guidelines. The remaining two studies did not institute a quality improvement program; however, all 11 studies used NSQIP performance feedback from ISSRs to monitor their outcomes both before and after the implementation of NSQIP (Table 1).Since studies which implemented a formal quality improvement program had first monitored their results by using their ISSR’s, any changes in morbidity or mortality rates were in addition to the changes seen via the monitoring of ISSR’s alone.
The quality of included studies was evaluated based on an adaptation of STROBE guidelines. All included studies reported on study design, study setting, variables analyzed, statistical methods, outcome data, and summary/interpretation of key findings. Overall, each included study contained at least 60% of the recommended STROBE checklist items (S1 Table).
Overall 30-day post-operative mortality
Only one study presented data on 30-day post-operative mortality[9]. In this study of 10 Tennessee hospitals that did not implement a specific initiative for mortality reduction, no decrease in mortality was observed after joining NSQIP (RR 1.03; 95%CI 0.88–1.19).
Overall 30-day post-operative morbidity
Two studies presented data on 30-day post-operative morbidity (defined as any complication within 30 days of surgery)[10,11]. In one of these studies, no specific quality improvement program was used and a small reduction in morbidity was observed (RR 0.87; 95%CI0.77–0.99)[11]. In the other study, a quality improvement program was instituted and consisted of a three-session team-based training program for surgical staff, followed by the implementation of a standardized protocol using preoperative briefing and post-operative debriefing checklists[10]. The reduction in morbidity was greater at that institution, but it was not statistically significant (RR 0.52; 95% CI 0.23–1.18).
Surgical infections
Nine studies presented data on the incidence of superficial surgical site infections (SSIs), 7 presented deep SSI information, and 6 presented organ/abdominal space infection information (Figs 2–4). Modest improvements in superficial (Fig 2) and deep wound infections (Fig 3) were observed at centers that did not institute a quality improvement program, with pooled risk ratios of 0.81(95% CI 0.72–0.91) and 0.82 (95% CI 0.64–1.05), respectively. However, no improvement was observed for organ/abdominal space infections, with a pooled risk ratio of 1.15 (95% CI 0.96–1.37) (Fig 4). Among centers that reported specific interventions for the prevention/treatment of SSIs, reductions in infection rates were substantial. For superficial, deep, and organ/abdominal space SSIs, the pooled risk ratios were 0.55 (95% CI 0.39–0.77), 0.61 (95% CI 0.50–0.73) and 0.60 (95% CI 0.50–0.71), respectively (Figs 2–4). Funnel plots to assess potential selection bias are presented in Figs 5–7.
Thromboembolic and other complications
Two studies reported on deep venous thrombosis complications, one which implemented a quality improvement program, and one did not. The study which did not implement a quality improvement program[9]observed an increase in the rate of deep venous thrombosis, with a relative risk of 1.35(95% CI1.04–1.76). The study which did implement a quality improvement program [12] observed a decrease in the incidence of deep venous thrombosis (RR 0.3; 95%CI 0.002–5.42). For all other complications, such as pneumonia [13] and septic shock, modest or no improvement was observed unless a quality improvement initiative was used (S2 Table).
Discussion
The American College of Surgeons’ National Surgical Quality Improvement Program provides comprehensive and accurate performance feedback to hospitals. However, upon conducting a systematic review of the literature published from NSQIP-participating hospitals, our analysis suggests that implementing a formal quality improvement program may be more effective than merely monitoring surgical outcomes using NSQIP Individual Site Summary Reports. Indeed, in NSQIP hospitals that implemented a quality improvement program, there was a reduction in the incidence of almost all post-operative complications.
This review captured studies implementing a broad range of quality programs. All of the seven studies that reported such initiatives for a common post-operative complication of wound infection reported a successful reduction in surgical site infections[10, 14–19]. Common components of these programs were: warming of the patient prior to surgery/maintaining peri-operative and post-operative normothermia, proper pre-operative, peri-operative, and post-operative use of antibiotics, and peri-operative and post-operative glucose control. An eight-step program to reduce pneumonia rates was introduced in one study that included education of all surgical ward nursing staff on their role in pneumonia prevention and twice daily oral hygiene with chlorhexidine swabs. At that center, pneumonia relative risk was reduced from 0.8% to 0.2%. Unplanned intubation rates were successfully reduced by implementing a three-step program, including the creation of a working group, the identification of operational problems, and development of processes to change practice[10]. Overall 30-day morbidity was reduced by having surgical staff participate in a three session team-based training program followed by the implementation of a standardized protocol using preoperative briefing and post-operative debriefing check lists[10]. In summary, the quality improvement programs were often directed at specific outcomes and, in general, were highly effective. Although shown to be effective in reducing specific morbidities, the cost of participating in the NSQIP program must be taken into account, especially for small-volume institutions.
There are several potential limitations to this comprehensive review. As with all systematic reviews, we are limited by the quality of the published studies and authors may have preferentially reported programs that have shown an improvement, resulting in a publication bias (Figs 5–7). It is possible that improvements seen in the incidence of SSIs after a formal quality improvement program were due to publication bias. We were not able to adjust for differences in case-mix when comparing evaluation periods. Therefore, the true impact of NSQIP participation may be over or under-estimated if the baseline patient risk changed over time at individual institutions. A limitation of the present analysis is the fact that the majority of NSQIP participating institutions are located in English speaking countries and that all publications were from the United States. A main limitation of our review was the small number of studies included in the final analysis. In certain instances, outcomes such as any 30-day morbidity or sepsis were only reported by two studies and overall 30-day mortality was only reported by one study. Therefore, our ability to form definitive conclusions about these outcomes is limited. Total morbidity rates were only discussed by two of the included papers. While improvements in the incidence of specific morbidities were reported by other papers, conclusions regarding the true, overall surgical quality improvement were limited by the small number of publications regarding overall morbidity.
Most hospitals within the United States are accredited by the Joint Commission, which requires an intrinsic quality improvement process. This means that the majority of the hospitals in our study had an ongoing improvement program in place prior to joining NSQIP. However, the intrinsic program required by the Joint Commission targets overall patient care improvement within a hospital. There is no identification of potential areas for improvement or areas of weakness at each specific hospital. Quality improvement programs directed at reducing the incidence of specific morbidities or reducing morbidities associated with a specific procedure are likely results of the NSQIP ISSR’s. Lastly, the phenomenon of regression to the mean may explain some of the improvements observed in hospitals that reported on outcomes that had historically performed worse in than similar-volume institutions. However, this effect on NSQIP institutions appears to be minimal, as NSQIP-related improvements were found to be independent from this effect[5].
Conclusion
To our knowledge, this is the first systematic review to attempt to evaluate the effectiveness of the National Surgical Quality Improvement Program in improving post-operative morbidity and mortality. No analysis was performed on mortality rates, as there was insufficient data. Our analysis of morbidity rates suggest that implementation of a formal quality improvement program based on NSQIP feedback was more effective compared to passive monitoring of surgical outcomes generated by NSQIP’s Individual Site Summary reports. These findings have noteworthy implications for hospitals that participate in NSQIP but have not yet adopted quality improvement programs to address deficiencies. Establishing and implementing outcome-directed quality improvement programs may be of significant benefit to NSQIP-participating hospitals that aim to improve surgical quality by reducing the incidence of specific post-surgical complications, however it is not known if this translates to overall surgical quality improvement.
Supporting Information
S1 Table. Assessment of study quality based on STROBE guidelines.
https://doi.org/10.1371/journal.pone.0146254.s003
(DOCX)
S2 Table. Specific morbidity rates before and after National Surgical Quality Improvement Program implementation.
https://doi.org/10.1371/journal.pone.0146254.s004
(DOCX)
Acknowledgments
The authors would like to thank The Ottawa Hospital librarians Johanne Gohier and Risa Shorr for developing the search strategy and retrieving all articles.
Author Contributions
Conceived and designed the experiments: JM RHB KW DS. Performed the experiments: JM AB TF. Analyzed the data: JM RHB SC KW LTL. Wrote the paper: JM RHB DAF DS SC LTL KW AB TF.
References
- 1. Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X, et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg2013;217(2):336–346.e331. pmid:23628227
- 2. Fuchshuber PR, Greif W, Tidwell CR, Klemm MS, Frydel C, Wali A, et al. The power of the National Surgical Quality Improvement Program—achieving a zero pneumonia rate in general surgery patients. Perm J2012;16(1):39–45. pmid:22529758
- 3. Shiloach M, Frencher SK Jr, Steeger JE, Rowell K, Bartzokis K, Tomeh MG, et al. Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg2010;210(1):6–16. pmid:20123325
- 4.
American College of Surgeons National Surgical Quality Improvement Program. User Guide for the Participant Use Data File. Available: http://site.acsnsqip.org/wp-content/uploads/2012/03/ACS-NSQIP-Participant-User-Data-File-User-Guide_06.pdf.
- 5. Hall BL, Hamilton BH, Richards K, Bilimaria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg2009;250(3):363–376. pmid:19644350
- 6.
Strengthening the reporting of observational studies in epidemiology (STROBE) Statement. STROBE Checklist for cohort, case-control, and cross-sectional studies. 2009. Available: http://www.strobe-statement.org/.
- 7.
UCLA Surgical Quality and Patient Safety. Surgical Care Improvement Project. Available: http://www.scip.mednet.ucla.edu/pages/scipabout.
- 8.
Agency for Healthcare Research and Quality. Using a Comprehensive Unit-based Safety Program to Prevent Healthcare-Associated Infections. Available: http://www.ahrq.gov/professionals/quality-patient-safety/cusp/index.html.
- 9. Guillamondegui OD, Gunter OL, Hines L, Martin BJ, Gibson W, Clarke C, et al. Using the national surgical quality improvement program and the Tennessee surgical quality collaborative to improve surgical outcomes. J Am Coll Surg2012;214(4):709–714. pmid:22265639
- 10. Bliss LA, Ross-Richardson CB, Sanzari LJ, Shapiro DS, Lukianoff AE, Bernstein BA, et al. Thirty-day outcomes support implementation of a surgical safety checklist. J Am Coll Surg 2012;215(6):766–776. pmid:22951032
- 11. Henke PK, Kubus J, Englesbe MJ, Harbaugh C, Campbell DA. A statewide consortium of surgical care: a longitudinal investigation of vascular operative procedures at 16 hospitals. Surgery2010;148(4):883–892. pmid:20719350
- 12. Stachler RJ, Yaremchuk K, Ritz J. Preliminary NSQIP results: A tool for quality improvement. Otolaryng Head Neck2010;143(1): 26–30.e23.
- 13. Wren SM, Martin M, Yoon JK, Bech F. Postoperative pneumonia-prevention program for the inpatient surgical ward. J Am Coll Surg 2010;210(4):491–495. pmid:20347742
- 14. Berenguer CM, Ochsner MG Jr, Lord SA, Senkowski CK. Improving surgical site infections: using National Surgical Quality Improvement Program data to institute Surgical Care Improvement Project protocols in improving surgical outcomes. J Am Coll Surg 2010;210(5): 737–741. pmid:20421041
- 15. Ceppa EP, Pitt HA, House MG, Kilbane EM, Nakeeb A, Schmidt CM, et al. Reducing surgical site infections in hepatopancreatobiliary surgery. HPB2013;15(5):384–391. pmid:23557410
- 16. Cima R, Dankbar E, Lovely J, Pendlimari R, Aronhalt K, Nehring S, et al. Colorectal surgery surgical site infection reduction program: a national surgical quality improvement program—driven multidisciplinary single-institution experience. J Am Coll Surg2013;216(1):23–33. pmid:23127793
- 17. Compoginis JM, Katz SG. American College of Surgeons National Surgical Quality Improvement Program as a quality improvement tool: a single institution's experience with vascular surgical site infections. Am Surg 2013;79(3):274–278. pmid:23461953
- 18. Lutfiyya W, Parsons D, Breen J. A colorectal "care bundle" to reduce surgical site infections in colorectal surgeries: a single-center experience. Perm J2012;16(3):10–16. pmid:23012593
- 19. Wick EC, Hobson DB, Bennett JL, Demski R, Maragakis L, Gearhart SL, et al. Implementation of a surgical comprehensive unit-based safety program to reduce surgical site infections. J Am Coll Surg2012;215(2):193–200. pmid:22632912