Improvement Science in Anaesthesia

Anaesthetists are well placed to lead improvement projects within hospitals due to their multidisciplinary skill sets and working practices. They tend to have strong numerical skills and routinely examine data for trends in clinical settings. They have been becoming increasingly familiar with a safety culture, surgical checklists and care bundles over the last 10 years. In the UK, the Royal College of Anaesthetists (RCoA) has incorporated aspects of QI into the training curriculum, and trainees are expected to complete an audit or QI project every year.

Improvement Science is starting to gain traction in the field of Anaesthesia. One interesting example is the spread of large-scale monitoring and feedback programmes such as the National Surgical Quality Improvement Project (NSQIP), led by the American College of Surgeons. A systematic review of the benefits associated with participating in NSQIP reported that mere involvement in the programme reduced surgical morbidity although these advantages were greatest when formal QI programs were implemented [ 16]. This dataset has now been used to produce over 700 academic publications.

In the UK, the National Emergency Laparotomy Audit (NELA) has now been running for 3 years. Superimposed upon this platform for collecting and feeding back data have been several successful initiatives, such as the Emergency Laparotomy Collaborative and EPOCH (described below). Going forward, the Perioperative Quality Improvement Programme (PQIP) led by the RCoA and its Health Services Research Centre will prospectively collect process and outcome data for major elective surgery across the UK. It aims to use Improvement Science methods to feedback data in near real-time with appropriate support to enable hospitals to drive local quality improvement.

Context clearly affects the success or failure of an improvement project in Anaesthesia as in any other specialty. For example, one qualitative study reported on the differences in compliance with the WHO safer surgery checklist between high and low income settings. They reported that some resentment of the checklist was present, especially where there was conflict between the underlying philosophy of the checklist and local social, cultural and economic contexts. Understanding these contexts may therefore help predict and optimise implementation of improvement projects [ 17]. Much further work has gone on to classify context, for example at macro (national/regional), meso (organisational) and micro (team/department) levels [ 18].

The development of electronic health records (EHR) over the past 10–15 years has provided large datasets that can be used to support quality improvement. By 2014, over 80% of non-federal acute care hospitals across the USA had adopted at least a basic EHR system [ 19]. A basic EHR was defined as system that allowed electronic recording and access to patient demographics, problem lists, medication lists, discharge summaries and diagnostic test results. In the UK, there are plans to make the NHS ‘largely paperless’ by 2020 [ 20]. This increase in EHR adoption allows the use of routine data to provide baseline performance and ongoing monitoring of quality improvement initiatives. Etzioni et al. published work assessing the impact of the American College of Surgeons National Quality Improvement Program (NSQIP), this included analysis of 345,357 hospitalisations in 113 different academic institutions [ 21•]. As well as institutional clinical datasets, there are also national administrative datasets such as hospital episode statistics (HES) in the UK, which is controlled by NHS Digital. Linking to such datasets can provide additional information and reduce the burden of data collection for local teams. In the UK, this approach has been taken by national projects such as the NELA and the recently launched PQIP which are both run by the National Institute of Academic Anaesthesia’s Health Services Research Centre based at the Royal College of Anaesthetists.

The concept of ‘big data’ in healthcare research is relatively new, and its increasing use is directly related to the adoption of EHRs. The term ‘big data’ is a fluid term that is dependent upon the interpretation of the user but involves five concepts of volume, variety, velocity, value and veracity [ 22]. By using electronic health records, electronic data capture, and linking to existing national datasets, all the five Vs can be increased.

Of course, there are potential problems with using large datasets and electronic health records for quality improvement. The accuracy of data entered in to them directly influences the validity of any findings, and where routine datasets are used, the quality of this cannot always be assured. Quality control methods, such as those highlighted in Table 1, are critical to help ensure the accuracy of any effort to collect, analyse and report data [ 23•].

By considering data quality control throughout the entire design of an improvement project investigators will be able to improve the face validity of their findings. In return, this should improve uptake of results and adoption of change processes.

The Darzi report [ 24], which was a seminal report on the future of the NHS in England published in 2008, stressed the need for a more positive approach to measurement for quality improvement. ‘In order to work out how to improve we need to measure and understand exactly what we do. The NHS needs a quality measurement framework at every level.’

Clinical trials almost unanimously use outcome measures such as mortality to report the clinical effectiveness of a treatment or intervention. Improvement science uses a variety of measures such as engagement, process and outcome measures.

An engagement measure would be used to assess how well an individual or hospital is interacting with a project and may include measures such as patient recruitment or data completeness. Process measures show whether steps proven to benefit patients are followed correctly [ 25]. They measure whether an action was completed—such carrying out a preoperative risk assessment or enrolment of a patient to an enhanced recovery pathway.

Measuring processes provide a clearer understanding of what we do. Understanding them can provide additional insight into variation in outcomes measures. By using process measures, the development of targeted interventions to improve them can be more straightforward than using outcome measures alone.

Outcome measures take stock not of the processes, but of the actual results of care. They are generally the most relevant measures for patients and the measures that providers most want to change [ 25]. Meaningful comparisons of outcomes within the health care system generally require risk adjustment—accounting for patient-associated factors before comparing outcomes across different patients, treatments, providers or populations [ 26]. The reasons for this are obvious, patients with a poorer health status are on average likely to have poorer outcomes. Risk adjustment aims to account for differences in intrinsic health risks that patient brings to their healthcare encounters. For comparing outcomes, it thus “levels the playing field,” ensuring that “apples are compared to apples, not oranges” [ 26].

The need for risk adjustment poses a potential problem for quality improvement initiatives. Risk adjustment requires a suitable model that has been calibrated and validated for the cohort it is being used for. Model development often requires a large cohort of patients that may not be available to single institution projects. For data to be used for improvement it must also be made available to local teams in a timely fashion. Waiting months or even years after collection for risk-adjusted outcomes to be available does not support improvement.

There is a range of data visualisation options available within improvement science. Some of these aim to allow continuous displays of risk-adjusted outcomes in a timely fashion, such as the visual life-adjusted display, which is discussed more later.

There is compelling evidence that clinicians in other healthcare fields can improve clinical outcomes after receiving feedback on their performance. The need for individual monitoring and reflection has increased with the introduction of revalidation in the UK. A 2012 Cochrane review reported that ‘feedback may be more effective when baseline performance is low, the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and when it includes both explicit targets and an action plan. In addition, the effect size varied based on the clinical behaviour targeted by the intervention’ [ 27].

However, anaesthetists often feel that there is a lack of credible performance/quality indicators available to them. A recent qualitative study of anaesthetists’ perceptions of individualised feedback of recovery data reported that they preferred: indicators which reflect areas of care that they felt they could control; graphical data presentation; longitudinal and comparative data reporting and personalised reporting [ 28•].

The multidisciplinary teamwork nature of anaesthesia suggests that system level data is needed to be able to enact organisational changes.

To support quality improvement, time-series data is generally used. This allows the visualisation of change over time, to assess if processes and outcomes are improving following intervention.

The simplest example is a line graph, with time on the x-axis and the process or outcome of interest on the y-axis. To support monitoring and alert users to variation in results over time, there has been a variety of additional data displays developed. These include the run chart (which has the addition of a median line) and statistical process control chart (which has the addition of a mean line and control limits). It is beyond the scope of this article to discuss these in detail, but articles by McQuillan et al. [ 29••] and Benneyan et al. [ 30•] serve as an excellent introduction to their application and interpretation.

Displays have also been developed to allow the continuous visualisation of risk-adjusted outcomes. One example of this is the visual life-adjusted display (VLAD). The VLAD chart shows expected vs. observed outcomes and has been used in a variety of settings including paediatric and adult cardiac surgery, intensive care and trauma [ 31– 35].

Figure 1 shows an example of a standard VLAD chart of the difference between the expected number of deaths over a series of 210 patients, calculated by summing the estimated risk for each episode, and the observed number of deaths. The trace rises with each survival (more so for high-risk episodes) and falls with each death (more so for low-risk episodes). In the example chart, the ‘high-risk death’ had a predicted mortality risk of 25%, compared to a predicted mortality risk of 1.6% for the ‘low-risk death’. The negative deflection is greater for a low-risk death compared to that of a high-risk. Over time, if observed outcomes matched expected outcomes, the VLAD chart will be hover around the zero point on the y-axis.

There are also other forms of display proposed that incorporate risk adjustment such as the risk-adjusted cumulative sum, risk-adjusted sequential probability ratio and risk-adjusted exponentially weighted moving average charts [ 36].

Obtaining reliable measurements of a system is a necessary but not sufficient step in making improvement. Indeed, no association was found between participation in NSQIP and improved clinical outcomes in large clinical cohort of patients undergoing elective general/vascular surgery in the USA [ 21•]. This surprising finding may be explained by recent UK studies which have reported how little National Clinical Audit data gets used for practical improvement [ 37•]. A large multimethod study of the NHS found that ‘organisations were putting considerable time, effort and resources into data collection..[but] the degree to which this translated into actionable knowledge, and then into effective organisational responses, differed markedly between organisations’ [ 38]. An evidence scan by the Health Foundation reported that insufficient use of data was a barrier to implementation at every stage of the design, delivery and dissemination of improvement strategies in the NHS [ 39]. This failure to use data effectively represents a waste of both resources and opportunities.

The reasons why data may not be used to its full potential may lie with the data collection and feedback systems, the individuals receiving the data or the context in which the healthcare system is situated.

Making data ‘usable’ for improvement not only relies upon the criteria for effective feedback as described above, but also requires understanding the needs of the data’s users. Two often conflated but potentially conflicting needs are those of quality assurance (QA) and quality improvement (QI). Typically, QA identifies compliance with an agreed standard, whereas QI reflects iterative positive change towards best-practice [ 40]. In the UK, the Healthcare Quality Improvement Partnership (HQIP), the commissioner of National Clinical Audits, is currently collaborating with the Care Quality Commission (CQC), the healthcare regulator, to optimise their data for both QA and QI. This will likely culminate in a National Clinical Audit dashboard, which aims to reduce the data collection burden; co-localise audit results in the same place to enable hospitals to take a cross-cutting view of quality; increased dynamism and interactivity [ 41].

The choice of quality indicator clearly matters. Clinicians are naturally driven to improve patients’ outcomes, but process measures may provide more scope for improvement in the short term. The challenge of using reliable patient-reported outcome measures is well documented; a recent review concluded that they were best sued as ‘tin-openers’ (highlighting problems or areas of success) rather than ‘dials’ (quantifying change) [ 42].

Potential solutions are emerging to support data usage and include efficient trial designs such as that demonstrated by the EPOCH trial [ 43•]. This study aims to support clinicians in 90 UK hospitals use QI methods and NELA data to help implement a care pathway to nationally agreed standards, and is being evaluated by stepped wedge cluster randomised trial.

Finally, unintended consequences of reporting quality data should be anticipated as well as expected benefits. Individual level outcome data of surgeons in the UK has been met with some resistance, with concerns ranging from doubt over risk-adjustment processes to potential risks of gaming [ 44, 45]. A hospital ‘Report Card’ initiative for cardiac surgery in New York and Pennsylvania in the early 1990s resulted in surgeons selecting to operate on healthier patients compared to neighbouring states without report cards [ 46]. HQIP notes that the potential risks of their proposed new dashboard include over-simplification, duplication and gaming.