Identifying and understanding benefits associated with return-on-investment from large-scale healthcare Quality Improvement programmes: an integrative systematic literature review

Included articles covered different healthcare levels and disciplines. Primary care included public health, child and maternal health, and mental health. Secondary and tertiary healthcare included mental health, medical and surgical care, critical care, accident and emergency and acute care services, paediatrics and neonatal care, outpatients, pharmacy, and laboratories. One article covered both health and social care, and another article was about QI in a healthcare related charitable organisation. Articles were from these global regions: Africa, Asia, Europe, Australia, and Canada. The mostly represented regions were the US and the UK. Only 15 of 68 articles were economically focused. ROI was a specific subject of only four articles [68‐71], and five authors discussed development of QI business cases [33, 72‐75]. One article discussed cost–benefit analysis from a qualitative perspective [76], there were two economic systematic reviews, and three economic evaluations. de la Perrelle et al. [77] also found this lack of economic evaluations in their systematic review. However some authors reported their implementation costs [78‐80]. The summary of included studies can be found as Supplementary Table 3.

Thirty articles were not subject to quality assessment. These were conceptual articles, unsystematic literature reviews, and brief reports. Thirty-eight articles were subjected to quality assessment: 19 quantitative studies, three qualitative studies, eight mixed-methods studies, and eight systematic reviews. Of the 38 studies, 39% reported or addressed 80%-100% of all items required, 43% reported on 50%-79% of the data required, and 18% reported below 50% of items required by their respective reporting tool. The main areas of poor rigour were: ethics (29%), statistical analysis methods (75%), discussion and management of study limitations (42%). For some mixed methods studies (29%), integration of quantitative and qualitative data was unclear. In some cases, these issues may merely reflect poor reporting. However in the absence of data, poor rigour was assumed. Overall, the quality of the studies was summed-up as moderate. The quality assessment summary is provided as Supplementary Table 4.

Authors either directly studied QI outcomes, reported additional QI outcomes and benefits, and or discussed QI goals and missed opportunities. A number of papers reported financial savings or had savings as a goal [77, 81‐88]. Gandjour & Lauterbach [89] noted that cost-saving was more likely when improving an over-use or misuse problem. For example, an article reported cost-reduction from malpractice suits [74]. Financial benefits through QI were mostly internal to organisations, and a small number involved societies and healthcare funders [73, 75].

There was a shared view that quality and patient safety should be more central to QI and investment goals than financial outcomes [72, 88, 90‐95]. This view had not changed over time. Thus, QI goals were primarily improving patient outcomes through systems, structural, process, and behavioural improvements. This enabled improved efficiency and productivity. Efficiency and productivity enabled managers’ abilities to manage, minimise, or reduce costs, and eventually save costs [73, 94, 96‐98]. Systems efficiency helped improve staff efficiency, effectiveness, productivity, and experience, which benefited patients [84, 99, 100]. Improved systems enabled improved organisational capacity, capability, and resilience [93, 101‐106].

Most authors highlighted that good quality and patient safety relied upon good staff outcomes and leadership. A few studies focused on some of these specific areas. Examples include Mery et al. [71] who studied QI programmes as an organisational capability and capacity development tool. Hatcher [83] studied QI as a staff safety promotion tool, Lavoie-Tremblay et al. [99] evaluated QI as a tool for team effectiveness. Furukawa et al. [107] and Heitmiller et al. [84] focused QI towards environment sustainability. MacVane [96] saw QI as a governance tool. Williams et al. [100] focused on both staff and patient outcomes. QI was also used to operationalise organisations’ strategies [93, 108]. Staines et al. [108] found that a positive QI reputation allowed recruitment of a suitable CEO.

There was a general recognition that QI does not always achieve its intended goals. Additionally, some QI strategies were more successful than others [80]. Particularly, some literature reviews and empirical studies reported variable, mixed, or inconclusive results [86, 109‐115], even a decline in quality [99]. A few articles discussed negative unintended outcomes [81, 100, 104, 110, 112, 114, 116‐119]. de la Perrelle et al. [77] noted this lack of reporting of negative findings in their review. They suspected this to be due to publication bias. Rationales for not achieving goals were given as implementation difficulties related to contextual and behavioural challenges [78, 114, 120, 121].

Some authors noted that overall benefits accrued over time during phases of a programme’s implementation process [80, 122]. Morganti et al. [123] noted different measures of QI success but suggested that spread of a programme was a measure of lasting success. Sustainability of outcomes was therefore also seen as an important achievement by most authors. This was supported by some of the literature which also indicated that successful QI built legacies mainly through spreading, embedding, and sustaining improvements [78, 93, 101‐106]. This finding was confirmed by impact studies, extensive QI programme evaluations and discussions of overall QI impacts [69, 85, 87, 93, 103‐106, 108, 115, 116, 119, 121, 124‐126]. These literatures elaborated on QI goals, failures, and successes, as well as the lessons learnt. Authors suggested that lessons and cultural changes as a result of QI were essential to meeting patient safety needs [93, 109]. Authors highlighted that ultimately, QI benefited a wide range of stakeholders at different levels in different ways.

Based on the findings, we compiled data into four overarching themes (Table 3). These themes aligned with our ROI preliminary framework; however, adjustments were made to reflect the findings. Organisational capacity and capability was renamed organisational development to acknowledge the broader organisational outcomes. This included all the outcomes that develop and improve organisations’ ability to fulfil their duties. Resilience and QI legacy were additional sub-themes under organisational development. External relations was renamed external outcomes to reflect the broad outcomes beyond relationships with regulators, communities, and other organisations. External outcomes were extended to include collaboration, societal and environmental outcomes, and incentives. Incentives included accreditation, awards, ranking, competitiveness, influence, power, and financial rewards.

Negative unintended outcomes include any negative impact resulting from a QI programme. These were external imposition, top-down distortions, duplication, high resource demands, loss of revenue, and loss of buy-in. Authors reported that at times external or managerial agendas were superimposed over other QI goals [108, 116, 127, 128]. At times this caused duplication of processes (e.g., data collection) and or increased demand on already stretched services. In addition, successful QI can cause loss of funding as services become absolute [108]. Eventually different negative outcomes may cause staff or leaders to disengage from current or future QI.

Positive unintended outcomes were difficult to delineate as often programmes were geared towards patient outcomes but impacted other parts of an organisation in the process. However, as improvement strategies involved changing systems and human behaviours, improvement of these aspects must be intended. We therefore had this sub-theme only include new innovations and opportunities. The final overarching themes were named 1) organisational performance (two sub-themes), 2) organisational development (12 sub-themes), 3) external outcomes (five sub-themes), 4) unintended outcomes (two sub-themes).

Based on the themes, we updated our ROI preliminary conceptual framework to map the four overarching themes that represent QI-ROI (Fig. 3). The beneficial outcomes are presented under the headings “gains, benefits, returns”, whilst negative outcomes are presented as “losses, costs, investments”. These concepts are technically defined differently. They are used together here to denote their co-existence within QI programmes. For example, loss of revenue is a potential investment lost, high resource demands may require investment or incur costs, duplication is inefficient and costly, loss of buy-in is a costly setback. All will raise money spent or lost if not well managed or avoided. They may also affect organisational performance and development, as well as stakeholder engagement in future programmes. Thus, impacts are both monetary and non-monetary.

Authors also saw investments as both in monetary and non-monetary forms. These were viewed as both equally essential for patient safety and quality. Some of these investments were part of ongoing organisational strategies. Investments included staff time, recruitment and retention costs, training costs, patient engagement costs [68, 69, 77, 95, 108, 113, 114, 116]. Some investments depended on the goodwill of the staff and patients and were seen as priceless [119]. Staines et al. [108] referred to two types of investments: “hard” infrastructure (e.g., technology) and “soft” infrastructure (e.g., awareness, commitment, and culture).

The literature also noted that QI outcomes are interlinked and interrelated, and as such QI-ROI may not be readily observable. Deducing ROI may require studying “cause-and-effect chains” [92] (p. 2) or an ROI chain; the link between events from a given investment to a given outcome. Sibthorpe et al. [113] saw this as important for understanding QI impacts and attracting QI investment. This can be done by tracking inputs, processes, outputs, and outcomes as much as possible throughout a programme. By doing this, the integrity of the ROI chain may be assessed by identifying areas where QI-ROI is created, lost, or influenced. This may then help maximise QI-ROI. However, tracking this chain in complex contexts may be a challenge.

In complex systems, programme inputs, processes, outputs are not a once-only event, occurring only at initial implementation. Outcomes of earlier inputs, outputs, and processes become inputs in the next phase and so forth until the final impact is achieved (end-ROI). It may therefore be helpful to recognise and celebrate earlier achievements [33, 97]. Further, before a final impact is realised, a programme may act and interact with several variables. Due to this complexity, the linkages may resemble a web rather than a chain. The literature attested to the fact that QI impacts are unpredictable, and difficult to measure [33, 113, 119]. QI inputs may or may not be converted into active QI ingredients that will affect organisational change and improvement [80]. For example, if one of the strategies is to train staff; do they actually learn what is needed? The answer would depend on several internal and external determining factors [78, 79, 114, 120, 121]. Such factors may force adaptations, influence fidelity to strategies, sustainability, and decisions to proceed, de-implement or disinvest.

The ROI chain is illustrated here in Figs. 4 and 5. Figure 4 demonstrates that the overall ROI results from changes in processes, structures, and systems. These may be visible through behavioural (human and systems), and technological improvements, before final impact and ROI can be detected. Two-tier order mechanisms are alluded to here; the first order mechanisms operationalise QI strategies and provide non-monetary ROI, whilst the second order mechanisms convert QI efforts into financial returns. A first order mechanisms may be for example increased staff proficiency leading staff development, whilst a second order may be improved productivity due to increased proficiency. Productivity may then help save costs.

In summary, different investments are made towards a QI programme and a change is propagated through changing and improving processes, behaviours, systems, and structures. Technical (e.g., skills) and social (e.g., culture) changes and improvements may be achieved. These changes and improvements can then lead to improved efficiency and productivity. Efficiency and productivity can then improve cost-management. Better cost-management and control can then lead to cost-reduction, cost-minimisation, cost-avoidance, cost-containment, and cost-saving. All these are outputs, immediate and intermediate outcomes that become mechanisms through which monetary ROI is achieved. Before then, the outputs present as non-monetary returns-on-investments either as enabled abilities (e.g., cost-management, cost-reduction, cost-minimisation, cost-avoidance, cost-containment), outputs or intermediate outcomes (e.g., improved behaviour, productivity, efficiency).

Non-monetary ROI can also be achieved through organisational development e.g., staff development and collaboration. Organisational development is the basis for safe healthcare systems and may lead to cost-saving, and hard cash ROI. Improvements in staff and process outcomes may improve culture, which may also improve patient and financial outcomes. Improvements in patient outcomes may lead to further benefits (e.g., incentives), and become an organisation’s legacy (culture, capacity and capabilities). This can help an organisation become more resilient and sustainable. QI culture and QI legacies are the basis from which future organisational development as well as patient and financial outcomes can be achieved.

Altogether, the QI outcomes contribute to higher goals such as organisational learning, transformation, financial stability, value-based healthcare, and high reliability [101, 102, 105, 116]. Although intended goals and short-term outcomes may be achieved earlier, long-term sustainable impacts depend on successful implementation, embedding a QI safety culture, and developing legacies that support future improvement efforts. Whatever the end-outcome, lessons may be learnt, research, innovation and development may ensue, capacities and capabilities may improve. As Banke-Thomas et al. [68] stated, “ The application of (S)ROI … could be used to inform policy and practice such that the most cost-beneficial interventions are implemented to solve existing (public health) challenges” (p.10).

Figure 5 illustrates the updated QI-ROI conceptual framework in a phased format. This figure represents the current conceptualisation of QI-ROI based on our analysis of the healthcare QI evaluation literature. The processes described here are more complex but have been simplified for clarity. The figure contains the ROI-like concepts from our first study (e.g., efficiency, productivity, effectiveness, cost saving). These concepts are seen here as building blocks of financial ROI. However, some of these also form part of improvements in other organisational performance and developmental goals. Such improvements can be seen as non-monetary ROI which includes improved abilities, development, and overall improved outputs and outcomes. Together, these are the building blocks of the QI-ROI concept as indicated by the literature.