The sustainability of Lean in pediatric healthcare: a realist review

The review followed established realist guidance [21‐24]. Realist synthesis are useful to make program theories explicit by developing testable hypotheses on the mechanisms by which complex interventions are successful or not, and how certain contexts can trigger different mechanisms that in turn generate different outcomes [25, 26]. Interventions such as Lean can have many potential change processes and outcomes that are non-linear and multifaceted in nature and dependent on social context [27]. A realist approach offers methodological strengths to unpack the “black box” of interventions in comparison to traditional synthesis approaches [28]. From a realist standpoint, to understand the effectiveness of an intervention one needs to develop an understanding of the mechanisms (M) and the contexts that affect whether or not they operate (C) in order to generate an outcome (O) (C + M = O) [21]. The terminology used in the review is outlined in Additional file 1.

A program theory can be used to frame and evaluate how, for whom, why, and under what contexts complex interventions work or not [29]. Prior to this review, we developed our initial program theory on Lean sustainability in healthcare using a multifaceted approach: (1) iterative brainstorming sessions within the review team; (2) realist methodological expertise (see acknowledgements), a scoping search of literature on Lean, QI, and sustainability; (3) use of a Lean operational definition [18]; (4) use of substantive theory (Normalization Process Theory (NPT)) [30, 31] and a sustainability model (National Health Service (NHS) Institute for Innovation and Improvement Sustainability Model (SM)) [32, 33]; and (5) use of the NHS SM definition for sustainability [32], and a comprehensive definition of sustainability [34].

The NHS SM provided process, staff, and organization contextual factors that potentially explain and increase the likelihood of sustainability and continuous improvement [32], while NPT offered insights into the potential mechanisms that promote or inhibit the embedding of complex interventions into routine everyday practice and the likelihood of sustainability [30, 31]. These underpin each of the initial CMO hypotheses from our initial program theory.

Using the context (C) + mechanism (M) = outcome (O) heuristic, our initial program theorizing comprised of mapping the terrain of contexts, mechanisms, and outcomes. Subsequent to that, seven initial CMO hypotheses were formulated; these hypotheses reflect our initial program theory. It became evident that unpacking the causal pathways in implementation are a necessary precursor to theorizing and testing sustainability CMO’s. We hypothesized that outcomes at implementation (e.g., shared understanding, improved team work), the resources provided during implementation (e.g., Lean training), and the scale of implementation (micro, meso or macro), shapes the contexts (e.g., value congruency, high performing teams), mechanisms (e.g., sense-making, staff engagement, empowerment, accountability), and outcomes for the sustainability of Lean efforts. This concept known as the “ripple-effect” is premised on the idea that Lean is a series of “events in the history of a system, leading to the evolution of new structures of interaction and new shared meanings” p. 267 [35]. Our initial program theory depicts that Lean becomes a complex intervention when implemented to a complex adaptive system (healthcare) across multiple levels of a system (micro, meso, and macro) to multiple stakeholders (organizational leaders, clinical leaders, and front-line staff). Our seven initial CMO hypotheses were categorized according to these elements.

Consistent with a realist approach, our search strategy was multi-pronged, iterative, and purposeful in nature. We developed search strategies for the following databases which were searched from date of inception until June 2016: Medline (OVID), EMBASE (OVID), CINAHL (Ebsco), and Dissertation Abstracts (ProQuest). The search strategy consisted largely of keywords since the databases searched did not contain controlled vocabulary for Lean management concepts. Methodological filters were not used, since the goal of a realist review is to identify both qualitative and quantitative reports. We also searched for the term “pediatric” and synonyms in an EndNote database of 5000 references compiled from searches for a systematic review on lean management in healthcare [36].

We conducted reference list searches for each included source. Our gray literature search was purposeful and multi-pronged, undertaken on the following organizational web sites: Institute for Healthcare Improvement (http://​www.​ihi.​org) and the Agency for Health Care Research and Quality (AHRQ) (http://​www.​ahrq.​gov); and Google. All web sites were searched for the terms Lean, healthcare and healthcare synonyms; we scanned the first three pages of Google results. We also contacted three organizations known to implement Lean in healthcare settings: Saskatoon Children’s Hospital, Cincinnati Children’s hospital, and Virginia Mason hospital. Our search strategy is provided in Additional File 2.

Following a two-stage process, two reviewers (RF and SW) independently screened the titles and abstracts of all records (stage 1), and then independently screened the full text (stage 2) of any document that made it through stage 1. For inclusion, documents had to discuss Lean implementation (exclusively or blended, that is Lean and another QI approach) as defined by our operational definition [18], with a pediatric focus (exclusively or blended, that is pediatric and non-pediatric foci in the same study), and sustainability outcomes as defined by NHS SM [33] and Moore et al. [34]. For sustainability outcomes, documents had to provide: (a) measures of sustainability and/or (b) a critique or review of ideas related to how Lean is or is not sustained in pediatric healthcare, and/or (c) stakeholders opinions or accounts of how Lean is or is not sustained in pediatric healthcare. Documents were not excluded based on methodological quality. Due to feasibility reasons, we only included documents in the English language. For stage 1 screening, we applied the inclusion criteria to the titles and abstracts of our search results; all yes and unsure documents moved forward to stage 2 screening, which consisted of full-text screening based upon the inclusion criteria.

Adopted from Wozney et al. [37], we assessed each document in terms of the richness and relevance of content to context, mechanism, and outcomes. Each document was rated as low/no contribution (no or little information), medium contribution (some information), and high contribution (well-described information). We also assessed relevance by objective (empirical) versus subjective (anecdotal) evidence. Empirical evidence was determined as research based data (e.g., qualitative or quantitative findings), primarily found in the results section of included documents. A document was classified as providing anecdotal evidence when no empirical evidence supported the author’s interpretations, typically found in the discussion sections of the included documents. We used and adapted the Mixed-Methods Appraisal Tool (MMAT) [38] to assess methodological quality of the included primary studies, resulting in a methodological rating of 0, 25, 50, 75, and 100% (with 100% being the highest quality). We adapted the MMAT for multi-method studies by assessing each segment of a multi-method study separately and then selecting the lowest quality rating. Documents were not excluded based on MMAT score; the purpose was to examine and gain insight into the rigor of existing research in this field. Documents were also not excluded based on anecdotal evidence—our main concern was finding information with strong CMO contribution. This information was logged during data extraction.

Using a standardized data extraction form on Microsoft excel, we extracted descriptive information from each document (e.g., QI initiative purpose, stakeholder type, setting, theory, and level of change). We applied Colquhoun et al. [39] three conditions for classification of a theoretical basis [39], in order to understand to what extent the sustainability evaluation in each document was guided by theory. We also extracted intervention and contextual factors, mechanisms, outcomes, and any evidence or information related to our initial program theory and CMO hypotheses on the sustainability of Lean efforts. To promote consistency, a coding dictionary was developed and used during data extraction. Two authors (RF and SW) conducted and cross-checked data extraction decisions for each of the included documents. No discrepancies arose during this process. The two authors that conducted data extraction met at after independently completing two extractions and met twice weekly during data extraction.

Data analysis and synthesis were iterative, using a multi-stepped approach to identify and organize information from the included documents. The purpose was to understand what about the contexts where Lean implementation occurred, triggered certain responses (mechanisms) by stakeholders that contributed to the sustainability or otherwise of Lean efforts (outcomes).

Drawing from abductive and retroductive analysis [40, 41], RF examined each document for evidence that supported, refuted, or refined our initial CMO hypotheses. This form of synthesis required the researcher to move between theory and data, analyzing data that were not in the initial program theory (abduction) and moving between theory and observable data (retroduction). This analytical approach enabled the formation of new ideas beyond the initial theoretical basis of our initial program theory and CMO hypotheses and required the researcher to bring assumptions on what factors contribute to Lean sustainability and a priori knowledge on Lean implementation in one health system to question the conditions for a theoretical basis. Retroduction involved inductive and deductive logic where the research team theorized what causal powers may be at play to produce observed patterns in the data. This involved using the teams’ insights and experiences on Lean in healthcare and implementation science. Abductive reasoning involved theorizing the best possible explanations of observed outcomes, thinking about the potential mechanisms and contexts that produced certain outcomes, where data was missing. Data gathered from the included documents that were not explained by our initial CMO hypotheses were used to refine our initial program theory. This process was tracked through reflective notes; integration of NPT and NHS were applicable and through regular team discussion.

Of the 11 included documents, six were published primary research studies [42, 43, 45, 47‐49], two were published quality improvement case studies [44, 46], and three were unpublished quality improvement case reports [50‐52] found from our citation searching [51] and gray literature searching [50, 52] (Table 1). Of the 11 documents, seven used Lean exclusively [44, 45, 48‐52]; two used Lean and Six Sigma [42, 43]; one used a combination of Lean, Six Sigma, and the Institute for Healthcare Improvement (IHI) Model for Improvement [46]; and another used Lean with “other” QI classic methods [47]. Improvements were targeted at the meso (e.g., unit or organization level) (n = 8) [42‐49] and macro (e.g., policy, system) (n = 3) [50‐52] levels of the healthcare system. No improvement targeted the individual, micro level. Documents focused on a variety of problems, clinical (n = 1) [42], process (n = 2) [43, 48], clinical and process (n = 1) [46], or process and system problems (n = 7) [44, 45, 47, 49‐52].

All documents used a multi-disciplinary team approach to lead implementation [42‐52], six of which also included a physician lead within the multi-disciplinary team [42, 43, 47‐49, 51]. Many reported organizational leadership involvement (n = 8) [44‐46, 48‐52] and/or clinical leadership involvement (n = 7) [43, 45‐47, 50‐52]. One reported patient involvement [43]. There was variation between the use of internal QI support coaches [44, 45, 47] versus external Lean experts or consultants [48‐52].

The methodological quality (Table 2) of the six primary research studies varied, three scored 75% [42, 48, 49], two scored 25% [45, 47], and one had no score, 0% [43] on the MMAT. Relevance (CMO contribution) also varied across the primary research studies. Two studies with an MMAT of 75% had high contribution [48, 49]; however, in contrast, the third study with an MMAT of 75% had low contribution [42]. One study that scored 25% on the MMAT had a high contribution [45]. The remaining two studies with MMAT scores of 25% and 0% had medium contribution [43, 47]. Of the quality improvement case reports (n = 5), four had medium contribution [44, 46, 50, 51] and one had high contribution [52].

There was variation as to how sustainability was defined and measured. For example, six documents referred to sustainability as a change that had lasted over a certain period of time ranging from 6 months to 4 years [42, 43, 46‐49]. Outcome measurements were not widely reported, and the description on sustainability was poor, primarily consisting of descriptive and experiential accounts (e.g., “Long-term sustainability requires staff engagement, charismatic champions and leaders, and a culture that sustains the change despite staff turnover”) [47]. Of the six primary research studies, three reported positive sustainability outcomes [42, 43, 49], and three reported mixed (positive and negative) sustainability outcomes [45, 47, 48].

All the primary research studies reported clinical, process, and performance outcomes as the proxy measure for sustainability [42, 43, 45, 47‐49]. For example, one study reported, “process changes were implemented, resulting in an increased mean documented communication rate from 13% pre intervention to 65% post intervention that was sustained for more than 2 years (P<.001)” [43]. One of the studies that reported mixed outcomes stated that, “we lack hard data on these measurable outcomes of their long-term sustainability” [45]. The same study reported some negative outcomes that the implementation of Lean had short-term gains and failed to achieve more widespread and sustained improvements; these data was gathered through qualitative interviews [45]. Outcomes reported from the remaining included documents were based on subjective data from descriptive QI reports [44, 46], or case study reports that had collected primary objective data but presented summary findings [50‐52].

By using a realist approach, we have been the first to uncover some of the contexts and mechanisms that influence the sustainability of Lean efforts in pediatric healthcare. Three substantial issues have emerged and have supported our initial program theory.

First, the degree of success or failure in the sustainment of Lean efforts relies on the ways in which people “make sense” of Lean, align their values, and the values of the organization to the values of Lean. Sense-making (the process through which people assigns meaning to experience), staff engagement, and empowerment were identified as core mechanisms to the sustainability or non-sustainability of Lean efforts. The activation of these mechanisms was facilitated or hindered by Lean resources, such as Lean education [42, 46, 47, 50], Lean training [43, 45, 49, 50, 52], external Lean consultants [45, 48‐52], internal QI support coaches [44, 45, 47], and knowledge translation strategies [42, 43, 47]. The degree to which these mechanisms were activated or not was influenced by certain conditions or contextual factors, such as external pressures to use Lean [43, 45, 46], a culture shift prior to implementation (organizational readiness) [44], an existing QI structure [43, 44] staff turnover [45, 47, 48], the silo nature of healthcare [49], the complexity of care processes [48], the fit between Lean and local context [47, 48], and other competing needs or demands [47]. It is important to note that none of the contextual factors identified were unique to pediatric contexts. The relationship between these contexts and mechanisms led to multiple heterogonous outcomes on the sustainability of Lean efforts.

Second, outcomes from Lean implementations shifted to become the contexts for sustainability. That is, in some cases, there was a “ripple-effect” where outcomes from implementation served as facilitating or hindering contexts that triggered mechanisms for the sustainment or otherwise of Lean efforts. For example, “sense-making” and value congruency are outcomes at implementation that serve as contexts in sustainability which then trigger staff engagement and empowerment to lead and sustain Lean efforts. Hence, the efforts taken and approaches used at implementation are critical to the success of sustaining Lean efforts.

Implementation approaches and processes contributed to the sustainability or non-sustainability of Lean efforts across our included documents. The use of multi-disciplinary-led teams [42‐52], patient involvement [43], physician leads [42, 43, 47‐49, 51], organizational leadership involvement [44‐46, 48‐52], and/or clinical leadership involvement [43, 45‐47, 50‐52] contributed to the sustainability of Lean efforts. For example, large-scale transformation was reported to have greater likelihood of sustainability than small-scale incremental QI improvements [48], with top-down leadership commitment [44]. However, it was noted in another document that a top-down implementation approach was less well received and sustained [51]. These contradictory findings demonstrate that top-down approach was equivocal in terms of sustainability. External Lean consultants were also reported as a facilitator to sustainability [48].

Finally, Lean is complex in the context of healthcare; its implementation and sustainability are complex as it occurs across multiple levels of organizations [53] within complex adaptive systems [54] with multiple stakeholders. Contexts, mechanisms, and outcomes at one layer of a health system (e.g., organizational leadership) had an impact on the contexts, mechanisms, and outcomes at another level (e.g., clinical leadership), demonstrating the need for a theoretical complexity lens to the implementation and sustainability of Lean in healthcare.

The concept of the “ripple-effect” enabled a better understanding of the causal relationship between Lean implementation and sustainability, and how processes from implementation to sustainability occur. Our review findings demonstrated instances where outcomes of Lean implementation served as facilitating contexts for subsequent stages of sustainability (C¹M¹O¹– > C²M²O²) [55] as illustrated in Fig. 2. This concept of a “ripple-effect” was substantiated in CMO hypothesis 5 and 6, where staff engagement was an outcome at implementation, in turn a context for sustainability, which triggered mechanisms of staff empowerment and outcomes of sustained Lean efforts.

As demonstrated through this review, sense-making about Lean may occur at implementation but is also crucial to sustainability, if the philosophy, principles, and activities of Lean do not make sense to those tasked with implementing or using Lean, then it is unlikely that Lean efforts will be adopted and subsequently sustained. This finding supports and substantiates CMO hypotheses 1 and 4 from our initial program theory (Table 3.). The more people value the change being implemented as important or worthwhile, the more likely that they will engage in the implementation efforts [53]. Some empirical evidence has shown that when staff and managers did not understand Lean, this had a negative meaning throughout the organization [54]. Sense-making is associated with productive self-organization [57], a process whereby natural order forms irrespective of the interventions intentions [58]. Creating and maintaining an institutional culture underpinned by shared vision and values are central to Lean success [59]. Supportive culture with leadership engagement and team involvement was an identified facilitator to maintaining Lean efforts [13], demonstrating that engagement must occur across different layers of the organization.

Another challenge with implementing and subsequently sustaining Lean efforts as intended was the dichotomy between the vision and values of Lean with those of the organization and/or key stakeholders within an organization. Without alignment of vision and values from senior leadership to the front-line teams, Lean may be reinterpreted and reshaped to ensure that it fits with the visions and values of the multiple stakeholders, which may also vary, making Lean efforts a highly contested process [10]. Under such conditions with potential conflict and disagreement, Lean efforts are unlikely to be maintained as originally intended. For example, Kim et al. [60] reported the misunderstanding of Lean as a cost-cutting measure, created fear in staff and disengagement in Lean. Another study reported on the overuse of “Japanese” terminology for Lean principles and activities by external Lean consultants in healthcare that do not resonate with health professionals or a patient centered approach. The authors from this study suggest the need to appeal to the personal values and reasoning of the potential adopters [61]. Another published paper on the “promise of Lean in health care” reported that Lean needs to be seen as a mindset that governs how one looks at the business or process. Human skills such as communication, problem solving, teamwork, and strong leadership are vital for Lean implementation success. It is resolute that organizational culture and poor change management are predominant reasons for Lean failures [62].

Active multi-disciplinary staff involvement in leading Lean efforts was recognized as critical to sustainability. This finding is supported by Lean literature in other healthcare contexts [6, 7, 13, 54]. Leadership support was also found in our review to be critical to sustaining Lean efforts, a finding that is shared across other published Lean healthcare literature [4, 11, 59, 62, 63]. Despite the recognition of the importance of these concepts to the embedding, normalization or sustainability of Lean efforts, there is a paucity of rigorous literature that explores or tests these concepts in Lean in healthcare.

We hypothesized that outcomes at implementation (e.g., shared understanding, collaborative improved team work collaboration), contextual factors at implementation (e.g., external Lean consultants versus internal QI coaches), and the scale of implementation (micro, meso or macro), shapes the contexts (e.g., value congruency, high performing teams), mechanisms (e.g., sense-making, staff engagement, empowerment, accountability), and outcomes for the sustainability of Lean efforts. This supports the argument by Pluye and colleagues [64] that program implementation and sustainability are not distinct processes but are connected to each other. However, other existing evidence has shown that the conditions that facilitate implementation may diminish overtime [65, 66]; hence, the conditions for sustainability are also susceptible to losing presence and influence, leading to discontinuation.

A limitation of this review is that we only sourced 11 relevant documents. This paucity of literature demonstrates a knowledge gap and weak evidence base on Lean sustainability in pediatric healthcare. The quality of the literature that served the basis of this review must also be acknowledged as a limitation, making unpacking CMO hypotheses challenging. There is a clear need for more rigorous evaluative studies on Lean sustainability in healthcare. We experienced positive reports about Lean, however, not always based on rigorous research design and method. Theorizing during this process came from expertise on Lean in healthcare, healthcare improvement experts and implementation scientists. Data from this review do support some CMO hypotheses from our initial program theory. A strength is that the results of this realist review are being tested in a subsequent realist evaluation on the sustainability of Lean efforts across four pediatric acute care units at one hospital in Saskatoon, Saskatchewan, Canada. Our realist evaluation takes place in a health system-wide transformation of Lean known as “the Largest Lean transformation in the world” [67]. This is an important next step to test our initial program theory and substantiated CMO hypotheses. A limitation of much of the research on Lean in healthcare and thus, this review, is the lack of reporting and measurement on sustainability.