Simple rules for evidence translation in complex systems: A qualitative study

The study was conducted using an analytical auto-ethnography and grounded theory approach (Fig. 1). An analytical auto-ethnographic approach was adopted reflecting that the authors of this paper were full members of the research setting (conducting ethnography of ‘our own people’ as members of the "core team" (Fig. 1)), visible as such a member in published texts, and committed to developing theoretical understandings of broader social phenomena [41].

Empirical data was collected through participant observation and document analysis of the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC), Northwest London (NWL) programme (UK), and 22 evidence translation projects (Additional file 1). This allowed direct access to and observation of actions, events, scenes and people in real-time over a 5-year period, with opportunities to follow-up on emergent patterns and problems. Concurrently, extensive literature was reviewed using a snowballing approach to identify frameworks, models, systematic reviews and other relevant literature (further details on data collection and literature review can be found in Additional file 2).

A grounded theory approach guided the data collection and analysis [42, 43]. Data was analysed using open, axial and selective coding, in parallel with theoretical sampling, to explore emergent categories and themes over time. This iterative analysis led to a process of ‘abduction’ to make sense of material that did not ‘fit’ into pre-established categories (including published frameworks and theories), thereby reconceptualising the challenge of evidence translation and improvement into a new substantive theory (to provide explanations and predictions related to the specific context of study) and conceptual framework (indicating how aspects of the theory are connected to each other). Further details are provided in Additional files 2 and 3.

This exploratory research approach was chosen to ensure that the resulting findings were empirically informed and theoretically grounded in the practical reality of evidence translation and improvement in real world (complex) settings. We chose not to build exclusively on any existing theories as no single existing framework fit well with our experiences. Whilst several fields of study were relevant, no single frameworks brought together concepts from different fields, including knowledge translation, implementation, improvement and complexity.

Results from the grounded theory analysis were interpreted through complex systems thinking [26, 28, 34, 35]. Emphasis was placed on developing a series of ‘simple rules’, which were identified through establishing relationships between challenges experienced by the project teams, and the actions and strategies that, if taken, had a positive effect on project progress and outcomes or, if they were absent or overlooked, were observed to have a detrimental impact.

The NIHR established the CLAHRC programme in England to accelerate the translation of evidence into practice for the benefit of patients. Thirteen regional CLAHRC programmes were funded, each led by academic and healthcare partnerships and with autonomy to decide how they would approach ‘closing’ the translational gap [44‐46].

The CLAHRC NWL approach brought together healthcare staff, including clinical, managerial and support staff (hereafter referred to as ‘staff’) with patients, carers, family members and the wider community (hereafter, ‘patients’) and academic partners from a diverse range of disciplines (hereafter, ‘academics’) into project teams of 5–15 people to translate evidence into practice in their local micro-systems. Project teams used a suite of quality improvement tools and methods, including the model for improvement, action-effect diagrams and plan-do-study-act cycles, process mapping, statistical process control, stakeholder engagement, and patient and public involvement combined with iterative evaluation, to guide and support the implementation process [47‐51].

During the first 5 years of the CLAHRC NWL (2008–2013), 22 diverse topics considered of clinical importance were explored with 55 teams over four rounds of 18-month projects (Fig. 1) in various settings (acute, community, primary care, mental health, etc.) (Additional file 1). All projects had a common goal of translating existing evidence into practice to achieve improvements in quality of care provision, with the aspiration of delivering corresponding improvements in patient outcomes. Two detailed case study examples are presented in the results section (Boxes 1 and 2).

This paper represents a consolidation of cross-project learning from the programme and peer-reviewed literature. Existing publications relating to evaluation of individual projects, cross-project analysis, use of quality improvement approaches, and external programme evaluations are listed in Additional file 1.

The theory of SHIFT-Evidence can be summarised as follows: to achieve successful improvements from evidence translation in healthcare, it is necessary to ‘act scientifically and pragmatically’ whilst ‘embracing the complexity’ of the setting in which change takes place and ‘engaging and empowering’ those responsible for and affected by the change.

SHIFT-Evidence reflects the nature of work and breadth of effort required to translate evidence into complex systems. The findings revealed that attention and effort was often drawn away from the original project focus in directions that were not anticipated in advance, such as dependent issues relating to people, processes or structures, or to resolve existing problems with ‘usual care’. We established that failure to resolve these issues compromised the success of an intervention and diminished the ability to draw useful conclusions about the effectiveness of any intervention in a real-world setting. As such, the SHIFT-Evidence framework is conceptually based on the premise that the implementation of evidence-based interventions is not necessarily sufficient to achieve improvements in care, and that it is not possible to fully anticipate what changes will be required in any individual setting. In short, evidence translation and wider systems improvement are inextricably linked within complex systems.

The accumulating data about the ‘daily realities’ of evidence translation and improvement required us to reconceptualise our understanding of the problem, and associated potential solutions. Our focus moved from evidence-based medicine and interventions, to focusing on the complexity of the systems within which we hoped to intervene. As such, literature relating to complex systems thinking grew in importance over time to become the primary lens by which we were able to make sense of our experiences (further details on this process of reconceptualisation are provided in Additional file 2).

Reflecting this reconceptualisation, ‘act scientifically and pragmatically’ was identified as the core category for selective coding. It was chosen to reflect the interaction between our starting world view (the need to use scientific evidence) and our core learning (the need to understand and respond to the constraints and opportunities of the local system). Our analysis indicated the tension between these perspectives, and also the opportunity for increased synergy between them, as follows:

Two further important key categories were identified, namely ‘embrace complexity’ and ‘engage and empower’. These three high level conceptual categories are termed strategic principles, reflecting the guidance on how to conduct and research evidence translation and improvement in complex systems. These principles are underpinned by 12 associated ‘simple rules’, which describe the actions required to achieve each strategic principle.

The three strategic principles and 12 ‘simple rules’ are the following:

Act scientifically and pragmatically: Knowledge of existing evidence needs to be combined with knowledge of the unique initial conditions of a system. Interventions need to adapt as the complex system responds and learning emerges about unpredictable effects. This strategic principle reflects the high level stages of an improvement initiative through the four simple rules:

Embrace complexity: Evidence-based interventions only work if related practices and processes of care within the complex system are functional. Evidence-translation efforts need to identify and address existing problems with usual care, recognising that this typically includes a range of interdependent parts of the system. This principle emphasises the need to investigate and understand the uniqueness of each local system and respond to complexity from the micro- to macro-system as reflected by the four rules:

Engage and empower: Evidence translation and system navigation requires commitment and insights from staff and patients with experience of the local system. Changes need to align with their motivations and concerns. The four rules reflect factors that influence engagement at an individual and team level through to supporting infrastructure and organisational level:

Relationship between SHIFT-Evidence principles: The process of evidence translation and improvement, as represented in SHIFT-Evidence, is intended to be a progressive iterative process. The ‘simple rules’ provide a conceptual framework to guide practice and research in complex systems, responding to emergent challenges and capturing generative learning (Fig. 2). In practice, feedback loops exist between each of the rules as learning emerges about the changes required and effectiveness of interventions. Few improvement initiatives follow a smooth linear pattern.

Our hypothesis is that all SHIFT-Evidence strategic principles and ‘simple rules’ are necessary to achieve successful and sustained improvements in care, and are interdependent. For example, ‘active engagement’ of healthcare professionals and patients is necessary to fully ‘understand practices and processes of care’. Equally, ‘active engagement’ of staff may reveal that their priorities do not align with current ‘strategic, political and financial’ incentives, and vice versa. Our hypothesis implies that such tensions, if unresolved, will negatively impact success.

Two of the 22 CLAHRC NWL project narratives are presented as detailed examples to illustrate the practical reality of evidence translation and improvement (Box 1 and 2). Both demonstrated measurable success against their original objectives, although each encountered unexpected obstacles. This is followed by presentation of the 12 simple rules, describing how the simple rules relate to the project narratives and substantive theory (Tables 1, 2 and 3), and reflecting on insights provided by complex systems thinking.

The project narrative in Box 1 outlines the challenges to embedding evidence-based practices and achieving improvements in care quality for patients with community-acquired pneumonia (CAP).

The second project (Box 2) illustrates the complexity of healthcare systems and how this was experienced by a clinical team attempting to improve medicines management (MM) for patients following discharge from hospital.

The strategic principle ‘act scientifically and pragmatically’ demonstrates that knowledge of existing evidence is only one part of the effort required to achieve sustainable improvements in care in complex systems.

The strategic principle ‘embrace complexity’ demonstrates that evidence-based interventions only work if supporting or dependent practices and processes of care are working sufficiently well.

The strategic principle ‘engage and empower’ demonstrates that evidence translation and system navigation requires commitment and insights from staff and patients with experience of the local care settings, and changes to a complex system need to align with their motivations and concerns.

Our findings demonstrate that intervening in complex systems requires an understanding of the unique initial conditions (problems, opportunities, people, practices and patterns) in each local setting that are influenced by historical path dependencies. Scientific evidence about which interventions to use needs to be balanced with local system requirements, rather than assuming the starting point will be the same in each setting, and a commitment to continual improvement is needed to allow for the fact that systems evolve and adapt over time. This temporal dimension of systems thinking is reflected in the SHIFT-Evidence framework by the strategic principle ‘act scientifically and pragmatically’.

These findings challenge current conventions of seeing implementation as a one-off or time-limited activity, and build on Hawe et al.’s [61] proposal that interventions are ‘events in systems’. Further, a single pre-planned intervention, or set of interventions, is unlikely to be sufficient to achieve evidence implementation and improvements. Instead, multiple interventions are likely to be required; with the need emerging only as changes are implemented and system understanding grows. This builds on quality improvement approaches that promote iterative development over time [59, 78, 79], and organisational learning perspectives that value generative learning (e.g. double (and triple) loop learning) [73, 80].

We propose, in light of these findings, that terminology shifts from the use of the noun ‘intervention’ to the verb ‘intervening’. We believe that the concept of ‘intervening to achieve an improvement’ better reflects the iterative and negotiated process required to test multiple interventions whilst noticing and responding to local system requirements over an extended period of time (cf. Snowden’s probe-sense-response) [81].

Interventions cannot be considered in isolation from the system they are implemented into. The uptake and effective utilisation of any specific intervention is dependent on established practices and processes of care. These practices and processes of care cannot be assumed to be working well, and often additional interventions will be necessary to address related and systemic problems. Intervening in complex systems requires an understanding of these dynamic and fluctuating processes. Understanding of system dynamics and interconnectivity is represented by the strategic principle ‘embrace complexity’.

This challenges current conventions of seeing interventions as bounded and discrete, and anticipating that such interventions will be used by people working in a rational linear manner. Interventions are inherently dependent on the context that they are used in, and it cannot be assumed that dependent processes and practices are working well. This builds on literature from operations management and patient safety in valuing an understanding of ‘work as is’ as opposed to ‘work as imagined’ [82, 83]; people in complex systems are challenged with making decisions in real-world conditions, under high pressure, with constrained time and resources, whilst balancing multiple priorities [84].

The implications of systems evolving over time and their dynamic, interconnected nature are that capacity and capability needs to be built into the system to reflect, experiment and learn about intervening within the system over time. The strategic principle ‘engage and empower’ emphasises the critical role local system members play in identifying and solving local problems (although each person individually can only partially know or see the whole system), and the necessity for their willingness and motivation to adopt new ways of working.

This challenges current conventions of implementation activities being designed and conducted by people outside of the system, and draws attention to the unique insights provided by people within the local system (healthcare professionals, patients, managers) about how they self-organise and how they experience attempts to intervene. This builds on literature on co-production [65, 66, 85] and co-design [86] that emphasises the importance of engaging local stakeholders to solve problems that matter to them in their local setting and acknowledges the extensive work on understanding individual psychology of behaviour change and group dynamics [87, 88].

SHIFT-Evidence is the first empirically grounded framework for evidence translation in complex systems that can help make predictions and provide explanations about challenges and influences on success.

This research adds to the complexity science literature, initially proposed by Plsek and Greenhalgh [28], describing healthcare as a complex system. Building on this perspective, it makes a unique contribution in considering the implications of complexity for deliberate attempts to intervene and introduce evidence-based practices [36, 37]. Our study focused on micro-level initiatives, but findings resonate with existing literature on complexity in relation to macro-level initiatives (e.g. policy, systems design) [29, 34]. By providing insights into the ‘sharp end’ of practice, SHIFT-Evidence can provide insights to policymakers and system leaders as to how ‘top down’ initiatives might be received in complex systems.

This study also contributes to literature on evidence translation and implementation. It advances research by Craig et al. [89] on complex interventions, and by McCormack et al. [90], amongst others, which recognised the importance of context in the uptake of evidence-based practices, and May et al. [38], which expanded their theory of implementation to consider context as a complex adaptive system. SHIFT-Evidence builds on these views to consider the interaction between interventions, implementation strategies and context as inseparable and interacting components of a complex system. This view is reinforced by complexity thinking by resisting the temptation to isolate or reduce a system to its component parts, and instead to take interest in the interactions and patterns that emerge across the whole system.

For academics, SHIFT-Evidence provides an explanatory and predictive framework. The substantive theory explains the challenges encountered during evidence translation in complex systems and provides a rationale for strategies and actions to overcome them. The ‘simple rules’ provide testable hypotheses about the actions conducive to success that can be tested through future research. In demonstrating the magnitude of the challenge faced, SHIFT-Evidence makes clear the need for interdisciplinary enquiries to advance understanding and practice.

For patients, practitioners, managers, policymakers and academics involved with designing, conducting or evaluating healthcare improvement initiatives, SHIFT-Evidence provides a common framework to guide their work and ensure they are considering the breadth of the practical realities of evidence translation and improvement. The strategic principles (‘act scientifically and pragmatically’, ‘embrace complexity’ and ‘engage and empower’) were designed to be intuitive, accessible and memorable. A common framework that represents the complex and dynamic nature of improvement should help practitioners, academics and patients collaborate more effectively to increase the likelihood of success. If practitioners and patients can easily access practical knowledge, they may be more willing to contribute to the creation of new knowledge and to participate in the design, conduct and evaluation of future change experiments. If researchers understand how their work directly helps practitioners achieve improvements, and influence the lives of patients, they may be more likely to produce outputs which in turn increase practitioner receptivity and access to research settings.

For policymakers, funders and senior managers, SHIFT-Evidence emphasises the significant investment required at all stages of improvement efforts, including providing frontline practitioners with the time to step back from their day-to-day activities and the support needed to overcome barriers and obstacles to improvement. Such resource commitment is often seen as a luxury rather than essential. By using this structured approach to support funding and prioritisation, this may allow optimal investment of available resources and disinvestment in initiatives that add little value.

The quality of theory should be assessed by how useful it is in solving societal problems recognising that “the published word is not the final one, but only a pause in the never-ending process of generating theory” [91]. Therefore, rather than be seen as a finalised theory, or a perfect set of ‘simple rules’, the value of SHIFT-Evidence needs to be assessed through its usefulness in practice (and research), and should act as a catalyst for further improvement and refinement of the theory as predictions are tested.

The first limitation of this work is the transferability of the substantive (context-specific) theory to other settings beyond NWL and a UK cultural context. While research drew on a range of real-world improvement studies from different settings and on diverse clinical topics, all cases were from a single region (London, UK). Our wider author and team experiences suggest the cases represent wider national and global challenges (e.g. [92]). However, there is a need to explore the transferability of SHIFT-Evidence into other global settings and to continue to assess the comparative importance of the individual principles in different contexts.

The second limitation of this work is the transferability of findings to different intervention types and implementation and improvement approaches. All of the projects included in the empirical study were led by clinical leaders who voluntarily took on the role and defined the improvement area and evidence-based solutions, and in many instances this was done in collaboration with their teams and wider stakeholders. In addition, the use of a specific, quality improvement approach was promoted and supported in all project teams, although actual use of the approach was variable [93]. Further work is required to explore the transferability of SHIFT-Evidence to a greater diversity of intervention types (including organisational, system or policy level change) and implementation and improvement approaches.

The third limitation is methodological. The auto-ethnographic role of the researchers provided benefits including proximity to the subject matter, extensive contact with project teams and long-term relationships to explore how issues evolved over time. As all authors were senior members of the programme, there is a risk that their access to conversations and their perceptions and interpretations of the findings would be affected by status. The collaborative approach adopted between authors and other members of the CLAHRC NWL team (including more junior staff) allowed access to feedback from other programme participants and different types of conversations and ‘behind the scenes’ encounters. In addition, regular engagement with CLAHRC NWL project team members helped to triangulate findings and gain different perspectives. As such, the findings represent a culmination of discussion and sense making between the researchers and participants over an extended period of time. Evidence of these shared reflections exists in the publications co-authored with project teams that demonstrate insights into the challenges and complexity experienced (e.g. [94‐96]). Interpretation of the results was further triangulated with other experts in the field and in analysis of extensive literature to support reflectivity and to increase the reliability and validity of the findings. However, further research is required to explore how different methodological or theoretical perspectives produce convergent or divergent findings.

Further research is required to examine how to effectively operationalise the SHIFT-Evidence ‘simple rules’ in practice [97]. For example, knowing that ‘understanding problems and opportunities’ is important does not provide detailed guidance on how to engage relevant stakeholders to access local knowledge nor how to make sense of complex system interactions. Many approaches, tools and methods for operations research [98, 99], network analysis [36], implementation and quality improvement have already been developed and studied [100‐104], and this knowledge should inform the generation of structured and practical approaches that enable the SHIFT-Evidence ‘simple rules’ to be enacted in practice. It will also be necessary to work with advances in complexity sciences to develop new approaches to the practice and research of intervening in complex systems.