Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges

Within the PARiHS framework, successful implementation (SI) is represented as a function (f) of the nature and type of evidence (E), the qualities of the context (C) in which the evidence is being introduced, and the way the process is facilitated (F); SI = f (E, C, F). Detailed descriptions exist in the literature on the development and empirical evaluation of the PARiHS framework [4‐9]. The framework has been refined through two phases of research and development and is currently in its third or current phase (see Table 1 for a comprehensive summary). The unique characteristic of the PARiHS framework was that it proposed a three-dimensional framework within which to interpret successful implementation, arguing that elements could be located on a continuum of "high" to "low" evidence and context

The main features and assumptions of the framework are:

The objective of the current phase of our work is to build on the concept analysis and clarification undertaken in phases one and two, and to evaluate the current framework through the development and testing of diagnostic and evaluative instruments to assist in the process of knowledge translation. Whilst conducting this phase, a number of challenges have arisen, which, whilst reflecting the particular complexities of the PARiHS framework's development, are also relevant to current debates in the field of knowledge translation. These include:

These issues will be considered in turn.

There is a growing interest in the literature around clarification of terminology used in implementation science and also around the use of such mental devices as conceptual frameworks, theories, and models [15]. However, despite the debate there is little consistency in the way these terms are used and in particular there is a tendency to substitute one term for another without due consideration of the deeper meaning attributed to such terms. For example, one of the earliest conceptual frameworks developed by Havelock and colleagues [16] was derived from Roger's Diffusion of Innovation Theory [17]. Their Research Dissemination Utilization Conceptual Framework was built upon two key components – knowledge building and institutionalizing. Knowledge building (or synthesis) would integrate theories to replace fragmented approaches, and their concept of institutionalisation enabled the new knowledge to be transferred through integrated, cross discipline, cross boundary programmes. Of primary importance is the relationship of trust and mutual co-operation that needs to be built up between researchers, policy makers, decision makers, and practitioners.

Greenhalgh et al.'s synthesis of the literature on diffusion of innovations has also produced a conceptual map [18]. Described by the authors as a "conceptual model" it is not expected to be used in any practical way to guide actions. Rather, it is a mental representation of the many elements that need to be considered. Each element was derived from rigorous review of the literature, and the antecedent theories informing (and shaping) the elements are described. For example, one element of the model is diffusion, and within that element several approaches to diffusion are described – social networks, marketing, expert opinion – all aspects/actions of a wider theoretical perspective.

In contrast, Graham et al. have offered their own conceptual framework to help elucidate what they believe to be the key elements of the knowledge-to-action process [15]. Essentially the framework is divided into two elements or concepts: knowledge creation and action, with specific steps within each element. (For a detailed explanation see Graham et al. [15].) The framework also begins to articulate the embedded theoretical positions that determine action. Graham and colleagues are particularly interested in theories of planned action [19] and have identified over 60 theories or frameworks (although the authors do not distinguish between these terms in their paper).

The PARiHS framework shares similar characteristics to the above conceptual frameworks – the identification of elements and relationships, embedded (either explicit or implicit) theoretical positions, and a way of trying to explain a complex set of phenomena that enable action to be taken. What distinguishes the PARiHS conceptual framework from the others is that as well as mapping the interrelationships, PARiHS has the potential to be used as a practical and pragmatic tool by practitioners and researchers at the local level. This is the claim, and it is also the hypothesis that we are continually testing.

In addition to the literature on conceptual frameworks, there is an emerging debate about theory use and development in knowledge translation work [20‐22]. Theory use is presented by its supporters as a promising approach to better understanding the 'black box' of implementation [23]. Estabrooks et al.'s description illustrates the vast array of theories that could be used to determine or explain the process of implementation, and also demonstrates the lack of agreement on terms – models, frameworks, and theories are used interchangeably [20]. What seems to be emerging is that the term theory is used when a set of relationships are explained and there is some predictive capability: for example, the use of social influence strategies to introduce clinical guidelines [24]. The use of the term model seems much more diffuse – ranging from prescriptions on how to implement research into practice [25‐27] to more specific descriptions of a theoretical perspective, e.g., Prochaska et al.'s transtheoretical model of health behaviour change [28]. However, to date, the prevalent view about theory and knowledge translation has tended to focus upon positivistic interpretations that favour deterministic explanations [23].

Therefore, one important question is whether it matters what we call these mental devices. Is there a difference between conceptual frameworks, theories, and models, and, if so, what and how would such differentiations help our understanding of the complex world of research implementation or knowledge translation? Identical questions have been posed in the discipline of public policy analysis and implementation, as well as theory development [29]. The policy world is complex, with multiple elements interacting over time. How can complex situations be simplified in order to understand them, and how can the tension be managed between the exploration of specific interventions within a system and the overall appreciation of the impact of the intervention on the whole system? In attempting to create a deeper understanding, Sabatier and colleagues have described three dominant approaches to policy analysis and implementation [29]. Within this analytic framework they have also put forward a typology for understanding the different 'mental representations' we could use to hold onto the complex world. This analytic framework, first proposed by Ostrom [30, 31], has been used as a way of trying to make sense of the different ways that frameworks, theories, and models could be used to inform our research activity. Ostrom [31] argued that:

"...given the need for multiple disciplinary languages and given the multiple levels of analysis involved in studying configural relationships between rules, relevant aspects of the world and cultural phenomena, the study of institutions does depend on theoretical work undertaken at three levels, namely frameworks, theories and models"

Both Sabatier [30] and Ostrom [31] argue that for the effective development of policy theory the following distinctions can be made. A conceptual framework identifies a set of variables and relationships that should be examined in order to explain the phenomena. Indeed, a framework can provide anything from a skeletal set of variables to something as extensive as a paradigm, where a paradigm is the notion which places emphasis on professional consensus within a particular scientific community. It stands for the entire constellation of beliefs, values, and techniques shared by members of that community [32]. A conceptual framework need not specify the direction of relationships or identify critical hypotheses. In contrast, a theory provides a denser and logically coherent set of relationships. Theories can offer views on the causal relationships and seek to explain the phenomena, although from an interpretative perspective theories also play a vital role in offering explanations rather than causal relationships [33]. Numerous theories may be consistent within the same framework. Models, by contrast, represent a specific situation, are narrower in scope, and are more precise in their assumptions [29, 31]This approach would seem to offer one way of testing conceptual coherence between the typological levels within the discourse of implementation science.

For Ostrom [31], a conceptual framework helps to identify elements and relations among those elements that one needs to consider for an analysis of organisations (at multiple levels of operation – individual, team, unit, and whole systems level) and their ability to absorb and adopt innovations. Frameworks also organise diagnostic and prescriptive enquiry and provide a more general list of variables that can be used to analyse types of institutional arrangements. Conceptual frameworks provide a meta-theoretical language that can be used to compare theories, and they attempt to identify universal elements of any theory relevant to the same kind of phenomena that would need to be included in order to understand the "bigger conceptual picture". Thus, for example, in Ostrom's analysis, the question would be whether the elements as identified in the PARiHS framework survive continuous scrutiny and testing against multiple theories at multiple levels within the organisation that have a relevance and coherence to research implementation strategies. So long as this is the case, the elements remain intact: once exceptions begin to emerge, the basic tenets of the conceptual framework are placed under further scrutiny.

Whilst the PARiHS framework has been subject to an ongoing development process, questions about it remain, including:

For the framework to usefully inform the development and testing of current and emerging theories, these questions need to be answered. Arguably, work to date has provided evidence of the framework's content and construct validity [6‐8, 11]; that is, we can be reasonably confident that PARIiHS is a conceptually robust framework. This is a sufficient basis upon which to begin testing a range of theories and building new theories inductively and deductively. The test of its effectiveness as a conceptual framework is whether it can generate such diagnostic, analytic, prescriptive, interpretative, and evaluative discourse.

According to Ostrom [31], key questions to test the coherence of any conceptual framework, include:

Questions used to test the usefulness of any conceptual framework in empirical research include:

And finally, in relation to conceptual frameworks, Ostrom's typology includes questions about the ease by which the framework aids the better understanding and dialogue across disciplinary boundaries:

These questions are helpful because they enable an assessment to be made of the PARiHS framework's stage of development and provide an agenda for further work. For example, there is evidence to indicate that PARiHS does help organise empirical research where theories are yet to be formulated [11, 34], and that the framework has led to better explanations of important phenomena [10, 35]. However, consideration still needs to be given to the framework's capability for theory application and development. Questions about the range and diversity of applicable theories still need to be explicated. Adopting Ostrom's typology, which acknowledges multiplicity, it could be argued that rather than placing PARiHS within one particular theoretical perspective or offering a single theory for research implementation, which could limit its applicability, the framework could be populated by multiple theories, at multiple levels. From this perspective the PARiHS framework would operate similarly to other frameworks outlined earlier, e.g., Graham et al.'s [15] Knowledge to Action (KTA) framework that focuses on planned action theories, or Greenhalgh et al.'s [18] conceptual model summarising the range of theories that influence diffusion of innovations.

Further consideration of these issues forms the basis of the next phase of work/development of PARiHS and in particular attempts to answer the key questions raised by Ostrom. These include whether the framework can provide a coherent language for identifying universal elements of theories attempting to explain an important range of phenomena, or indeed whether PARiHS can be applied to multiple levels of analysis – such as individual, team, unit, and organisational-wide level?

However, unlike the chess game, we still do not know the rules (should there be any) of the knowledge translation game and the movements of the different pieces are yet to be fully understood. Of course, this analogy only works if we accept the prior assumption that implementation processes are predictable, and that there are certain causes and effects at work. The converse position is to assume that all interactions are random, and that there is no predictive capacity because of the complexity involved in working with so many variables. Given that we do not know which of these positions is the more accurate, and it is largely dependent on one's world view of how these issues should be studied, we argue that it is legitimate to proceed with the "chess game analogy" until there is sufficient evidence amassed to disprove it. Taking such an a priori position is consistent to Kuhn's notion that all good scientific endeavours are about the business of empirically falsifying propositions within a theoretical framework [29].

Thus, to conceptualise the process of introducing evidence into practice, we are suggesting that to use the PARiHS framework, practitioners and researchers contemplate the interplay of evidence, context, and facilitation, as well as their sub-elements. Each element and sub-element has a conceptual and theoretical order that determines its intrinsic properties; the interaction of these elements is conditional on their state, maturity, context, and many other factors. The modelling or experimentation that can be constructed is a way of tracking the nature of the different elements and beginning to map the processes by which change occurs through the interaction of these elements.

Table 2 illustrates how the PARiHS framework elements (evidence, context and facilitation) could draw on multiple theoretical perspectives. This, in turn, offers even more models that can then be used to explore systematically the consequences of these propositions in a clearly defined and controlled set of outcomes. What begins to emerge when looking at Table 2 is that, depending on the theoretical approach taken, there is any number of entry points into testing elements of the framework.

How researchers and practitioners "make sense" of the bigger conceptual framework is a fundamental question and an on-going challenge reflecting the complexities involved. The choice of theoretical perspective will necessarily put a boundary around the area of investigation. For example, if we want to investigate the impact of opinion leaders on research implementation using transformational leadership theory, then we will still be left with the job of integrating these findings into the bigger conceptual picture of how research findings get into practice. Holding one piece of the conceptual jigsaw without negating the possible impact of other factors is very important but very difficult to manage. The proposed links are hypothetical and illustrate the conceptual challenges of any nascent discipline.

Such an approach to framework and theory use and development requires researchers to be flexible and holistic. To date, the knowledge translation literature describes theory use and development as a linear and discrete process in line with more traditional scientific thinking and the search for understanding causation [21, 22]. Looking to other methodologies that are less concerned with causation and more focused on explanatory understanding and action (such as realist evaluation) [33] may limit reductionism, and provide enlightening findings about the interactions and complexities involved in knowledge translation activity [23]. However, we still need to be mindful of the relationship between the theory and the subsequent methodology and consider their fit with each other and with philosophical perspectives.

An additional set of definitional challenges that Sabatier and Ostrom's typology raises is their definition of models. Ostrom describes models as precise assumptions about a limited set of parameters and variables [31]. Logic, experimentation, and a variety of simulations can be used to explore systematically the consequences of these assumptions in a limited set of outcomes. Multiple models are compatible with most theories and frameworks.

So, for example, in this typology, we could set up an experiment that would test the model of audit and feedback as a precise intervention. The theoretical underpinning of the model could be decision theory or learning organisation theory, both embedded within the bigger conceptual framework of evidence, context, and facilitation. The challenge then is to draw sufficiently cogent paradigmatic boundaries around the framework so that it does not become a catchall of ideas and conjectures. How we do this is where the real scientific discipline comes into focus and where logical coherence and consistency of terms and relationships are set out for scrutiny, and it is where causal processes seek to explain how certain patterns of phenomena have come about. And, of course, this requires the ability to measure the variables under scrutiny. Our deliberations over the years as to how we move the PARiHS framework from a conceptual artefact to becoming a measure of knowledge translation has led us down the path of whether we have to disaggregate the elements because there is no conceptual coherence or whether we dissect the elements in different ways. The notion that the framework becomes a diagnostic and evaluative measure on the evidence and context axes and informs the facilitation or intervention process has been an important development.

Estabrooks et al. have outlined the challenges of measuring knowledge utilisation in health care [36]. These include a lack of underpinning theory, construct clarity, measurement theory, psychometric assessment, and a presumption of linearity. Additionally Rich claimed that there tends to be a bias to measure things that are easy to capture [37]. These measurement challenges reflect the general complexity of research implementation. As described above, the purpose of the PARiHS framework is to provide a map to enable others to make sense of this complexity, and the elements that require attention if implementation is more likely to be successful. The next step is to consider whether the PARiHS conceptual framework lends itself to guiding the development of diagnostic and evaluative approaches and instruments, which could be used by both researchers and practitioners. Our emerging hypothesis is that the PARiHS framework will guide measurement development, and there is growing evidence to support this [10‐14]. Given that more theoretical work needs to be conducted on the PARiHS framework, these ideas are at an early stage.

To this end, a briefing summary – PARiHS Framework: stages of refinement – [See Additional File 1] shows some draft questions, which may begin to facilitate the identification of those elements within 'evidence' and 'context' that require development work, and active intervention(s) ('facilitation') to be successfully introduced within specific implementation projects. The questions developed in the tool could enable individuals and teams to test their appreciation and understanding of evidence, context and facilitation. For example, using the four sub-elements of 'evidence', the tool enables the development of a better understanding of assumptions and perceptions about the research base, how this conflicts with and/or supports clinical experience, professional judgement, and patient preferences, and whether routine information is sufficiently robust to be able to offer data on current practices as well as what needs to change. Similarly, the questions about 'context' encourage an evaluation of the preparedness of the context to embrace and sustain implementation. These questions could be answered individually and/or through a facilitated dialogue where each team members' assumptions, prejudices, views about existing practice, and the proposed change are discussed and debated. Through this process the team would come to an agreed ranking of the 'readiness' of the team to embrace the new practice, evidence, or innovation.

We suggest one way of testing this state of readiness is to aggregate responses to the questions, and then translate them onto a grid that plots the position the team judges themselves to be in before they embark upon the implementation process. An example of this is presented in Figure 1 – The PARiHS diagnostic and evaluative grid. At this stage, the location on the grid enables an assessment of the type of facilitation support that would most effectively lead to the successful implementation of evidence, likely requiring changes in behaviour and working patterns. The diagnosis identifies the position of the team. The trajectories in Figure 1 illustrate examples of three possible positions:

Whilst it is likely that many sources of information will be required to decide on an appropriate course of action, the diagnostic score may provide an indication of the starting point. Facilitation, as the intervention, can then combine a range of approaches ranging from task focused (e.g., project management, resource identification) to more enabling processes (e.g., personal development, action learning). The role of the facilitator then is concerned with assessment of the situation, assessment of individual, team and workplace readiness, development of change and evaluation strategies, support of the implementation process, and coaching and mentoring the team though the change.

The final task is to evaluate and check whether the self-assessment scores have migrated further towards the top right hand quadrant. By considering the evaluation questions, both individuals and teams can evaluate their implementation efforts. These could be administered as both process and summative evaluation opportunities, and progress mapped.

Using PARiHS as the basis for a tool may shift thinking away from conventional, arguably narrow, notions of measurement to more wide ranging and eclectic approaches to evaluation. Given that we predict so much variation in appreciation of evidence and context, the way to test the measurement tool has to accommodate variety and multiple interpretations. Similar messages are coming through from other research teams [15, 18, 20]. This is why we wish to set up networks of researchers and practitioners who are willing to work together to test out these assumptions and ideas.

The concept of facilitation, defined as "a technique by which one person makes things easier for others", continues to be central to the PARiHS framework [6]. Facilitation is part of a range of roles that have been reviewed in the literature [38, 39] which tend to demonstrate that effective implementation of knowledge into practice is a multifaceted process, requires flexibility, and has more to do with the ability to combine a range of different techniques than rigidly prescribing a discrete intervention. From Harvey et al.'s concept analysis the following positions have emerged [6]:

As a key dimension of the PARiHS framework, facilitation needs to be further refined and tested. Our deliberations have led us to propose that facilitation will be more effective following a diagnosis of the context (C) into which the new knowledge is being introduced and an assessment of practitioners' understanding of and acceptance of the evidence/new knowledge itself (E). The diagnostic should provide data to determine the most appropriate facilitation approach – which, consistent with the evidence, should lead to a multi-faceted, flexible, tailor-made intervention being constructed. Information on individual and teams' understanding of and receptiveness to the new evidence will help determine how much new learning and change is needed. An assessment of the readiness for change from a contextual perspective will cover the PARiHS framework sub- elements of context, culture, leadership, and evaluation [11, 14].

The role of the appropriately prepared facilitator, along with the team(s) they are working with, is to construct a programme of change that meets the individual and team's learning needs. The actual ingredients making up this intervention could draw from a whole range of methods – from very task based, planned change programme approaches to much more experiential, action learning approaches.

What we are proposing is consistent with the early observations of Havelock and colleagues, where they argued for greater alignment or linkage between those who generate and implement new evidence and those who are expected to be involved in its application [16]. Equally, Van de Ven and colleagues' work on the management of innovation demonstrates the importance of flexibility and skilled support in achieving successful implementation throughout the length of the implementation process [45]. Also, Greenhalgh et al. recommend much greater involvement of practitioners in determining the nature of interventions at local level [18], a perspective that is consistent with the emerging interest in such theoretical perspectives as realistic synthesis [46]. In this interpretivist approach, the research interest in interventions is not so much about their generalisability and standardisation but more about understanding the mechanisms that connect events in a way that changes them (either in the desired direction or in an unexpected way) within a particular context involving particular participants.

What are the next practical steps for developing a series of studies that would begin to test the hypotheses outlined above? First obvious steps are to refine the diagnostic process and associated measures, and the second step is to agree the content of a facilitation training programme that would equip appropriately (and consistently) trained facilitators to work with a number of practice areas wishing to engage in knowledge translation activities.

We argue that more careful theoretical work, modelling, and testing of the concept of facilitation is required because it is the process by which individuals and teams first interact and engage with evidence (either as guidelines, research reports, or any new innovation entering the system) and then try to negotiate its adoption/acceptance into their organisation.

These conclusions would lead one to surmise that the future direction of travel will be around the development of much more complex and bespoke interventions that fit local contexts. Whether this is different from what the PARiHS framework terms "facilitation" is an appropriate question to ask.

There is a small, but growing body of evidence from research and practice that shows the PARiHS framework has conceptual integrity, face and concept validity.

However, there are major challenges ahead if the framework is to help in the systematic exploration of these complexities around the art and science of implementation.

The three challenges outlined include the need to integrate theoretical perspectives into the framework in a way that enables us to make sense of the complexities and to construct appropriate models to explore what works in knowledge translation. PARiHS was developed inductively, which points to an interpretive lens on theory application and development. However, using Ostrom's helpful analytical approach it may be more helpful to begin to see the framework as being populated by various theoretical positions, which some would view as strength, some as a weakness. These issues have yet to be debated in the knowledge translation literature generally, and in relation to the PARiHS framework specifically.

A second area of investigation is the development and testing of diagnostic and evaluative methodologies and associated instruments based on the elements and sub-elements of the PARiHS framework. What seems to be emerging is the need for a high level set of principles (conceptual framework) that can help people on the ground understand what they can do. The principles can offer a framework within which a number of approaches or attempts at implementation and evaluation of the effectiveness of the intervention can be made by both the players on the ground and any researchers involved with them.

Thus, it would seem that the current knowledge base around successful implementation of innovations into practice emphasises processes of engaging local practitioners as well as outlining a set of key principles that help guide the activity.

Whilst there are some studies underway [32], to date there have been few, if any, systematic investigations of facilitation as an intervention. We believe that much more conceptual clarification is needed before the science around it can improve. Equally, the arguments put forward here, that future implementation tools ought to have both diagnostic and evaluative properties also need to be tested. Our proposal to create communities of researchers, practitioners, and other stakeholders undertaking pieces of work to test the whole framework is presented as a way of moving the agenda forward. We see the need for this collaborative approach, not only between researchers but also between research teams and those practitioners at the local level who actually have the task of implementing evidence into practice. This need for greater alignment of these two groups has been reinforced by Greenhalgh et al.'s work [18], where they broaden out the notion of implementation of evidence to include innovations in general and by practice development researchers [46]. We also note the semantic and conceptual shift in the discourse and seminal work of Van de Ven and colleagues [47] who had, two decades earlier, tried to measure these very same complexities. Their elegantly designed studies could help us to construct more appropriate studies that take account of the multiple elements at work.