Evaluating complex interventions and health technologies using normalization process theory: development of a simplified approach and web-enabled toolkit

Recent years have seen steadily more sophisticated approaches to the evaluation of complex interventions and technological innovations in health care. In particular, evaluation frameworks like that proposed by the UK Medical Research Council have emphasized the need to understand the complex components and contingent underpinnings of outcomes studies, especially clinical trials [1, 2]. At the same time, there have been calls for theory-driven approaches to such work [3, 4]. Theories are valuable in such work not because they provide clear and unambiguous solutions to outcomes problems, but because they can provide robust, generic, and transferable explanations of the processes that shape these outcomes. They perform the further useful function of making transparent the assumptions of researchers and others that underpin research questions, methodology, and explanations [5, 6].

Normalization Process Theory (NPT) [7], and its predecessor, the Normalization Process Model [8, 9] provides a conceptual framework to assist in understanding and explaining the dynamic processes that are encountered during the implementation of complex interventions and technological or organizational innovations in healthcare.

Robust social science theories already explain some important features of implementation and integration processes: individual differences in attitudes and intentions in relation to new technologies and practices (e.g. Theory of Planned Behavior [10]), the flow of innovations through social networks (e.g. Diffusion of Innovations Theory [11]), and reciprocal interactions between people and artifacts (e.g. Actor Network Theory [12]). NPT differs from these theories because it offers an explanatory model of the routine embedding of a classification, artefact, technique or organizational practice in everyday work. NPT focuses on the agentic contribution--the things that people do--of individuals and groups. It thus explains phenomena not well covered by existing theories.

NPT was initially developed as an applied theoretical model to assist clinicians and researchers to understand and evaluate the factors that promote and inhibit the routine incorporation of complex healthcare interventions in practice. It started from a set of empirical generalizations derived from secondary analyses of qualitative data collected in a wide variety of studies of complex interventions in healthcare. This resulted in the original constructs of the model [8]. The further empirical applications of the model showed that while it could explain factors that promote and inhibit collective action, how participants came to engage and support the practice and how they reflected on and evaluated it remained unexplained. Through the development of further constructs, accounting for how people make sense of a practice, participate in it and appraise what they do, the model became a theory. Over the past four years it has been developed as a middle-range theory of socio-technical change [7], which characterizes the mechanisms involved in the embedding of social practices within their immediate and broader social contexts.

The starting point of NPT is that to understand the embedding of a practice we must look at what people actually do and how they work [7]. NPT focuses on four theoretical constructs, which characterize mechanisms that are energized by investments made by participants.

These mechanisms, and their underpinning investments, are constrained (and released) by the operation of norms (notions of how beliefs, behaviours, and actions should be accomplished); and conventions (how beliefs, behaviours, and actions are practically accomplished). In this context, mechanisms, investments, and constraints form processes of organized, dynamic, and contingent interaction between: agents (the individuals or groups that interact in encounters around a practice); objects (the classifications, artifacts, practices and procedures employed by agents); and contexts (the technical and organizational structures in which agents and objects are implicated) [8, 9]. The primary focus of NPT is therefore the analysis of social action. As far as possible, its central constructs and their dimensions refer to observable social mechanisms [13‐15] that shape the practical workability and integration of some complex intervention or technology. For health services researchers interested in process evaluation NPT provides a verifiable and empirically grounded model of the operation of factors that promote and inhibit the routine incorporation of interventions in everyday practice. For social scientists, NPT provides a well characterized middle-range theory of socio-technical change.

Although it is a relatively new theory, it has been used to:

Theories of all kinds are formed through complex interpretive processes that lead to inherently abstract products. Abstraction is, in fact, a necessary condition of a theory, since it must be sufficiently context-independent to be applicable to the range of relevant cases that it might be required to explain [39]. The problem that users of a theory face, then, is translating the theory from its abstract context-independent form into a form that can be used to solve problems in everyday settings. NPT is no exception. Our aim in the work reported here, therefore, has been to translate NPT's constructs into a set of statements that can be used by managers, clinicians, and researchers to work through problems of design and implementation in relation to complex interventions and new health technologies. These simplified constructs were translated into a set of statements that form the basis of a toolkit http://​www.​normalizationpro​cess.​org for clinicians, managers and policy-makers interested in utilizing NPT in their work.

The purpose of this simplification work was to develop a set of generic statements that could be configured as the 'front end' of a web enabled toolkit for users of NPT. For this reason, we sought engagement and critique from NPT's user communities (Health Services Researchers, Clinical Researchers, and Social Scientists). The co-production of theories is normal in large scale investigations in the natural sciences but is much less common in the social and behavioural sciences. In such circumstances, peers are usually asked to test theories rather than collaborate in defining the means by which they are operationalized. We have sought to be as transparent as possible in the generation of the theory, and as inclusive as possible in its operationalization and stabilization in practice. Our view is that this continuous 'road testing' of basic constructs and components of the theory has done more than ensure construct validity. It has ensured that the theory is relevant to its users. In this paper we present a simplified set of 16 statements that express key elements of NPT but which can be applied without a detailed knowledge of the underlying theory. However, we must also offer a caveat. Our objective in this work was to simplify a set of theoretical constructs for heuristic purposes, and not to develop a set of validated questions that could be immediately embedded in quantitative research instruments or qualitative interview schedules. The purpose of this paper is to make transparent the process by which the 16 statements and explanations were generated, and thus be clear about the foundation of the claims we make about them.

Respondents' critical comments on statements and explanations, as we have noted, were important and useful. We learned much about how the statements and their explanations were read and understood by a purposive international sample of researchers and practitioners. While respondents were enthusiastic and supportive about the statements and explanations, and valued the translation work that they represented, they also provided criticisms that focused our attention on problems in the way that the theory was understood when it was simplified in this way. This left us with three problems to solve.

First of all it was clear that we needed to rephrase individual statements to make their meanings clear, and to reduce problems of 'fine distinction' and overlap that affected some of them - especially in relation to statements 2, 3, 14 and 15. In fact, we rewrote almost all of the statements, working not only to clarify their meanings but also their purpose as heuristic devices to help users think through implementation processes rather than measure them. This involved producing and then choosing - by means of a simple vote by each member of the project team - alternative forms of words for each statement, and where necessary the explanation. We then undertook a final amendment phase to make them workable. The progression from original statement and explanation - through respondents' criticisms, alternative wording, voting choice, and to final version - is shown in detail in Additional File 1. We repeated this process after users had responded to the web-enabled tool. This led to the final set of statements.

The second problem was whether to do additional work to marry statements and their explanations more effectively, or whether to remove the explanations themselves. Some respondents had made a strong case for removing the explanations on the grounds that they would confuse novice users or distract expert ones. In this context, we also had to take account of the usability of the statement and explanation in the on-line toolkit. The combination of these factors led us to decide to include explanations on the web-interface (see Figure 1), and on its reports (see Figure 2). They are also embedded elsewhere in the on-line Users' Manual for NPT, where they are linked to more detailed accounts of the theory's constructs.

Finally, and rather less importantly, we had to decide whether or not to acknowledge the specific theoretical origin of each statement by assigning it the name of the component of NPT to which it referred. We chose to drop these from the toolkit. However, they remain elsewhere in the on-line User's Manual. The limitations of this study are that our sample may be biased towards a favourable view of NPT by virtue of their previously expressed interest in NPT and earlier personal contacts. A second limitation is that is it also biased towards respondents working in some capacity in academia over those working as full-time practitioners. As such, the practitioner group is relatively small and this may have implications on the potential usability of the tool for this group. Clearly, irrespective of researcher enthusiasm, practitioners, managers and policy makers, alongside patients and careers, are central to the successful embedding of interventions. However, we should note that many of these academics also had commitments as clinical practitioners, healthcare managers, and policy makers. A third limitation is that limitations on time and resources did not permit us at this stage in the project to perform cognitive interviews in which users of the statements worked through them while thinking out loud. Overall, using email and a web based tool to collect qualitative (textual) data from a purposive sample of international researchers and practitioners was highly successful, with a very small number of non-respondents. The 59 researchers and practitioners who responded to our qualitative data collection tool, and the 13 who commented on the beta version of the toolkit at http://​www.​normalizationpro​cess.​org were supportive and helpful, and consistently provided us with valuable critical comments.

The funding program that supported the work described in this paper was intended to support the translation of social science research into products that would have value for the wider polity. Our aim in this paper is to show how we worked towards this objective. Our aim for the project itself was to take the core constructs and components of a sociological theory and translate them into the simplest possible set of statements. These statements were designed to be used as heuristic devices in an on-line toolkit for users of the theory, and not to define questions that could be used as the basis of an instrument to measure variables derived from NPT's constructs and their components. As a result of this work we have been able to develop a simplified set of statements and explanations that translate a sociological theory into a 'user friendly' form of words. This is an important step in crossing the translational gap between the complex language of academic expert communities and the multiple everyday needs of researchers and practitioners in applied settings [46].

CRM led the program of theory building underpinning work described here, and all authors have contributed to the development of NPT.