Challenges to evaluating complex interventions: a content analysis of published papers

There is continuing interest among practitioners, policymakers and researchers in the evaluation of complex interventions. This interest stems from the need to further develop the evidence base on the effectiveness of healthcare and public health interventions, and an awareness that evaluation becomes more challenging as interventions move along the spectrum from ‘simple’ towards more complex interventions [1]. This focus on complexity is also driven by ongoing debate about the most appropriate methods for evaluating health systems, and the recognition that it is important to know not just whether health system interventions ‘work’, but also about when, why, how and in what circumstances such interventions work well [2, 3].

A further stimulus has been the Medical Research Council’s (MRC) ‘A framework for development and evaluation of RCTs for complex interventions to improve health’, originally published in 2000 [4] and revised and extended in 2008 [5]. This guidance was published in response to the difficulties faced by those attempting to develop complex interventions and evaluate their impact. It describes complex interventions as being ‘built up from a number of components, which may act both independently and inter-dependently’ [4]. These components include behaviours, behaviour parameters and methods of organising those behaviours, and they may have an effect at individual patient level, organisational or service level or population level (or all of these in some cases). The MRC’s 2008 guidance also emphasises the numbers of components and their interactions, behaviours, organisational levels and outcomes, and goes further than the 2000 framework in outlining the variability of desired outcomes and the degree to which flexibility or tailoring of the intervention is permitted. Both documents highlight the importance of establishing both whether an intervention is effective and how it works.

The term ‘complex intervention’ is now used extensively in the academic health literature to describe both health service and public health interventions. Complex interventions have been the topic of numerous conferences and meetings, the focus of funding calls, and will be the subject of a new chapter in the Cochrane Handbook for Systematic Reviews of Interventions [6]. The common usage of the term indicates increasing recognition of complexity and its implications for the development and evaluation of interventions. It may also be the case that, as the term has achieved wider application, it has come to be used strategically by researchers to add authority and currency to funding proposals and academic articles. However, it is not always clear that ‘complexity’ is being used to refer to the same things, nor what measures researchers are taking to evaluate it. It has been suggested, for example, that what is described as ‘complexity’ is actually just ‘complicatedness’ – a very different concept [7].

We undertook an analysis of published journal articles in the field of health in which complexity was an important element. Our aim was to identify the aspects of complexity described by writers; the fields in which complex interventions are being evaluated; and to describe challenges experienced due to the complexity of interventions and how authors dealt with these. This paper outlines the findings of this documentary analysis focusing, in particular, on the challenges of designing, implementing and evaluating complex interventions described by authors.

The search resulted in a sample of 221 papers of which full text of 216 (98%) was obtained and 207 were included in the analysis. Nine papers were excluded because their subject matter was not relevant for our purposes. A small number of the papers included were published online in 2011 but in print in 2012.

The 207 papers broadly cover clinical (45%), methodological (27%), health promotion (23%) and public health (3%) topics with a small number of ‘others’ (1%). All those included in the analysis are listed in Table 1. Some papers focus on particular health conditions, such as cancer, diabetes, HIV and mental illness; some on health and social interventions, including palliative care services, complementary therapies and decision aids; and others on methodological and theoretical issues such as causality, the use of normalisation process theory, and approaches to health promotion.