Quality of reporting of complex healthcare interventions and applicability of the CReDECI list - a survey of publications indexed in PubMed

Researching complex interventions in healthcare and nursing has gained increased awareness following the publication of the UK Medical Research Council’s (MRC) framework for development and evaluation of complex interventions in 2000 [1] and its update in 2008 [2, 3]. The MRC framework defined complex interventions as interventions comprising several components which may act either independently or inter-dependently [2]. However, there is no distinct boundary in the classification of whether an intervention is complex or not. Characteristics of complex interventions are, for example, the number of different professions or organisational levels targeted by the intervention and/or the degree of flexibility permitted for the intervention [4].

While the first framework [1] implied a linear approach comprising one preclinical and four clinical phases, the updated framework [2, 3] builds on a circular model which gives a better reflection of the flexibility or even non-linearity of the research process. However, both versions cover the same methodological steps of development, piloting and evaluation of a complex intervention, and the long-term implementation after the complex intervention has demonstrated its effectiveness. Also, both frameworks emphasise that the development of a complex intervention should include a theoretical basis and a clear description of the intended change processes. Prior to the evaluation, the intervention should be piloted in the target setting and its feasibility has to be tested. The evaluation study should not only focus on effectiveness and cost-effectiveness but also include the evaluation of the intended change process by investigating the dose delivered, fidelity and reach of the intervention (process evaluation) [5‐7]. Thus, the development and evaluation of complex interventions requires several studies using different methods and study designs [1‐4].

Comprehensive reporting of all steps of the development and evaluation of a complex intervention is crucial. Sufficient information must be available for the judgement of the intervention’s clinical benefits, for replication, or for adaption of an intervention to different settings or countries [8‐10]. However, several analyses revealed shortcomings of the reporting of core aspects of complex intervention research, e.g. regarding the theoretical basis and assumptions guiding the development and piloting of the intervention, and the description of intervention delivery [10‐13]. These studies evaluated interventions on defined topics, e.g. care of people after stroke [13] and reduction of physical restraints in geriatric care [12], or specific methodological aspects of research on complex interventions [10, 11]. An evaluation of the reporting quality of a sample of systematically identified complex interventions explicitly adhering to the MRC framework has not been performed so far.

We have recently published a list of specific criteria for reporting complex interventions in order to offer a structured guidance for researchers and authors covering the first three phases of the MRC framework [14]. In contrast to other reporting guidelines (e.g. CONSORT), CReDECI does not comprise design-specific items. The development and evaluation of complex interventions requires the use of different methodological approaches and the criteria list includes only items covering these specific methodological aspects [1‐4, 14]. Therefore, CReDECI should be used alongside established study design-specific reporting statements.

Employing the CReDECI criteria, the aim of the present study was to evaluate the reporting quality of the development and evaluation of complex interventions adhering to the MRC framework. The second aim was to test the applicability of the criteria list.

A total of 274 publications were identified via database search and forward citation tracking. Eight trials met the inclusion criteria (Figure 1). Main reasons for exclusion were that the interventions were not labelled as complex or the studies did not adhere to the MRC framework. A total of 26 publications were identified as reporting on the eight included trials. In two cases, authors were contacted for missing information. The mean number of publications per trial was 3.25 (range 1 to 6). For one trial, only one publication was available [16]. Four trials were conducted in UK [16‐19], two in Ireland [20, 21] and one in Germany [22] and the Netherlands [23], respectively. Most of the complex interventions offered structured approaches for improving the quality of care for patients with chronic or complex conditions. Target groups were people suffering from a stroke [16, 19], cancer [18, 22], coronary heart disease [20], or diabetes mellitus Type 2 [21], patients at risk from prescription and medication management errors in general practice [17], and frail elderly people [23]. The interventions offered improved care concepts or pathways [17, 19, 20, 22] or alternative or modified forms of information or support [16, 18, 21] e.g. peer support groups for patients and/or their relatives [21]. Table 2 presents an overview of the included trials and the underlying interventions. The quality of reporting on development and piloting of the intervention was judged as good (Table 3). Seven out of eight trials reported on all six items referring to the intervention’s development [17‐23] and one trial reported on three items [16]. Most trials presented sufficient information on the feasibility and pilot phase. Concerning the introduction and evaluation phase, the quality of reporting varied. Only one trial reported on all eight items [21] and one trial on seven items [17]. Five trials reported on one to five items [18‐20, 22, 23] and one trial offered no information on any item [16].

The number of items with different ratings per trial ranged from 0–5 with a mean of 2. The two reviewers disagreed on 16 out of the total of 128 ratings (12.5%). The time needed to rate a trial ranged from 30 to 90 minutes, depending on the number of publications per trial.

We identified a small number of trials whose authors explicitly stated having adhered to the MRC framework for developing and evaluating complex interventions [2]. The quality of reporting in these trials was judged as good for the intervention development and piloting phases. Thus, our findings are contrary to previous analyses of the quality of reporting on complex interventions [12, 13, 41]. However, the trials included in our sample are more likely to fulfil the CReDECI criteria, since all trials confirmed adherence to the MRC framework. For the evaluation phase, quality of reporting varied. While two trials reported on nearly all criteria, six trials offered information only on half of the items or even less. These results confirm the findings from previous studies [9, 11, 12, 42]. Only half of the trials included an evaluation of the implementation process (CReDECI criterion 12). However, this information is required in order to get a deeper understanding of the effects of the evaluation and to describe barriers and facilitators influencing the intervention’s implementation. A process evaluation should be an integral part of the evaluation of complex interventions [2, 5, 6]. Only half of the trials offered information on the characteristics of the care delivered in the control group, which was often only described as usual or standard care. The lack of information hampers the replication of studies and the adaption of the intervention to different settings or countries [9, 10].

The integration of the local context and detailed information on the process evaluation has recently been described as frequently underreported [43, 44]. In our sample only a small number of trials included a process evaluation but most trials offered information on the integration of the context in the intervention’s development. However, we did not assess the completeness of the information provided.

The CReDECI list has proven its applicability and practicability. A high rate of agreement was reached by both reviewers. The time needed for assessment varied, depending on the number of publications per trial. Since both reviewers were familiar with complex interventions research and the MRC framework, application of the CReDECI list by untrained users might be more time-consuming.

The currently available reporting guidelines specifically address a defined study design, offering recommendations to improve the reporting of design-specific methodological aspects [45]. The CReDECI list employs a different approach since it comprises items covering three phases of complex interventions’ research: development, feasibility/piloting, and introduction of the intervention and evaluation. However, we recommend the additional use of design-specific reporting guidelines alongside the CReDECI list. Currently, a CONSORT extension for social and psychological interventions is being developed [46]. This extension might be a valuable supplementation to the CReDECI list.

In this study, adhering to the MRC framework for the development and evaluation of complex interventions seems to have a positive impact on the quality of reporting of the complex interventions’ development and feasibility/piloting. These results are in contrast to former analysis [10‐13]. Reporting on the evaluation phase could still be improved.

The CReDECI list seems to be a practical instrument for checking the quality of reporting in publications on complex interventions. It could be used alongside established design-specific reporting guidelines such as the CONSORT statement. To further validate the CReDECI list, a formal consensus process with researchers and stakeholders is scheduled.