Comparison of registered and published intervention fidelity assessment in cluster randomised trials of public health interventions in low- and middle-income countries: systematic review

Recent methodologically oriented systematic reviews have found evidence of improvements in the design and analysis of CRTs, while also noting deficiencies in trial implementation [18, 19]. Previous systematic reviews have emphasised the potential importance of process evaluation to mitigate these methodological problems, which can affect the internal and external validity of trial results [3, 18, 20‐22].

‘Intervention fidelity’ refers to the degree to which an intervention is delivered as initially planned [23]. Fidelity assessment is the practice of evaluating intervention fidelity. Considered an aspect of process evaluation, fidelity assessment aims to understand and measure to what extent the intervention is being implemented as intended and identify specific reasons for the success or failure of the intervention [9, 23, 24]. The benefits of evaluating intervention fidelity include increased confidence in scientific findings, increased power to control for confounding factors and increased ability to evaluate the performance of theory-based interventions [25]. Fidelity assessment can improve internal and external validity of CRTs [18] by providing evidence that trial results are due to interventions themselves rather than to external factors and facilitating generalisation of results to contexts that may differ substantially from the original trial setting [9, 23]. Fidelity assessment may be particularly important for public health interventions, as these interventions tend to be complex and constituted by multiple components [10, 26] that act independently or interdependently [27], allowing greater potential for variation during implementation [23].

Table 1 outlines the conceptual framework used in this review, based on the work of Carroll et al. [23] and refined by Hasson [28]. We selected this framework to guide the review because it provides a comprehensive synthesis of previous work on intervention fidelity and has been widely influential.

The Consolidated Standards of Reporting Trials (CONSORT) group was created to provide guidance to improve the quality and transparency of RCT reporting [29], including CRTs [2, 30]. The CONSORT statement recognises that the trial protocol for a given study may not have been followed fully by some trial participants for a wide variety of reasons, including failure to receive the entire intervention as planned [29]. Cases of protocol non-adherence may influence interpretation and credibility of the results and thus the validity of the conclusions [18, 25, 31, 32]. To preserve the ability to make strong inferences about intervention effects, CONSORT recommends that all participants randomised be retained in the analysis and analysed according to their original assigned groups, an approach known as ‘intention-to-treat’ (ITT) analysis.

Reasons for protocol non-adherence in individually randomised RCTs may differ from those in CRTs. In a clinical trial setting, non-adherence depends largely on the actions of the trial participant and the treatment provider, which may in turn be related to issues such as treatment side effects and safety. In CRTs of public health interventions, protocol non-adherence may occur because complex interventions that include multiple components are delivered with poor fidelity. However, both for individually randomised parallel group trials [29] and CRTs [2, 30], current CONSORT guidelines offer no advice on methods to assess protocol non-adherence.

As earlier methodologically oriented systematic reviews have demonstrated, CRTs of complex public health interventions may be particularly at risk of experiencing protocol deviations and non-adherence, and these may compromise the validity of their findings [18, 19]. Although process evaluation techniques, such as evaluation of intervention fidelity, can help to identify these problems and mitigate their negative effects, current reporting guidelines for CRTs offer no specific guidance on intervention fidelity assessment. Wide divergence in current practices is therefore likely. We undertook a methodologically oriented systematic review of current practices related to assessment of intervention fidelity within CRTs of public health interventions in LMICs, so as to inform best practices for CRTs. To our knowledge, no other systematic review addresses this question. Although CRTs have been widely implemented to evaluate public health interventions in both high-income countries and LMIC contexts, interventions, approaches and outcomes may differ substantially between settings. Given the limitations of the resources, these challenges may be even more important in CRTs of public health interventions in LMICs. We therefore limit our focus to LMICs.

We conducted a methodologically oriented systematic review to address the following research questions:

Literature search strategies were developed in collaboration with an academic librarian. The search strategy was limited to the period from January 2012 to May 2016 and, due to resource constraints, was not updated towards the end of the review (12-07-2017). Search strategies used MESH and text words related to cluster randomised trials, developing countries and public health interventions. The electronic database search was developed first for MEDLINE (ovid) and then adapted for EMBASE (ovid), CINAHL (ovid), PubMed and EMB Reviews (ovid). The full search strategy is available in an online supplement (Additional file 2). Searches were filtered to articles concerning humans, written in English, French or Spanish. We added relevant studies as suggested by the review team. Identified records were uploaded into EndNote software and duplicates were eliminated.

Study screening and selection were performed manually within EndNote. To ensure the availability of study protocols, we limited the search to CRTs that have the word “regist*” in the abstract and used these results to begin the process of screening and selection. We validated this procedure by examining a subset of excluded articles. Screening and selection were performed in two stages by two independent reviewers (MCP and NM). In the first stage, reviewers independently screened the titles and abstracts of each identified reference against the inclusion criteria to eliminate irrelevant publications. In the second stage, each reviewer independently screened the full text of all studies that appeared to meet the inclusion criteria, or for which eligibility was uncertain. For each study, reviewers identified additional articles of potential relevance by reviewing references from the main trial report, consulting the trial registry record and searching the PubMed database for new publications by the lead trial author or using trial identifiers. To aid in article screening and selection, the team developed and tested a full text screening sheet. Any disagreements between reviewers were resolved through discussion and, as necessary, through arbitration by a third author (MJ). A summary table describing studies excluded with reasons is available in an online supplement (Additional file 3).

The search and selection process for this review was designed to identify two quantities required for calculation of outcomes based on proportions, namely (1) numerator: these are studies that meet all inclusion and exclusion criteria. As for all systematic reviews, these studies were our principal focus and were included in the review. (2) Denominator: this is the total of studies for this review (N), which we defined as all studies that satisfy all inclusion and exclusion criteria, with the exception of the outcome criterion (planned or report IF assessment).

The primary outcome for this study was the proportion of overall agreement between the protocol and trial report concerning occurrence of IF assessment. This corresponds to research question 4. Data were summarised in a two-by-two table comparing assessment of intervention fidelity in the trial report to that in the protocol. ‘N’ represents the set of recently published CRTs of public health interventions in LMICs that have registered the study protocol in a publicly availably trial registry. For each CRT in N, we determined whether IF was assessed in the registered (or published) protocol or in the trial report (or associated documents). Studies judged to have assessed IF were coded as “1”; others were coded as “0”. The proportion of overall agreement is defined as the proportion of eligible CRTs for which the protocol and the trial report agree in the practice of intervention fidelity assessment (i.e. both positive or both negative). It was computed as (a+d) /N.

To address research questions 1, 2 and 3, we also calculated:

We used the Cochrane Collaboration tool to assess the risk of bias in individually randomised trials [40] and included additional criteria relevant to assessing risk of bias in CRTs [40‐42], which are available in an online supplement (Additional file 4). We assessed each criterion as low, high or uncertain risk, and provided a sample text that illustrates the reasons for this judgment. The initial evaluation was performed by the primary reviewer (MCP or NM) and verified by the other reviewer (MCP or NM). In case of doubt, a third reviewer (MJ) was consulted. To investigate whether study quality influences IF practices, we stratified the sample to create categories of low risk studies (7 or more of the 10 criteria were evaluated as low risk, n = 36 studies), versus uncertain or high-risk studies (6 or fewer of the 10 criteria were evaluated as low risk, n = 29 studies). Cut-points were chosen empirically to create groups of roughly equal size.

A narrative synthesis was used to explore relationships and findings within and between the included studies. As our review focuses on methodological issues, we did not assess potential publication bias.

The electronic search identified 6876 publications, of which 226 potentially relevant studies were assessed in full text. Stage 1 of the full text review was used to define the study denominator, as follows: of the 226 articles screened, 90 were CRTs of public health interventions in LMICs with a study protocol in a publicly available trial registry published from January 2012 to May 2016. In Stage 2 of the full text review, we reviewed these 90 studies to identify those that in addition considered IF in either the protocol, the trial report or both. We excluded 25 studies (28%) that did not plan or report assessing IF. The remaining 65 studies addressed at least one IF dimension [43‐107]. Figure 1 shows the PRISMA flow chart.

The 65 studies were published in English between January 2012 and May 2016. The included studies are very diverse in terms of sample size and target population, ranging from 190 to more than 280,000 eligible participants per study. Table 2 presents general characteristics of the studies included in the review. Characteristics of the individual studies included are summarised in an online supplement (Additional file 5). Interventions reviewed addressed diverse public health aims; a detailed classification is available in an online supplement (Additional file 6).

Of 65 studies assessing IF, 71% (46/65) described fidelity assessment only in the main trial report, 20% (13/65) of studies published a process evaluation, and 9% (6/65) of studies detailed IF in a complementary document. We found that fidelity components and moderating factors were not systematically assessed in CRTs (Table 4). In addition, only 28% (18/65) of studies monitored events taking place in control groups.

Among the 65 the CRTs reviewed, a variety of procedures were used to assess, maintain and monitor intervention fidelity, including (1) training in intervention delivery, (2) consultation of manuals and standardised guidelines, (3) regular meetings between personnel involved in intervention delivery and researchers to provide feedback, support and monitoring during implementation, (4) logistical and management support for trial coordinators, (5) support teams composed of local area and site managers, (6) supervision and monitoring to ensure data quality, and (7) field notes, self-reported evaluation or checklists to verify implementation progress. Researchers often reported the strategies used to promote intervention fidelity but failed to describe methods used to assess IF [108].

All studies reported accounting for clustering in the analysis. Due to the nature of the interventions, 74% (48/65) of the studies presented a high risk of bias in the category of ‘blinding of participants and personnel’. Only one study (1.5%) had low risk of bias for all dimensions evaluated [46]. The risk of bias graph for individual studies is available in an online supplement (Additional file 7). The overall risk of bias summary is reported in Fig. 2.

To better understand heterogeneity of results concerning IF assessment, we compared studies at low risk of bias versus those at uncertain or high risk of bias (combined). We found no clear differences in IF assessment between subgroups. Results are available in an online supplement (Additional file 8).

The strengths of this review include publication of the study protocol, adherence to the PRISMA guidelines, the use of five bibliographic databases providing comprehensive information on randomised trials [110], and recognising that word limits for scientific journal articles are highly constrained and that the current CONSORT reporting guidelines for CRTs do not require a description of elements related to IF, extending the search for IF assessment to associated documents beyond the main trial report.

Our study also has limitations. First, we included only CRTs that have the word “regist*” in the abstract. Therefore, we may not have identified all CRTs. However, we reviewed 16 excluded studies to assess this risk but none met our inclusion criteria. Thus, the effect of this limitation is likely to be small. Second, according to the original protocol [34], we were unable to compute outcomes for research questions 4b (Are trial reports with negative findings for the ITT analysis more likely to report a per-protocol (PP) analysis?), 4c (What is the overall agreement between ITT and PP analyses concerning intervention effectiveness?), and 4d (Does the magnitude of the intervention effect differ for PP as compared to ITT analyses?) for two main reasons. Firstly, it was not always possible to determine whether articles had performed an ITT analysis to assess the intervention effect. Although the CONSORT statement requires that authors indicate whether analysis were performed on an ITT basis, 16 (25%) studies did not explicitly declare the analysis as ITT or PP. These findings confirm that ITT analysis is “often inadequately described and inadequately applied” [111] in relation to missing outcome data in RCTs [111, 112]. Secondly, many articles reported more than one primary outcome; 58% (38/65) of the studies had more than one primary outcome, while 42% (27/65) of studies have results for one primary outcome. Within studies, conclusions regarding the effectiveness of interventions sometimes diverged; 55% (36/65) of studies reported at least one outcome effective.

This is the first published systematic review focussing on methodological issues related to fidelity assessment in CRTs. Previous reviews have shown that the information required to ensure internal and external validity is often poorly reported in CRTs [18, 113]. This study complements previous reviews by documenting considerable variation in the practice of fidelity assessment for CRTs of public health interventions in LMICs and suggesting the potential need for methodological improvement in this area.

This review demonstrates that fidelity assessment is not currently a systematic practice in CRTs. Public health interventions are complex and may be affected by multiple contextual factors affecting the internal and external validity of the results. In CRTs, contextual differences at the cluster level could favour local adaptation of the intervention and differences in participants’ exposure to intervention components [114, 115]. This variability can lead to the heterogeneity of fidelity, reducing power to detect significant effects where they might be present. Fidelity assessment could help to fill this knowledge gap to aid researchers in attributing outcomes to the actual intervention and identify contextual factors that influence intervention in CRTs [18, 20, 27]. However, current reporting guidelines for CRTs offer no specific guidance on intervention fidelity assessment.

Failure to consider fidelity assessment in CRTs can be explained by several factors. First, approaches to assess intervention fidelity within randomised trials have not yet been made accessible to practitioners. Carroll et al. synthesise previous work [23, 25, 28, 109, 116], establishing a conceptual framework to evaluate intervention fidelity [116‐119]. However, this framework does not provide guidance on methods to assess specific fidelity dimensions in practice. Second, fidelity assessment is not required at present by the CONSORT guidelines for CRT reporting. This does not encourage researchers to think about the importance of assessing intervention fidelity, nor to report their evaluations if conducted. Third, scientific journals limit the number of words in the final report, which may restrict authors’ ability to report their evaluations.