A review of current practice in the design and assessment of internal pilots in UK NIHR clinical trials

Running a clinical trial can be expensive and complicated. Anticipation of hurdles at the design stage will optimise trial conduct. One method is to run a pilot trial, mirroring the larger-scale trial but with fewer participants and fewer resources. Pilot trials can impart unique insight into the day-to-day issues in running a trial and can provide a justification for progressing to, or abandoning, the full-scale trial. Different types of pilot trial exist [1‐3]. An external pilot (also referred to as an external feasibility study) stands alone from the main trial and can be used to test the feasibility of elements within the trial including recruitment. Data from the external pilot are not used in the main trial analysis. An internal pilot is a phase within the main trial where data from the experience over a pre-specified period are used to determine the design of the rest of the trial, amended or not from the original proposal. Data from the internal pilot are used within the final analysis. The internal pilot should have clear pre-specified progression criteria for moving on to the main trial.

To optimise the benefit of including an internal pilot the progression criteria must be chosen carefully. Recommendations are made in the literature for developing these criteria including: basing recruitment targets on rates per centre per unit time, as opposed to an overall target number at a set time point, to account for unpredictability in site openings; inclusion of intervention adherence and cross-over; event rates; and review of missing data in terms of primary and secondary outcomes. Inclusion of both the funders and Trial Steering Committee (TSC) in the internal pilot review process is recommended [4].

In terms of choosing the pilot sites, it is advised that these must be representative of the sites to be included in the main trial [4, 5]. The protocol should detail the timing of the internal pilot analysis and outline the progression criteria [2, 4, 5].

Little is known, however, about internal pilot trials in practice [4] and few empirical studies are undertaken yet they are widely recommended by funding bodies. The aim of this study was to provide an insight into current practice regarding internal pilots in clinical trials, specifically the design and review process, by assessing a cohort of studies funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme.

The cohort of clinical trials involving an internal pilot, funded by the NIHR HTA programme, was extracted in March 2017. Trials were identified and data extracted via a search of the NIHR Management Information System by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). All trials from May 2012 that were funded and active or in the publication process with the NIHR Journals Library were included to the point of date of extraction.

After approval from the respective trial chief investigators (CIs) was obtained by NETSCC, the following information was collated: project descriptions; a summary of changes from the first stage; feedback about the full application; monitoring notes and progress report history. Protocols were accessed from the online NIHR Journals Library [6]. Development of a classification for continuation to full trial was carried out by the team and confirmation and checks were discussed with the NETSCC Research on Research (RoR) team. The classification consisted of the following categories: criteria met, pilot passed; criteria not met, pilot passed; unknown whether criteria met, pilot passed; pilot did not pass.

Data were extracted on: choice of progression criteria; the review process; the decision on continuation and any changes made to the design based on the pilot.

Data extracted were quality checked via double-data extraction, carried out for three trials (5%). The second reviewer was chosen on the basis of having had some involvement in the trial, i.e. CI or statistician, in order to assess whether the information available depicted the trial well.

Descriptive statistics were calculated using SAS Version 9.4.

Seventy-four trials were identified as eligible. Seventeen were removed from the cohort (four CIs declined, four organisation contracts offices did not reply, two CIs had not been able to consult their TSC for approval and seven did not respond to the email); a total of 57 trials comprise the cohort. For the 17 trials excluded, four had their pilots extended and the remaining 13 had passed.

The available information for each trial is summarised in Table 1. Sixteen trials had missing project descriptions, summaries of changes from the first stage and feedback about the full application, due to the trials pre-dating the development of the NETSCC Management Information System (April 2012).

The majority of internal pilots were first proposed in the initial stages (Table 1).

Thirty-six trials (63%) specified the progression criteria in the latest available version of the protocol and of the 22 with available previous versions, 19 (86%) specified the progression criteria in all available previous versions of the protocol.

Fifty-four trials (96%) included the target number for recruitment in their progression criteria, 16 (29%) the retention/primary outcome ascertainment rate, 13 (23%) the rate of treatment adherence, 11 (20%) the rate of randomisation, 10 (18%) the number of sites actively recruiting, 8 (14%) the consent rate, 5 (9%) the number of sites opened, 3 (5%) the proportion of eligible participants. One trial did not specify the criteria.

In the majority of cases (91%) the internal pilot followed the design agreed with the funder. Five (9%) pilots did not follow their original design: two internal pilots were extended due to poor recruitment, one of which also amended the progression criteria to remove the first criterion of the number of centres recruiting their first participant and revised the criterion of a target number to be in terms of average patients recruited per site per month; three were delayed – one due to recruitment being put on hold, one due to a delay in recruiting the trial manager and one due to early delays with no further information given (Table 2).

Alongside the funder, the TSC was most commonly reported to be involved in the reviewing of the progression criteria, reviewing 94% of trials. No discussion of, or rationale for which committees were chosen to review the results, or in what order, was given for any study.

Fifty-six trials (98%) continued to the full trial, although 14 did not meet the pre-specified progression criteria (see Additional file 1: Table S1); 5 (36%) of which underwent a second review prior to continuation where new criteria were agreed by the trialists and the funder, 3 (21%) put a recovery plan in place and 3 (21%) continued with close monitoring. For nine trials it was unclear whether the criteria were met. For one trial the decision on continuation was on hold at the time of data extraction due to delays in the pilot period.

Forty-four trials (79%) gave a planned number of sites to participate in the internal pilot, 16 (36%) of which specifically named the sites in the protocol.

Twenty-eight (74%) trials adhered to the plan regarding the number of sites. Ten (26%) trials did not adhere to the plan: six opened more sites due to poor recruitment and four opened fewer sites than planned. There was no information on adherence to pilot site choice for six of the trials.

Changes were made to the design of the main trial as a result of the internal pilot in 14 (25%) trials; see Table 3.

There is increasing use of an internal pilot design to optimise main trial conduct. This review examined the design of internal pilots and how progression criteria were chosen and used to inform progression to the main study. Of 57 trials from an HTA cohort the majority were first proposed in the early stages whether of funder- or researcher-led trials. Although most trials specify the progression criteria in the protocol, a considerable proportion of trials do not (37%), despite recommendations in the literature. It was clear from the included trials that progress to the main trial includes a number of steps and can at times be complex. The decision-making process from the internal pilot to the main trial included a multidisciplinary approach, whereby the funders, trialists, TSC, Independent Data and Safety Monitoring Committee (IDSMC), Trial Management Group (TMG) and sponsor contributed to the progression of a trial. If the reporting of the progression criteria were more transparent in the protocol, this could help to eliminate any uncertainties during the internal pilot stage.

The pilot and feasibility trials extension to the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement states that objectives of the pilot study should be stipulated in the trial report [7]. The objectives were clear for the trials in this review but none provided a specific rationale for choosing the progression criteria; giving the rationale within the protocol may aid decision-making if certain criteria are not met.

Previous work has examined how feasibility trials (i.e. standalone randomised pilot studies) or ‘external pilots’ funded by the NIHR Research for Patient Benefit (RfPB) programme progressed to main studies. In a cohort of 89 trials which had closed by May 2016, 20 (30%) were judged ‘not feasible’, of which 6 (7%) were intending to apply for funding for the main trial [8]. However, in the current review all but one internal pilot, which was still under review at the time of analysis, continued to the main trial, even when progression criteria had not been met. This is reassuring as a standalone feasibility study is deliberately performed when the level of success for the main trial is much more uncertain than when an internal pilot is undertaken. However, our findings also suggest that a nuanced approach is used to assess the viability of trial when reviewing an internal pilot which can include a second review, ongoing monitoring and changes to the design of the main trial in terms of recruitment and outcomes. Although some of the trials underwent further reviews, none specified this in the protocol; consideration should be given to include plans for this in the protocol when pre-specifying the progression criteria in the event that criteria are not met. Sometimes changes to the trial design are invoked by the results of the internal pilot; in such cases these changes should be documented clearly within the trial report [6].

The majority (77%) of CIs agreed to share their monitoring data and only 4 (5%) CIs explicitly declined; this is important for a move towards more transparent reporting of the decision-making process. Given that the pilots in excluded trials either passed the progression criteria or had their pilot phases extended there was no bias in terms of non-response due to non-progression to the main trial.

One limitation of the review is that the data extraction came from various, sometimes incomplete, sources. An extension of the work would be to contact the respective CIs to ask for a review of the data extracted for their trial to confirm validity or to add where data may have been missing. This was not carried out for this review as firstly, it would have been too time-consuming and, secondly, it was assumed that the information provided was sufficient to give an overall summary which was confirmed by the double-data extraction. These findings may not necessarily be extrapolated to other funding bodies; thus, this work could be extended to include trials from other NIHR programmes as well as other funding bodies to compare progression criteria and progress across funders.

Seventeen trials had a qualitative element which either ran alongside the internal pilot or was incorporated within it. Objectives of the qualitative study were mainly focussed around understanding the barriers and facilitators to recruitment and the consent process, including patients’ motives for giving or refusing consent. There were qualitative elements in some of the other included trials; however, these were either scheduled to take place pre or post internal pilot. Further work can be done to assess the impact of these qualitative elements on the decision to proceed to the main trial.

This review has provided an overview of current practice in planning and carrying out internal pilots and gives a behind-the-scenes insight into the review process with the hopes of moving towards transparent reporting in the design and analysis of internal pilots. Protocols should state progression criteria and give consideration to reporting the rationale of each criterion. Consideration should also be given to add plans for second reviews of the progression criteria to the protocol and any changes made to the trial design as a result of the internal pilot.