Progression criteria in trials with an internal pilot: an audit of publicly funded randomised controlled trials

In the financial year 2015/16, the National Institute for Health Research (NIHR) spent £247.9 million on their research programmes and almost a third of this was spent on their Health Technology Assessment (HTA) programme [1]. With such large amounts of public money being spent on health research, it is important that the funds are used wisely and that money is not wasted on trials which are not likely to succeed.

An internal pilot is a phase in a trial after which progress is assessed against pre-specified targets/criteria [2, 3]. They are an opportunity to stop trials which are not likely to reach their recruitment, retention or site set-up targets (among others). Unlike an external pilot, data collected during the internal pilot phase contribute towards the final analyses of a trial. This makes internal pilots potentially more cost-effective than running an external pilot followed by a full trial. Including an internal pilot in a study allows funders to take on more risky trials, such as trials where recruitment to time and target is uncertain due to a lack of previous research in the clinical area or a trial population with a rare disease; if the internal pilot shows that the trial is not feasible, it can be stopped short to save resources.

Internal pilots give trialists the opportunity to investigate other elements of the trial such as a sample size re-estimation [4], assessments of futility and adherence to intervention. However, for the purpose of this research the focus is on internal pilots as a means to evaluate or monitor study recruitment, and other progression criteria specified by audited trials were ignored.

Trials fail for a variety of reasons including: not recruiting the target sample size; higher levels of drop-out or non-compliance than planned for; or flaws in the trial design such as an impractical method of randomisation. Recruitment is a key area of interest since failing to hit a recruitment target could leave a trial with less power to detect a clinically meaningful and statistically significant result. According to an audit of randomised controlled trials (RCTs) funded by the NIHR HTA programme, only 56% (85/151) of studies achieved their target sample size [5].

This article aims to provide a summary of the continuation criteria used in trials with internal pilots funded by the NIHR HTA programme as well specific examples of good progression criteria.

Through documentation of funding decisions found on the NIHR webpage [6], 242 studies were identified. Of these, 134 (55.4%) were included in the audit. Reasons for exclusion included the following:

Efforts were made to contact study managers for studies where the protocol was missing from the NIHR Journals library, but it was not always possible to obtain the information needed. The CONSORT-style diagram in Fig. 2 shows the flow of studies through the audit, including reasons for exclusion from the study.

On average, 26.8 (12.19) trials were approved each year and met our inclusion criteria. However, only funding decisions made in February were available for 2012. This means it is only sensible to look at data from 2013 to 2016 when assessing trends across the years but the data from 2012 have been included for completeness. For all other years, data were available for at least 3 months, with data for 1 month more available in 2013 which explains the slight increase in studies from that year.

Out of the 134 studies included in the audit, 72.4% (97/134) said that they would include an internal pilot in their protocol. Of those that did not include an internal pilot, 37.8% (14/37) had done an external pilot prior to their study approval. This means that 82.8% (111/134) of studies audited included some form of pilot/feasibility work.

The Big CACTUS study [10] approved in 2013 included an internal pilot phase having already completed an external pilot [11]. This was due to significant changes made after the external pilot, including the addition of another arm to the trial.

Table 1 presents the properties of the studies included in the audit, broken down by whether an internal pilot was included. There appears to be no difference between the types of studies with or without an internal pilot.

Figure 3 shows the breakdown of how many studies included pilots (either internal or external) by year of funding approval. The proportion of studies including an internal pilot has increased over the years but 10.0% (3/30) of RCTs approved in 2016 still did not have any pilot work either internally or externally.

Of the 97 studies that indicated in their protocol that they would include an internal pilot, 89.7% (87/97) gave criteria for progression from the pilot phase to the full trial. Table 2 presents the proportions of studies using stop/go or red/amber/green systems and whether the number or the rate was given for targets involving recruitment. The most popular format for criteria was a stop/go system with a recruitment target given in terms of the number of patients to be recruited; this was seen in 44.8% (39/87) of the studies that gave criteria.

In 2016 there was an increase in studies adopting the red/amber/green structure for their progression rules, with 61.5% (16/26) of studies using the system compared to just 11.8% (2/17) in 2015. There was also a rise in the number of studies giving a target recruitment rate in their progression criteria: 42.3% (11/26) in 2016 compared to 35.3% (6/17) in 2015. Figures 4 and 5 show the change in the proportion of studies using different types of criteria across the years. Proportions were calculated excluding trials with an internal pilot but no progression criteria stated in the trial protocol.

Table 3 presents details of what proportion of a trial is used for the internal pilot phase. The average proportion of recruitment months used in the pilot is 33.5% (SD 12.6%). However, the average proportion of the sample size aimed to be recruited in the internal pilot was 18.5% (SD 10.4%).

A typical study with an internal pilot having a target sample size of 510 over 24 months aims to recruit one-fifth of their total target sample size within the first one-third of their recruitment time; this ratio of proportion of sample size to proportion of recruitment length (3:5) allows for slow initial recruitment during set-up of centres. We are not looking here at a within-site lag to recruitment (i.e. a slow start caused by staff familiarising themselves with the protocol) but, rather, a lag caused by the process of setting up multiple centres whilst recruitment is ongoing. In this sense, we would expect larger multi-centre studies to have a longer lag phase across the whole study resulting in a lower, more generous ratio of the proportion of the sample size to proportion of recruitment length and we would expect a more ambitious ratio, closer to 1, for studies with only a few sites which could all be set-up and recruiting close to the start of recruitment.

However, there appears to be little association between this ratio and the total number of centres involved in a study. Figure 6 suggests that studies with few centres (< 7) have a fractionally more ambitious ratio than studies with more centres but there is no clear association. This suggests that studies with fewer centres are not being ambitious enough with their recruitment target for their internal pilot, although most studies do not aim to open all of their sites during the internal pilot phase and we cannot exclude the possibility that overall site set-up has been allowed for in the recruitment target.

However, the target proportion of patients recruited for the internal pilot did vary depending on the relative length of the internal pilot compared to the full trial.

Figure 7 shows that, as expected, the larger the proportion of the recruitment period included in the pilot phase, the larger the target sample size for the internal pilot as a proportion of the main target sample size. For example, studies whose internal pilot took up less than 25% of their recruitment months aimed to recruit 9.8% of their target sample size in this time, whereas studies whose internal pilot took up between 33.3% and 41% of their recruitment months aimed to recruit 15.3% of their target sample size.

Coming up with progression criteria can often feel like an abstract concept and pre-specifying desired recruitment rates along with thresholds at which changes should be investigated or the trial should be stopped is difficult. It is helpful to look at examples to get a picture of what clear criteria look like. The following trials have given well thought out progression criteria with a red/amber/green structure and criteria based on rates.

Although they are the gold standard, clinical trials are an expensive way of making an assessment for a new health technology. To ensure optimal value for the research funding—particularly for public funding—it is good practice to have some formal decision criteria within a trial to help the trialists and the funding body decide whether it is feasible to continue with the trial.

Stop/go criteria within the trial are a method to help to determine whether the trial is feasible for the budget set where the criteria are set before the commencement of the trial and are agreed with the funder. In recent years, a traffic light system has been proposed where decisions are set out such that red equates to stop, green to go and amber to further action. This format of setting out progression criteria has increased in popularity amongst HTA-funded trials in the past few years and has the advantage of providing an amber zone that can be used to prompt remedial action rather than close down.

Choice of timing and targets for an internal pilot are important. Regardless of the timing, specifying the decision criteria as an average recruitment rate per month per site allows for a check as to whether the pilot has shown that the trial is feasible. If you multiply the recruitment rate per month per site achieved in the pilot by the number of sites and months across the whole trial you should get the total sample size.

Our investigation has looked only at studies funded by the NIHR HTA programme, as such this work has limited generalisability but will hopefully be of use to those preparing grant applications for this and other publicly funded programmes where funding is limited and there is pressure to obtain results from resources used for research. It would be interesting for further work to explore whether similar trends in internal pilots have been seen across trials funded by other streams of the NIHR and other funding bodies.

A limitation of this work is that results presented cannot, in general, be applied to cluster RCTs because they were excluded from the audit due to complexities surrounding recruitment strategies. For example, in some cluster RCTs individuals are not directly recruited; in the PLEASANT trial [14], GP practices were recruited and randomised and routine data were collected through the Clinical Practice Research Datalink (CPRD). Further work to investigate progression criteria for internal pilots within a cluster RCT would have to consider the different models of recruitment used.

Progression criteria for an internal pilot are usually specified but targets vary widely. Red/amber/green systems have become more popular in recent years for specifying targets for progression. If these criteria are used with a target for the average rate per site per months in the pilot phase, this should produce the total sample size when extrapolated across the sites and duration of the full trial.