The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials

The sampling frame consisted of all delegates at the 2-day International Clinical Trials Methodology Conference (ICTMC) on 16–17 November 2015. The conference was open to both those involved and those who have an interest in clinical trials methodology. Approximately 638 people registered to attend the conference across a range of disciplines including trialists, clinicians, statisticians, health economists, information specialists and qualitative researchers. Ninety-five percent of the registered delegates were from the UK and the Republic of Ireland, with the remaining 5% from Australia, Canada, Denmark, France, Germany, Holland and the United States. The main UK research centres represented were Aberdeen, Birmingham, Bristol, Cambridge, Cardiff, Coventry, Glasgow, Leeds, London, Liverpool, Manchester, Oxford and Southampton. Conference delegates were first invited to take part in the survey during the opening plenary session, then by researchers from the INVEST team during breaks. The survey could be completed either on paper or online, with a closing date of 18 December 2015. The survey in full is available in Additional file 1.

There were 106 respondents, of whom 54 (51%) were statisticians, 8 (8%) were health economists and 18 (17%) worked in trial management. These are overlapping categories, i.e. respondents were asked to select all roles that applied to them. All respondents had spent some time working in the area of trials: 86 (81%) for at least 3 years and 32 (30%) for more than 10 years. Ninety-six (91%) respondents indicated that they had been involved in the design, setting up or running of trials (77 (80%) in a clinical trials unit and 9 (9%) in industry). Eighty-five (80%) indicated that they had been involved in trial design, 71 (67%) in trial conduct, 73 (69%) in statistical analysis and 52 (49%) had been involved in undertaking a systematic review of trials. Only three (3%) respondents indicated that they had not been involved in any of these. Full details are shown in Additional file 2: Table S1.

Our INVEST survey indicates a high level of support for the use of evidence synthesis to inform aspects of trial design and analysis. Support was generally high for using a description of previous evidence, a systematic review or a meta-analysis when designing a trial. Fewer respondents indicated support for the use of NMA, decision models and VoI analyses. Only a few respondents (approximately 5%) felt that external evidence about particular parameters should not be used in the analysis of a trial; however, many (up to 20%) did not know if such evidence should be used in practice. Our results indicate some discrepancies between the evidence synthesis methods that people think should be used and what they are using in current practice. In particular, respondents did not appear to be using systematic reviews, meta-analyses, NMAs, decision models and VoI analyses as much as they wanted across all aspects of trial design. The greatest perceived barrier to using evidence synthesis methods in trial design or analysis was time constraints, followed by a belief that the new trial was the first in the area.

The sampling frame was approximately 638 people, but only 106 completed the survey, providing a response rate of approximately 17% and a potential for selection bias. We were unable to obtain information on the characteristics of the nonrespondents which would have enabled us to explore the representativeness of our sample, but it is possible that respondents were more enthusiastic about evidence synthesis methods than nonrespondents. Some 95% of our sampling frame were from the UK and the Republic of Ireland, so the results may not be generalisable to the international clinical trials community. Further, our sampling frame consisted of conference delegates closely involved in trial design and analysis, who are likely to have a strong interest in promoting good practice. As such, we might expect our sample to answer some of the questions more favourably than the wider population of people involved in clinical trials. In particular, half of respondents were statisticians (51%), who may be expected to be more open to advanced statistical methods (such as using evidence syntheses to improve precision in estimates of some parameters) compared with other contributors to the design, conduct or delivery of trials. Statisticians are also influential members of the multidisciplinary teams that are involved in trial design and may be useful advocates for the increased use of available evidence in trial design. Although it would have been interesting to explore differences across research centres and countries, we chose not to collect such geographical data to protect anonymity and minimise the burden of survey completion. To summarise the barriers to the use of evidence synthesis, we assigned scores based on an arbitrary assumption of linearity, i.e. such that an individual’s highest ranked barrier is three times as important as their third barrier. These scores, although helpful for summarising data, might not reflect respondents’ true views. We intended all listed barriers to be interpreted as reasons why a trial team might not seek or carry out evidence synthesis. However, it is possible that some respondents who chose ‘Believed to be the first trial in the area’ could have been thinking of the situation where a literature search or systematic review reveals no previous trials. The extent of this barrier would then be overestimated.

In trial design, for both whether a trial was needed and for choosing an intervention, more respondents said that a systematic review, rather than a less formal description of previous evidence, should be used. It therefore seems that respondents felt the need for a thorough, systematic approach in order to show convincingly whether there is a gap in the evidence base that merits a new trial. For the other aspects of trial design, there may not be sufficient available evidence to warrant a systematic review, so that a less formal description of previous evidence might be felt to be adequate.

The large proportions of respondents who indicated that they had either used evidence synthesis to inform trial analysis or that they believed evidence synthesis should be used in this way were surprising. Even more surprisingly, a sensitivity analysis including only statisticians provided slightly lower estimates of these proportions, although the small sample size precludes strong assertions. We feel that it is unlikely that these relatively advanced methods are being used so frequently in practice. As such, we suggest that many respondents may have interpreted these questions in ways other than intended. This explanation appears to be supported by the result that fewer statisticians than nonstatisticians claim to be using external evidence in this way: it is likely that confusion about these questions was higher among nonstatisticians although we have no direct evidence of this. In particular, respondents might have interpreted the incorporation of ‘external information about the treatment effect (including a meta-analysis)’ in trial analysis as meaning including the new trial results in an updated meta-analysis. Our intention had instead been to elicit views on the use of informative prior distributions in a Bayesian statistical framework. In retrospect, we should have clarified these questions about relatively complex issues using examples, although we were keen to be as concise as possible. We propose that future qualitative research should be conducted to explore the use of informative priors with particular focus on evidence about the treatment effect, potential biases and other quantities in trial analysis [19]. This work should investigate more thoroughly how trialists are currently using evidence synthesis to inform analysis, and the potential barriers to an increased amount of such use. We would anticipate more objections in principle to the use of informative prior distributions compared with less formal uses of evidence synthesis. The qualitative work should explore which types of external evidence might be considered most relevant and useful to trial analysis, and what level of such use might be acceptable in practice.

Funders of clinical trials often highlight the importance of taking into account existing evidence in grant applications [20]. However, it is still unclear how, and to what extent, funders or reviewers expect evidence synthesis to be used. We did not explore the views of funders or reviewers specifically but this could be another valuable avenue for future research, given the critical role that they could play in minimising research wastage.

The INVEST survey provides generally higher estimates of the use of systematic reviews in trial design than the recent review of Bhurke et al. [7], with the exception of ‘justification of the trial’ (Bhurke et al. 94% versus INVEST 73%). For example, 68% of our respondents indicated that they had used a systematic review to inform choice of outcomes and follow-up time, whereas only 16% and 6% of trials reviewed by Bhurke et al. had used a review to inform these two aspects, respectively. Similarly, 51% of our respondents said that they had used a systematic review to inform sample size calculations, seemingly in contrast to the finding of Bhurke et al. that only 9% of trials had used a review to inform the standard deviation and 3% to ‘estimate the difference to detect or margin of equivalence’. It is possible that other trials in the Bhurke et al. review relied on pilot trials to inform these parameters [21, 22], while the INVEST results seem to suggest that relevant information will often be available from evidence syntheses. However, the results are not directly comparable since we asked respondents to consider all trials that they had been involved in during the last 10 years, whereas Bhurke et al. investigated whether evidence synthesis had been used in specific individual trials. On the other hand, Bhurke et al. reviewed only publicly funded (NIHR HTA) trials, while trialists attending ICTMC are likely to also participate in company-funded trials, for which less justification is required and there is possibly a stronger expectation for independently clear results. In agreement with Bhurke et al. we found that important barriers to the use of evidence synthesis in practice include a new trial being the first in its area or being different from trials included in a previous review. However, by directly asking trialists instead of relying on documentation, we were able to see that the greatest barrier is time constraints. In attempt to overcome the issue of time constraints when synthesising evidence, many methods for rapid reviews have been proposed over recent years [23, 24]. Khangura et al. [23] developed their own eight-step approach of conducting a rapid review having reviewed the current literature. Implementation of their approach in HTA trials has been successful and can be applied to other types of trials [25]. However, more training on approximate methods and rapid reviews is needed to support their wider use in practice. Investment in adequate resources and training at this stage could lead to cost savings in the longer term, by reducing waste in research.

We found less support for the use of NMAs, decision models and VoI analyses in trial design which may be because they are more complex to conduct and require a greater investment of time and expertise. These methods could further help inform decisions but also require additional assumptions and ‘a priori’ parameter estimates, such as the cost-effectiveness threshold and parameters related to structural uncertainties in the case of VoI, which may not be available. A policy framework on when, and how, to perform such analyses and how they are used could be a useful next step [26]. We also note that most individual trials investigate a specific research question for one particular treatment: for example, in 2014, 80% of trials were still two-armed trials [27]. In contrast, NMAs, decision models and VoI analyses are commonly used to make decisions and inform policy when there is a choice between a number of concurrent treatment options. These methods could be considered less relevant in the design and analysis of an individual two-armed trial. VoI analyses, in particular, are usually commissioned in high-value trials, often in situations with many treatments and uncertainty as to which is best. However, a NMA could be more relevant to inform the interventions of a two-armed trial if used at the earlier part of the design process [28]. Trial-based economic analysis are sometimes secondary to the clinical aspect rather than being fully integrated within a trial design [29] meaning that the use of decision models and VoI analyses to inform trial design is limited. Only 6% (5/84) of our respondents had used a VoI analysis to inform whether a trial is needed, although all of those who had used a VoI analysis were in favour of its use more generally. Models in health economic analyses are a strongly simplified representation of disease history and treatment effects and are framed around a particular decision setting (e.g. UK) using setting-specific values for health care use, costs and health benefits. These values may change over time and are likely to be different in other settings. Streamlining of decision modelling and VoI analyses would, therefore, be particularly challenging. Despite the recognition that the VoI method does come with its assumptions and limitations, its potential to guide the need for and the design of new studies [30] warrant its wider consideration and further development.

Trial teams responding to the INVEST survey generally reported that they are using evidence synthesis in trial design and analysis more than we might have expected, but less than they might like to. Time constraints was identified as the greatest barrier to more widespread use. Further research on ways to undertake evidence synthesis more efficiently, and training on how to incorporate results from these into existing procedures will help to ensure the best use of relevant external evidence in the design, conduct and analysis of clinical trials.