Can we achieve better trial recruitment by presenting patient information through multimedia? Meta-analysis of ‘studies within a trial’ (SWATs)

Written information in trials has been criticised for length and complexity and a lack of clear structure to help patients find the information they need. Changes based on user testing and information design can produce information sheets that are easier for patients to understand [7‐9], although our previous programme of co-ordinated SWATs found that such changes did not lead to improvements in recruitment [10].

Provision of audio-visual information about trials may be another way to improve the delivery of information and enhance patient decision-making. Previous studies suggest that audio-visual presentation leads to a small increase in patient understanding, but may have less effect on recruitment [11]. Multimedia information is defined by its use of more than one format, and in terms of patient information has generally entailed the digital presentation of a combination of written text, recorded speech, pictograms and video (including scenario portrayal and animations). It can increase levels of user attention and engagement, not only through the choice of format, which allows users to ‘personalise’ or tailor the information, but also through dual channel stimulation and efficient cognitive demands [12]. It may better meet the needs of an audience increasingly accustomed to obtaining information digitally. There is some evidence that this can increase comprehension within the research consent process [13‐15].

Reviews of the impact of multimedia interventions on research participation have explored a variety of outcomes, including knowledge and understanding, recall, willingness to participate, perceptions of the value of research and decision-making outcomes. Only a small number of studies explored the effects of multimedia materials in the Cochrane review on improving recruitment to trials [16] and the overall conclusion was of uncertainty concerning the effects. Given the limited evidence base, further research through co-ordinated SWATs is clearly warranted. Testing the same strategy across multiple trials provides the opportunity to produce both a more precise estimate of effect and some exploration of whether effects vary by trial type. In this paper, we synthesise SWATs of multimedia use to test their effects on recruitment.

Multimedia content was generated by team members, informed by a review of factors identified by patients as determinants of decisions about trial participation [18], as well as input from patient and public involvement (PPI) contributors and qualitative experts on patient health experiences.

Content included study-specific information (e.g. study purpose, risks) and generic information (e.g. confidentiality). PPI contributors and qualitative experts developed study-specific components involving bespoke themes such as investigator details and benefits of participation. Generic information components included information on informed consent, randomisation and confidentiality. Existing video clips of patients discussing their experiences of participation were edited for length and carefully matched to these components. Additional file 1 provides an example of the range of material presented, with text and video material in separate tabs for the particular trial (for example, information about this study’, ‘what will happen during the study’) and trials in general (‘why get involved’, ‘leaving a study’). The material was designed to provide a much more flexible set of options for patients in terms of how much information they accessed, and in what order, as well as being designed to be more accessible and engaging. The multimedia intervention was developed by a commercial company for use on a range of platforms including desktops and smartphones.

Access to the multimedia resource was provided within the patient information sheet, with a URL link and QR code to assist with easy access (see Additional file 2 for the presentation of the resource to patients). Although we randomised participants to access the multimedia, it was entirely the choice of the participant whether they actually engaged with the multimedia information (alongside the written information) as part of their decision-making process about the trial.

In each SWAT participants being approached to take part were randomised to receive the multimedia alongside written information or standard written information alone. Individual randomisation was used where possible to maximise power and precision and minimise selection bias, but we adopted cluster randomisation where preferred by the host trial for logistical reasons.

The primary outcome was recruitment, defined as the proportion of participants recruited and randomised to a host trial following an invitation to take part. The denominator for the outcome was the total number of potentially eligible participants offered entry to the trial. Depending on the particular trial, this would include a mix of eligible and ineligible patients according to the formal inclusion and exclusion criteria.

Secondary outcomes were:

START was approved by the National Research Ethics Service (NRES) Committee, Yorkshire and the Humber – South Yorkshire (Ref: 11/YH/0271) on the 5 August 2011. Each individual host trial had its own ethical agreement and registration.

Analyses of recruitment were conducted according to a statistical analysis plan. Outcomes were first described separately by study arm and then compared using logistic regression to estimate the between-group odds ratio and corresponding 95% confidence interval. The data from each SWAT were meta-analysed using the Stata metan command (Stata version 14.2) using random effects models based on likely clinical and methodological heterogeneity. Statistical inconsistency was quantified using the I² statistic. In the meta-analysis, we used a two-stage strategy where each individual SWAT was analysed using the appropriate analysis methods (i.e. taking into account whether it was individually or cluster randomised) to generate trial-level summary statistics (e.g. odds ratios) first, and then the results from individual SWATs were combined across trials using the Stata metan command (Stata version 14.2).

We performed pre-specified subgroup analyses investigating differences between studies based on underlying recruitment rates (low defined as a recruitment rate of 5% or below in control group vs. higher rates). We hypothesised that when the baseline recruitment rate is low, the increase in the absolute recruitment rate associated with a recruitment intervention may be higher. A second planned analysis comparing patients with a known diagnosis versus participants ‘at risk’ was not conducted as it proved difficult to assign trials to the categories reliably.

We tested the effects of access to multimedia information on trial recruitment and retention. In a number of SWAT evaluations that run through a diverse group of host trials, the intervention did not improve acceptance, recruitment or retention rates among participants.

This programme of coordinated SWATs was one of the first to be initiated, demonstrating the broad feasibility of this strategy as a model for the more rapid development of an evidence base.

As with most SWATs, there was no formal sample size calculation for individual trials. The host trials undertaking the SWATs were self-selected and therefore the studies on which the programme was run represent a relatively specific (though diverse) group of study contexts, with most patients in the older age groups. It is possible that the variation in those contexts was sufficient to give the recruitment strategy a fair test across multiple designs and populations, and there was limited evidence of significant variation in effect. Even the pooled analysis of data from six trials left some imprecision in the estimate of effect. We have provided broad details of the patient populations sought for each of the trials, but we do not have detailed information on the demographics of those who took part in the SWATs. It is important to note that the participants in the SWATs are not the same as those taking part in the trials, as many people participate in the SWATs without entering the trial. Collecting such information is complicated and potentially burdensome for trial teams. One study was unable to provide the necessary data for our main analysis, which reduced the sample size available. In our experience, problems with delivering SWATs are fairly rare but these difficulties do highlight that SWATs can stretch the resources of already busy trial teams.

Although we planned to assess the use of the multimedia intervention, including its various elements, to provide better context to our outcome data, an error in the web-hosting software meant no accurate data on use were available. The SWATs only provided patients with a link to the multimedia resource and did not actively encourage use. It is unclear whether the intervention failed because it was not accessed, or because access to the intervention had limited impact on patient decision-making. We were unable to assess whether uptake and engagement varied across trials, or between different patient groups in individual trials.

The study was conducted pre-pandemic and the increase in the use of remote tools (including in the delivery of trials) may impact the future effectiveness of multimedia presentations in the context of trial recruitment. The creation and dissemination of the evidence was far from rapid, given recruitment began in 2012. This reflects a number of issues, including the fact that some SWATs extended beyond the funded START programme itself (hampering the completion of the meta-analysis). Some individual SWATs took a significant amount of time to complete recruitment or provide recruitment and retention data. Development of SWAT processes since that time has highlighted the need for greater efficiency, permitting faster publication of individual studies and ‘living’ meta-analyses at the level of a recruitment or retention strategy to better inform the trials community.

The participating trials were led by experienced investigators and teams, so the standard information sheets may have already been well designed, leaving less scope for improvement through intervention. To simplify ethical approvals, we compared our intervention plus standard information with standard information alone, but this may have reduced the impact of the multimedia compared to a comparison of multimedia versus standard information. Further developments in technology and media may mean that future iterations of these types of interventions could include more features and greater interactivity which might enhance effects (albeit at increased cost). All the host trials were done in the UK, making it unclear how applicable this evidence is to other countries.

We report here a linked series of pre-planned and co-ordinated SWATs testing the same recruitment intervention, rather than a retrospective systematic review of all relevant studies using this strategy. The studies reported here will eventually be integrated into the ongoing Cochrane review on strategies to improve trial recruitment [3], alongside similar data from studies outside the START programme. It is possible that access to multimedia has positive benefits on patient understanding, but that does not translate to improved recruitment. Assessing these sorts of impacts through a SWAT is difficult and qualitative research or process data may be required to explore such effects. It is also possible that multimedia and non-written information may be more effective for recruitment of some specific populations, for example some ethnic groups [25].

Most of the studies included in the current meta-analysis were restricted to adults. A linked study has explored the impact of multimedia on recruitment in younger populations. Data from three SWATs found that participants allocated to multimedia were more likely to be recruited to the host trial than those allocated to written information alone (OR 1.54; 95% CI 1.05, 2.28; p = 0.03), although multimedia did not show any impact on measures of decision-making, and the combination of multimedia and written information showed no comparative advantage [26]. Decisions about further SWAT evaluations of this technology can be based on published guidance [27], combining the results reported here, those in the TRECA study, and additional studies of this technology identified by the forthcoming Cochrane review update.