Providing multimedia information to children and young people increases recruitment to trials: pre-planned meta-analysis of SWATs

We publicised the Trials Engagement in Children and Adolescents (TRECA) study through printed flyers and email to UK trial centres and the UK Trial Managers’ Research Network. We emphasised that the cost of developing the MMI would be borne by TRECA, and we also offered host trials modest financial support (£1500) to compensate for additional administrative work. Interested trials were selected based on the following:

Ideally, we wanted host trials to cover a range of health conditions and intervention types and be recruiting CYP across a range of ages. Given the historical prominence of oncology trials in CYP trial research [49], we preferred at least one host trial to have that setting.

Finally, the sample size calculation for the meta-analysis (see below) meant that host trials needed to approach an average of at least 329 CYP about participation. It is important to note that in these recruitment SWATs, the outcome of interest is the number of participants recruited out of the number of people approached to take part in the host trial. The people to be approached were randomly allocated to different recruitment methods, and the outcome is calculated according to the proportion of people recruited to the host trial from each arm in the SWAT.

Host trials were offered SWATs with two arms (MMI-only versus PIS-only; or MMI-only versus PIS + MMI) or three arms (MMI-only versus PIS-only versus PIS + MMI).

Interventions used in the TRECA study are listed in Table 2. The PIS was the Research Ethics Committee-approved information being used in each host trial; the PIS content for each trial was not modified. The decision on who read the PIS (i.e. parent, child, both together, both separately) was made by individual participants.

The MMI for each trial was based on a template developed through co-design and extensive empirical work during the first year of the TRECA study, comprising a qualitative study with stakeholder groups [50]; a user testing study with CYP, parents, and carers [51]; and close collaboration with the TRECA Patient and Parent Involvement group (PPI) [52]. Two templates were developed (one for children aged 6–11 years and a second for children aged 12–18 years and parents). The MMIs were developed by a website and video production company (Morph Studios Ltd.) and contained all the content from the PIS, organised into six sections within the MMI (home page, about the trial, taking part, after the trial, questions, contacts) plus five short video animations with voiceover (one that was trial-specific, summarising the trial; four that were trial-generic, explaining different aspects of trials), and a series of short ‘talking heads’ videos (e.g. the trial principal investigator or a young participant talking about aspects of the trial including the experience of participation). None of the video clips included subtitles. The number of ‘talking heads’ videos varied among the SWATs (see Table 3 for links to example MMIs). The written text of each MMI was amended for clarity through readability indices [53]. The decision on who viewed the MMI (i.e. parent, child, both together, both separately) was made by individual participants.

The primary outcome was trial recruitment in the MMI-only versus the PIS-only arms, to test the effect of replacing printed information with MMI. Trial recruitment was assessed by the number of eligible participants who were recruited to the host trial, according to the allocated SWAT arm.

The secondary outcomes were as follows:

The TRECA study was registered on the ISRCTN registry (ISRCTN73136092) and the Northern Ireland Hub for Trials Methodology Research SWAT Repository (SWAT 97). The SWATs were undertaken through amendments to Research Ethics Committee approvals obtained by the host trial research teams.

The results from each SWAT were combined in meta-analyses. A two-stage random-effects meta-analysis was used in each case, where the results from each model were combined using an inverse-variance approach; no further adjustments were made. For each of the outcomes, any available data were combined using a meta-analysis.

Six host trials were recruited to TRECA: FORCE, CHAMP-UK, Thermic-3, BALANCE, BAMP, and UKALL-2011 (Tables 1 and 2), recruiting CYP with a variety of ages and health conditions in the UK (Table 1). Three of the SWATs generated sufficient data for logistic regression models, which were combined in a pre-planned statistical meta-analysis (FORCE, CHAMP-UK, Thermic-3). Unfortunately, three SWATs were significantly affected by adverse circumstances: the BAMP trial closed early due to funding restrictions; the UKALL-2011 SWAT took many months to approve, which limited recruitment before the host trial closed; and recruitment to the BALANCE trial was closed for several months due to the COVID-19 pandemic. Consequently, it was not possible to run logistic regression models for these three SWATs, due to being too small (2 trials) or having insufficient variation in outcomes (1 trial). Descriptive results are provided for each of the trials. However, the meta-analysis was based on the three SWATs where models could be fitted (total n = 1758).

The pathway through each SWAT is displayed in Fig. 3 and the results of the six individual SWATs are reported in Tables 4, 5, and 6. The results of the FORCE and Thermic-3 SWATs have already been reported individually [55, 56], and the results of the CHAMP-UK SWAT are reported in Supplementary Material 3.

MMI provision, rather than standard PIS, led to higher recruitment rates of CYP to trials. There was no effect on trial retention. Providing combined MMI + PIS resulted in no benefit on trial recruitment or retention rates compared to PIS alone.

The provision of MMI rather than PIS produced no effect on DMQ scores. However, the provision of combined MMI + PIS resulted in less favourable evaluations than from PIS alone.

The MMIs were developed through extensive empirical work, to ensure that their content, layout, and appearance would best meet the preferences and needs of CYP, parents/carers, healthcare practitioners, and researchers. Two templates were developed (one for children aged 6–11 and one for children aged 12–18 and parents), based on estimated literacy and cognitive maturity. Finer-grained age distinctions would be possible in MMIs although there would be both financial and practical implications.

The six host trials recruited CYP with a range of ages and health conditions across the UK, although we found that many otherwise eligible host trials were too small to include in TRECA. Unfortunately, we were able to include data from only three of the SWATs in the meta-analysis. However, the pre-planned meta-analysis increased the available sample size for evaluating the effects of MMIs, and allowed the comparison of similar interventions (i.e. template-driven MMIs) across several trials.

The study faced several challenges [57], the main effect of which was to reduce the size of the meta-analysis dataset. There was a good response to publicity about TRECA, but some potential host trials did not meet our criteria. In addition, trial recruitment postponements due to COVID-19 affected three SWATs, reducing the dataset size in two of them (by around n = 200 in total) and delaying the meta-analysis. Unfortunately, any adverse circumstance affecting host trial progress will almost always affect SWATs. On the other hand, the sample size estimate assumed a meta-analysis heterogeneity value of 50%, but in fact, a value of 0% was obtained, increasing power.

The DMQ return rates were around half that anticipated, reducing statistical precision. The questionnaire return rate was much higher in CYP recruited to host trials than in those who declined participation. It constitutes a significant limitation to the findings; potentially, it was also a source of bias in the DMQ dataset. Sending questionnaires via website or email (as happened in the CHAMP-UK trial), rather than providing printed versions, could have increased return rates, but this approach has not been evaluated [58]. We could also have sought other data sources on recruitment. For example, we could have interviewed people who did or did not consent in host trials, or clinical and research staff responsible for trial recruitment, although this approach would have had its own challenges.

Finally, a common limitation with SWATs is the application of host trial entry criteria to patient eligibility after the SWAT random allocation of individuals has happened. The effect is to lose SWAT participants and reduce the fidelity of the SWAT randomisation; consequently, a modified intention-to-treat (mITT) analysis was required in this study. In all, 18 participants were omitted from the analysis for being ineligible for the host trial (7 CHAMP-UK, 1 UKALL, 4 BALANCE, 6 THERMIC-3).

This study contributes to the growing evidence base of RCT-level evidence for interventions targeting trial recruitment and retention [45], particularly in CYP, which is currently lacking.

The retention rates of CYP at follow-up in the trials are reassuring, indicating that increased recruitment through MMI was not achieved at the expense of understanding the host trial when being recruited.

One aim of the study was to produce an MMI template for trial recruitment, in part to ensure that all the TRECA SWATs would be testing similar multimedia information, and to allow future practitioners and researchers to use the template in trials if the MMI was shown to be effective and/or acceptable. That opportunity is now available, and access can be provided on request.

There is growing evidence on the use of MMI in research recruitment and healthcare, although few interventions have been evaluated within SWATs using random allocation. As such, this study makes a meaningful contribution.

Finally, this study shows that using MMI in research recruitment is achievable and acceptable. Our view is that a hesitancy among researchers towards digital or multimedia information stems from two concerns:

This study demonstrated a benefit of MMI on trial recruitment in CYP, although the relevant evidence base is small and further research is needed. Furthermore, the effect of MMIs on recruitment generated an odds ratio statistic with the lower end of the confidence interval not far above 1 (with an associated probability of 0.03) and so there remains considerable uncertainty around its true effect. That is particularly the case for CYP, but it also applies to trials recruiting adults.

The combined provision of MMI + PIS resulted in no gains, and possibly less positive questionnaire evaluations. However, the small available dataset for this question greatly lessens the certainty of this finding. Given the time demands and complexity of generating participant information in two formats, this study does not suggest that using both MMI and PIS is a better option, although it would be helpful for further research to assess this point.

One SWAT could not provide participants with MMI access as planned due to poor hospital internet connectivity [57]. Consequently, in that SWAT, participants were given the MMI URL on a laminated card, to access at home, increasing the risk of non-access. A solution could be through social media or email/text message communication between the healthcare provider and patients, although there are several barriers (ethical and practical) that may need to be overcome.

One aim was to assess the relative impacts of MMI and PIS on more deprived and/or less health-literate populations. Suboptimal or complex patient information tends to have a disproportionately negative effect on people with low literacy (including low health literacy) [24]. Unfortunately, we were unable to examine these aspects. Demographic baseline data are generally only available for those who have consented to the host trial. Due to research ethics restrictions, data access is rarely permitted for people who have declined trial participation, placing a significant limit on assessing equity within ‘recruitment science’. There remains a need to assess whether carefully developed MMI has similar benefits on understanding and decision-confidence across population groups. Easy internet access in the population is commonplace but not universal. There is a likelihood of deepening of inequities if information is exclusively provided via MMIs, or both MMIs and PIS are provided but MMI content has greater clarity. MMIs may also have negative associations for some users: video and animations can appeal and engage, although these effects must not be achieved through superficiality or imbalance. Content in different formats may lead to duplication or ‘information redundancy’ [59]. Finally, a recent review identified that CYP with long-term health conditions may associate digital health technologies with concerns about privacy, trust, and confidentiality [60].

Information plays an essential role in research consent. Valid participation decisions (whether positive or negative) must be informed, and the person taking consent has an ethical duty to be sure that the participant is making an informed decision. The role of MMI in individual consent decisions, and in allowing recruiters to fulfil their ethical duty, is an area in need of research. However, the external context is changing: people are increasingly expecting digital provision of information, particularly younger people, and digital provision may also be altering content expectations [61, 62]. These are two of the factors which may increase participant acceptance and use of MMI in trial recruitment.

This study has generated some relevant findings from the DMQs, although low return rates (particularly from those declining trial recruitment) mean that there is considerable uncertainty about MMI evaluations. For example, the findings of a SWAT undertaken within a hypothetical trial were that the TRECA-generated MMI produced higher appraisals of ‘information was easy to understand’ and ‘I had confidence in decision-making’ than did PIS provision [63]. There is also a need for research with adults and CYP around the effects of MMI on communication between potential participants and those recruiting them; MMIs may impact recruiter-participant interaction, but this needs evaluation.

We should also know more about people’s use of MMIs to inform decisions, to ensure they are being informed about research and not just being entertained. Furthermore, it is unclear whether all the components of the MMI are required to achieve benefits. Two concerns are that:

The overall effect of the MMIs in TRECA was to increase recruitment in children and young people (indicated by a meta-analysis of three SWATs with negligible statistical heterogeneity), which indicates their potential as a recruitment tool. However, given the width of the confidence intervals, and a dataset based on just three SWATs, there remains a need for more research to produce a more precise estimate of how well MMIs work and in what settings. The TRECA dataset on combined MMI + PIS was small and further research would increase the certainty on its effects. Finally, we need to explore what components of MMIs are associated with benefits, and whether MMIs are cost-effective.