Identifying research priorities for effective retention strategies in clinical trials

Two different surveys of HTA chief investigators and registered CTUs were used to capture current retention practices. HTA chief investigators were surveyed to provide a clinical perspective of retention-based strategies used within specific trials representing a range of disease areas. In contrast, registered CTUs were surveyed to identify non-clinical expertise and insights pertinent across a wide range of trial designs. Whilst it is possible some HTA-funded trials engaged the expertise of registered CTUs, the two surveys were designed to elicit responses from the different perspectives.

A cohort of 76 NIHR Health Technology Assessment Programme (HTA)-funded randomised trials were identified through their website portfolio [15]. The NIHR HTA portfolio was chosen as it represents the largest public funder of current and ongoing trials and the start dates ensured at least 1 year’s recruitment following any pilot work or delays obtaining governance approvals. Two-arm, parallel trials were included. Exclusion criteria were pilot studies, patient preference trials, phase 1 and 2 trials, and studies evaluating longer-term follow-up of previous trials and substudies. This cohort was identified as part of a wider project on retention. Parallel trials with more than two arms were also excluded for consistency across linked projects.

Chief investigators were surveyed to identify whether retention concerns were identified or known at the trial outset and which missing data strategies were implemented either during the trial design or subsequently during trial conduct (Additional file 1). Survey questions included the causes of missing data within their current trials and whether observed levels of missing data were higher, lower or as expected compared to their estimations at trial outset. Respondents were also asked to recommend the three most effective retention practices based on their experience of clinical trials.

A second survey was emailed to 47 registered CTUs identified from the UK Clinical Research Collaboration (UKCRC) website in August 2014 (Additional file 2). Surveys were initially sent to registered CTU directors to complete or delegate as appropriate. If there was no response, attempts were made to identify contact details for other senior staff such as deputy directors, operational directors, senior trial managers or senior statisticians. Fifty-nine strategies to reduce missing data were collated from the Cochrane review and the published results of previous surveys [16, 17]. Registered CTUs were asked to identify which of these strategies they had used within their trials either during the initial design or later in response to specific challenges. Registered CTUs also had the opportunity to suggest strategies that they felt were missing. For each strategy selected, registered CTUs were asked whether it was used routinely or occasionally and whether they had evaluated its effectiveness. Evaluations were classified as either a formal nested study or an informal evaluation comparing retention rates before and after implementation. Questions also explored how frequently sample size calculations were adjusted for missing data, what percentage of missing data was used and the justification for this.

Both surveys were created in MS Word and distributed via email to allow collaboration within the relevant trial teams and registered CTUs.

One author (AK) categorised free-text fields and analysed binary responses from the surveys using SPSS 22.

A Delphi survey was used to gain consensus amongst the registered CTUs to identify which missing data strategies should be prioritised for future research to evaluate their effectiveness. Registered CTUs were invited to take part in the Delphi survey due to their knowledge across clinical trials so that established priorities would be pertinent to the majority of trials undertaken.

Forty-seven CTU directors, or their proxy from the previous survey, were invited by email to take part in a two-round online Delphi survey. During registration participants were given a unique ID number to allow the survey to be completed across a number of sessions and link responses between rounds.

A list of 67 missing data strategies were compiled for use in round 1 based on the results of the surveys shared with CTUs and HTA chief investigators. Registered CTU responses directed changes to the 59 strategies initially used: three strategies were amended for clarity; four strategies were separated to allow delineation between strategies aimed at sites and participants; and similar strategies were grouped together to minimise Delphi survey burden (Additional file 3: Table S1). Two topics were removed as they were not used by registered CTUs (Additional file 3: Table S2). Twelve new strategies were added, 10 of which were influenced by responses from the chief investigator’s survey. These predominantly focussed on the timing and frequency of strategies, e.g. newsletters, patient contact, site contact and questionnaires and were in added in response to comments in the chief investigator’s survey about ‘regular’ contact.

Participants were invited to score each strategy using a scale of 1–9 based on the GRADE guidelines [18]. Scores 1–3 indicated that research into the effectiveness of the retention strategy was not important, 4–6 indicated that it was important but not critical and 7–9 indicated that research was of critical importance. All survey participants could suggest additional retention strategies during round 1 and could abstain from scoring any strategy.

All participants who completed round 1 were invited to participate in round 2 where they were shown a summary of the group’s scores from the previous round and given the option of changing their individual scores or keeping them the same (Fig. 1). All strategies from round 1 were used in the subsequent round. Additional strategies suggested by participants were reviewed by CG and AK for inclusion in round 2.

Incentive prize draws of £75 and £25 high-street gift vouchers were offered for responses to round 2 within 7 and 12 days, respectively. The aim of the incentive was to improve response rates and minimise the need for reminders. Participants were notified of the incentive within the email invite to round 2.

Consensus was predefined using criteria used in a similar methodological priority setting exercise and core outcome development [5, 19]. Consensus that research was of critical importance was reached if > 70% of scores were 7–9 and < 15% of scores were 1–3. Consensus that research was not important was reached if > 70% of scores were 1–3 as long as < 15% scores were 7–9. All research topics from round 2 were ranked according to the percentage of participants scoring a research topic as critically important (scores 7–9). Where strategies achieved the same score they were then ranked order of the percentage of scores 4–6.

Thirty-three out of forty-seven (70%) registered CTUs responded to the survey of current practices. Twenty-nine (90%) registered CTUs routinely adjust their sample size to account for missing data of which 19 (66%) used evidence from other trials to inform the levels of adjustment, nine (31%) used their own past experience, four (14%) used pilot data, two (7%) used estimated figures from the chief investigator, two (7%) used a standard 20% dropout rate and one (3%) used a best guess (Additional file 3: Table S9).

Newsletters were the most routinely used missing data strategy reported by 23 registered CTUs (70%) (Table 3). A total of 30 registered CTUs reported using newsletters at some point to mitigate missing data, 20 of which used them to communicate with sites, one used them to communicate with patients and nine registered CTUs used them with both audiences.

Nineteen registered CTUs (58%) reported routinely using a timeline of participant visits and the inclusion of a prepaid envelope for questionnaires, 18 (55%) regularly used telephone reminders, data collection scheduled with routine care and site initiation training on missing data. Seventeen (52%) of registered CTUs reported occasionally using routinely collected data, 15 (45%) occasionally used follow-up through patient notes, 14 (42%) contacting GPs for missing data/to trace patients, 13 (39%) patient diaries, SMS (text) reminders and re-imbursement of participant expenses. No-one reported using a crèche service, behavioural motivation, charity donations or public draws such as national lottery tickets.

Thirty-one planned, ongoing or complete nested randomised control trials evaluations were reported for 26 out of the 59 (44%) different missing data strategies. Thirty-five informal evaluations of 23 interventions assessed effectiveness by comparing retention before and after the implementation of a retention strategy. In total, 36 of the 59 listed strategies had had some form of evaluation (Table 3). However, no assessments of site initiation training or a timeline of participant visits had been undertaken despite their frequent use by registered CTUs.

Thirty-five (74%) registered CTUs responded to round 1 of the Delphi survey with 34 (97%) completing round 2. Two people independently responded on behalf of one registered CTU. Both sets of answers were included and we report the analysis for 36 responses in round 1 and 35 responses in round 2.

In round 1 consensus was reached on two strategies; the use of routinely collected data (72%) and site initiation training on missing data (75%) (Table 4).

Participants suggested four new strategies during round 1 which were included in round 2: research nurse teleconferences; the use of patient-reported outcome measures (PROMs) versus clinician-collected outcomes; format of newsletters and mode of delivery; availability of blinded outcome assessors to ensure data availability and quality.

Nineteen (54%) of those completing round 2 of the survey responded by the first prize draw deadline, a further nine (26%) by the second prize draw deadline and three (9%) completed the survey after the incentives ended but before the survey closed. A further four (11%) completed the survey after it had officially closed following email or telephone communication but were also included.

In round 2 consensus was reached for seven interventions (Table 4). Consensus criteria for critically important research were met for: site initiation training on missing data; frequency of patient contact during the trial; use of routinely collected data; frequency and timing of reminders; triggered site training on missing data and the length of time needed to complete questionnaires. Research into Christmas and birthday cards for site staff was found to be of low importance by over 70% of respondents.

The ranked list of research priorities from the Delphi survey provides a ‘roadmap’ to address uncertainties and systematically evaluate key strategies. Six topics were deemed critically important and should be a primary focus of future retention research.

We strongly recommend that future studies take account of the range of causes of missing data reported by chief investigators and previous studies. Retention strategies applicable to a range of trial designs, such as site training, frequency of patient contact and the frequency and timing of reminders, should assess impact on patient withdrawal, patients lost to follow-up and clinical staff failing to record primary-outcome measurements as well as questionnaire response rates.

Many Delphi survey participants commented on the challenges of scoring the importance of retention strategies when these are often chosen to address challenges within specific trial designs or populations. Considering the number of strategies discussed it was not feasible to explore this and participants were encouraged to score strategies based on the trials frequently delivered within their CTU. Consequently, the results reflect a ranked list of priorities that will have the broadest impact across the range of trials currently undertaken in the UK. Future research is needed to explore the effectiveness of different strategies for specific trial designs, settings and patient populations.

To our knowledge this is the first study to develop a research agenda for evaluating the effectiveness of retention strategies and so provides a valuable reference point for future methodological work.

A systematic review and surveys of registered CTUs and HTA chief investigators were used to compile the initial list of retention strategies for the Delphi survey. The two different surveys sought to capture both the practical experience of chief investigators within specific trials and the broader experience of registered CTUs that work across multiple trials and different trial designs. Registered CTUs were invited to complete the Delphi survey in order to create a priority list that was applicable to the broad range of trials represented within the UK.

Whilst the study was researcher-led and did not include public or patient perspectives, all three surveys benefited from strong response rates and the Delphi survey retained the majority of participants. Only one person failed to complete round 2 suggesting that the results are unlikely to be affected by attrition bias. The high response rate is attributed to engaging an active network of registered CTUs that were invested in the topic having previously agreed that it was of critical importance [5].

Analysis of the number of retention strategies evaluated by registered CTUs should be considered with some caution as missing responses could not be identified. However, this information did not influence the Delphi process or the ranking of retention strategies for future evaluation.