Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map

Recruiting and retaining participants in clinical and health studies is a key determinant of research efficiency, but is highly challenging [1, 2], and poor rates of recruitment to randomised controlled trials (RCTs) are commonly reported [3]. In reviews of clinical trials funded by the UK Medical Research Council and/or the National Institute for Heath Research (NIHR) Health Technology Assessment (HTA) programme [4‐7] the proportion of trials achieving their original recruitment target ranged from only 31% (of 114 trials that recruited between 1994 and 2002) [4] to 60% (of 151 trials published from 2004 to April 2016) [7]. Although there appears to have been some improvement over time, the latest data [7] suggest that around 40% of trials may still fail to reach their intended enrolment numbers, despite considerable interest in approaches that might improve study recruitment [2, 8].

Once patients have been recruited into clinical or health studies the rates at which they are retained in the studies can be highly variable, depending on the nature of the population, condition, intervention, comparator and outcomes. For example, estimates from trials in Alzheimer’s disease [9] suggest that that there is an average dropout rate of 30% across trials and 85% of trials fail to retain enough patients [10], whilst in a systematic review of 87 RCTs of asthma inhalers we found that dropout rates ranged from 0% to over 40% [11].

Consequences of inadequate trial recruitment include: reduced statistical power of trials due to inadequate sample size, increasing the risk of type II error (failure to detect real treatment effects) [12]; wasting patients’ and investigators’ time and money (e.g. 481 trials that closed during 2011 due to inadequate recruitment had already involved over 48,000 patients [13]); extending study duration which increases costs and delays the adoption of trial findings [12] (clinical studies often double their original timelines in an attempt to meet their intended enrolment target [14]); and failure to identify participants who are eligible (e.g. up to 60% of potentially eligible patients may miss being identified [15], which disadvantages the potential participants as well as the study investigators, sponsors and end-users of the research). Inadequate retention of patients in clinical trials also reduces the available sample size for analysis and hence the statistical power to detect treatment effects [16]. Failure to achieve adequate recruitment and retention of participants in clinical trials therefore creates research waste [17] and may mean that policy-makers have to delay their decisions or base them on inferior evidence.

A survey of chief investigators of NIHR HTA trials and UK Clinical Research Collaboration (UKCRC)-registered clinical trials units (CTUs) identified that a diverse range of participant-retention strategies has been used in clinical trials, but for many of these, no research had been conducted to assess their effectiveness [18]. Another survey found that the top three priorities of UK CTU directors for trials methodological research included ‘Research into methods to boost recruitment in trials’(considered the highest priority) and ‘Methods to minimise attrition’ [19].

To address inadequate patient recruitment and/or retention in studies, CTUs, funding agencies and study investigators are increasingly exploring the use of digital tools to identify, recruit and retain study participants. The range of digital tools that could assist with the recruitment and/or retention of study participants is wide. For example, a systematic review of the use of computers for patient recruitment into clinical trials found 79 different recruitment systems [20] and a survey of trials and CTUs identified a wide range of retention strategies [19]. Digital recruitment approaches may help potential participants to locate trials for which they are eligible, and/or help trial investigators or health professionals to identify potentially eligible participants. Specific digital tools for recruitment and/or retention of people in trials include (among others): automated telephone messaging [2, 21]; audio messages [22]; videos [2, 22]; radio and television advertisements [23, 24]; online advertisements [23, 25]; Internet websites and tools, including online surveys [21, 24, 26, 27]; social media [21, 23, 25, 27, 28]; ‘smartphone Apps’ [25]; computer pop-up reminders [29]; email messages [2, 21, 25]; text-messaging [2, 21, 23, 25] and automated eligibility screening of electronic health records, data warehouses or other patient data sources [15, 30‐36]. The automated screening approaches may be subdivided according to the different algorithms employed for predicting patient eligibility, which include, among others, machine-learning approaches [34, 37] and case-based reasoning models [32, 33]. These digital tools may either be used alone or in various combinations, and may be combined with non-digital approaches. For example, a strategy to improve patient recruitment, enrolment, engagement and retention in a clinical trial of a weight-loss intervention for students included a smartphone App, television screens, email messaging, text-messaging, online advertisements and social media, as well as printed materials (flyers, coasters, pens, posters and postcards) [25]. In another example, an intervention to increase research participation of young female cancer survivors included Internet, email and social media components as well as newspaper advertisements that appeared both online and in print [38].

The range of digital tools potentially available to assist with the recruitment and/or retention of patients in clinical and health trials is diverse, and a number of systematic reviews have investigated whether digital approaches for recruitment and/or retention are effective [2, 3, 20, 39‐53] (summarised in Additional Table 1). However, it is difficult to get a comprehensive picture of the characteristics of the overall evidence base, because:

Given the growth in the availability of digital tools in the last decade and the importance of effective and efficient recruitment and retention, we constructed a systematic map to investigate what evidence exists to support the use of these tools.

The diversity, uniqueness, and, in some cases complexity, of strategies for improving patient recruitment and retention suggest that systematic mapping is an appropriate initial approach to the evidence synthesis, to identify and characterise the range of digital tools for improving patient recruitment and/or retention and characterise the studies that have evaluated these tools. Systematic mapping (NB also referred to in the literature as evidence mapping) begins in the same way as a systematic review, based on extensive searches for evidence and systematic screening to determine eligibility of the identified studies, but provides a descriptive output rather than an estimate of effects [55]. Systematic maps are helpful in summarising and describing a broad and/or heterogeneous evidence base in order to plan and prioritise focussed evaluative syntheses, e.g. using one or more subsequent systematic review(s) [55‐62]. Evidence gaps can also be identified where little or no primary research has been conducted. We chose systematic mapping as the method for this project because no comprehensive overview of the evidence base relating to the use of digital tools for trial participant recruitment and/or retention currently exists.

The question addressed by the systematic map is ‘what are the types and characteristics of the digital tools that have been evaluated for improving the recruitment and/or retention of people in RCTs?’

The aim of this research is to systematically identify and describe the research studies that have evaluated the accuracy or effectiveness of digital tools for improving the recruitment and/or retention of people in RCTs, and to describe the characteristics of those digital tools that have been evaluated.

This research has two objectives:

This research was part of a broader project ‘User-focussed research to identify the benefits of innovative digital recruitment and retention tools for more efficient conduct of randomised trials’. The project had three parts: Phase 1: Scoping searches to identify digital tools for patient recruitment and retention. Phase 2: A survey of the UKCRC-registered CTUs on what digital tools they currently use, and interviews with stakeholders to identify which digital tools for trial recruitment and retention are currently being used in practice and the performance characteristics required for digital tools to be judged useful [63]. This informed the development of logic models to capture the functions of different types of digital tools for recruitment and retention and the theory behind their mechanism of action in a generic way, without focussing on any specific digital tool, product or service. Phase 3: Systematic mapping to identify and describe the characteristics of research studies which have evaluated the effectiveness of digital tools for patient recruitment and/or retention. The research we are reporting here is specifically on the systematic mapping work (Phase 3).

The systematic map methods were based on a protocol (Appendix 1; see Additional file 4) which was peer-reviewed by the full project team and an Advisory Board prior to the commencement of the work. The Advisory Board included representatives of the NIHR research infrastructure (i.e. Research Design Service and Clinical Research Network), clinical trials units, a patient and public involvement representative, and academic members of a National Health Service (NHS) Foundation Trust (see the ‘Acknowledgements’ section). The systematic mapping was conducted by a team experienced in evidence synthesis (the authors of this paper) who met monthly with, and received feedback from, the full project team. The work was conducted in accordance with those steps of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Checklist [64] that are applicable to systematic mapping (see Additional Table 2).

A comprehensive search for relevant studies was undertaken by an experienced health information specialist in the following bibliographical databases:

The search strategies for MEDLINE, Embase, Web of Science and Inspec searches are shown in Appendix 2 (see Additional file 5). Where possible, the databases were searched from 1990 onwards, as specified in the protocol (Appendix 1; see Additional file 4). As noted below (see the ‘Eligibility criteria’ section), a further date limit was subsequently applied during eligibility screening.

All searches were conducted in the English language.

Titles and abstracts of references identified by the search were screened against the eligibility criteria (stated below) independently by two reviewers. In cases of disagreement a third reviewer was consulted to reach a consensus. Full-text articles were retrieved for those titles and abstracts which met the eligibility criteria or which had unclear relevance, and for references that did not have an abstract or summary (e.g. Internet pages and reports). Full-text articles were screened by one reviewer and checked by a second, and a third reviewer was consulted in cases of disagreement. The screening process is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline [65] (see the ‘Results’ section).

References were screened using an eligibility worksheet, which was pilot-tested on 29 studies identified by our scoping searches [16, 18, 28, 35, 61‐85] to ensure consistency of the review team’s interpretation. The final eligibility criteria (Appendix 3; see Additional file 6) were then applied to all the identified references. The same eligibility criteria were applied for screening titles, abstracts and full-text articles.

All studies meeting the criteria for inclusion in the map were classified by systematically applying pre-specified keywords to each study. The purpose of the keyword list was to capture the characteristics of the evidence base in a flexible manner such that the key attributes of studies of interest to end-users could be clearly summarised in the map, and specific combinations of characteristics of interest explored. The map, and hence the keywords, did not seek to critically appraise, synthesise or evaluate the results of the studies, for which systematic review and meta-analysis would be more appropriate evidence-synthesis methods. The draft keyword list and coding process were initially pilot-tested on eight studies that were identified in scoping searches (reported in five papers [21, 31, 67‐69]), to ensure that the map would consistently capture relevant information. The draft keyword list was refined based on consultation with the full project team and the project Advisory Board (which we consider representative of most stakeholders likely to consult the map), to give a final version (Appendix 4; see Additional file 7).

Each study was coded by one reviewer and a random sample of 20% of the studies was checked by a second reviewer. The coding decisions for each study were recorded in a Microsoft Excel spreadsheet template, which contained the list of keywords (rows) and a list of the included studies (columns) to generate a data matrix that would form the systematic map database. In cases where checking identified that refinements to coding would be appropriate these were agreed by the review team and applied to all the included studies to minimise the risk of introducing bias. Once the coding of all the included studies had been completed, Excel chart and pivot table functions were applied to the data matrix to produce a descriptive map of the key characteristics of the evidence.

A flow diagram summarising the eligibility screening process is shown in Fig. 1. After removing duplicates, searches identified 9163 unique references published from January 2008 to mid-July 2018. The majority of these (n = 8806) were excluded because their title and/or abstract did not meet the eligibility criteria, leaving 357 articles for full-text screening. A further 251 articles were excluded at full-text screening (Fig. 1) (a full list of the 251 excluded studies and the reasons for exclusion is provided in Additional Table 3). The remaining 104 articles, reporting 105 unique studies, passed the full-text assessment and were included in the systematic map [28, 32‐36, 70‐167].

The full systematic map database is provided in Appendix 5 (see Additional file 8).

The 105 identified studies of digital recruitment and retention approaches have nearly all been conducted in single countries (100/105; 95%), with five multi-national studies identified (5%). The 100 single-country studies were conducted most often in the USA (61%), UK (17%), Australia (9%), Germany (4%) and Canada (3%), with a slight increase in the diversity of countries involved over time (Fig. 2).

The health areas studied most often were health promotion and public health (37/105; 35%), cancers (17/105; 16%), circulatory system disorders (13/105; 12%), mental health (10/105; 10%) and endocrine and metabolic disorders (7/105; 7%) (Fig. 3). Note that the total number of studies in Fig. 3 is more than 105, since several studies covered more than one health condition.

Studies on health promotion and public health and on circulatory system disorders have recently increased, but those on cancers, mental health and endocrine and metabolic disorders have remained relatively infrequent (Fig. 4).

The systematic map (Appendix 5; see Additional file 8) shows that the most common specific health topics that the 37 studies on health promotion and public health investigated were smoking cessation or tobacco control (11 studies) and sexual health promotion (seven studies).

Most of the studies (96/105; 91%) have investigated digital approaches for recruitment, with fewer (20/105; 19%) investigating digital approaches for retention, and 11/105 (10%) investigating both recruitment and retention (85/105 studies (81%) were on recruitment only and 9/105 studies (9%) were on retention only). The greater frequency of studies on recruitment than retention has been consistent across the different health topics investigated (Fig. 3). The number of studies published each year has approximately doubled during the past decade; this primarily reflects an increase in studies of recruitment, with only a slight increase evident in the number of studies that assessed retention (Fig. 5). The study designs have been primarily observational and/or retrospective (90/105; 86%), with 11 randomised experiments (10%) and five non-randomised experimental studies (5%).

Among the 11 studies that evaluated digital tools for both recruitment and retention, five (45%) used different digital tools for recruitment and for retention [79, 80, 97, 104, 162]. Four studies employed at least one digital tool that was common to both recruitment and retention, i.e. email [73, 106, 126, 139] and Facebook [126], but in each case the tool was tailored to function differently for recruitment (it included adverts) and retention (it included reminders or facilitated engagement). The remaining two studies assessed the effect of their digital recruitment approach on retention, effectively assuming that the recruitment tools also acted as retention tools [99, 150].

The systematic map (Appendix 5; see Additional file 8) shows that the digital interventions for recruitment and/or retention tested in the studies were a single digital approach in 41% of the studies (43/105), digital approaches combined with non-digital approaches in 50% (53/105), and multiple combined digital approaches in 10% (10/105) (totals exceed 100% as one study included different approaches for recruitment and retention).

Nearly half of the studies (46/105; 44%) did not explicitly state that they included a comparator, but they presented outcomes in such a way that a comparator was discernible (e.g. outcomes were reported separately for different tools). Sixty-four studies defined their comparator(s) (the total number of studies exceeds 105 as some studies included more than one comparison). Of these, 13% of the studies (8/64) employed a comparator that was a single digital approach, whilst 48% (31/64) employed a comparator that was a single non-digital approach. In 16% of the studies (10/64) the comparator comprised a mixture of digital and non-digital approaches whilst in a further 16% the comparator was a mixture of different non-digital approaches. Only one study (1%) employed a combination of multiple different digital approaches as the comparator. The remaining 4/64 studies (6%) gave unclear information, classified as “other” in the map.

The types of recruitment and retention outcomes assessed in the studies are summarised in Fig. 10.

The most frequently reported recruitment outcome in the map (Appendix 5; see Additional file 8) was the recruitment rate, reported in 79 of the 96 recruitment studies (82%). Seventeen of the recruitment studies (18%) reported the quantitative accuracy of recruitment compared against a reference standard, and the time to complete one or more parts of the recruitment process was also reported in 18%. The costs of the digital approaches were reported in 30 of the 105 studies in the map (29%), but the focus was on recruitment, with only three of these studies mentioning any costs in relation to retention. Twenty-eight of the 105 studies in the map (27%) reported the retention rate, but eight of these had investigated digital tools for recruitment rather than for retention. Few studies assessed the satisfaction with digital tools perceived by study personnel (5/105 studies; 5%) or study participants (2/105 studies; 2%), or people’s attitudes towards using the tools (also 2%).

In addition to the pre-specified keywords, the map coding (Appendix 4; see Additional file 7) allowed us to record ‘other’ outcomes if they appeared relevant. We recorded other outcomes for 21 studies. The textual descriptions for these outcomes in the map spreadsheet (Appendix 5; see Additional file 8) indicate that five studies assessed the descriptive characteristics of recruited people, three studies assessed follow-up questionnaire response times, and two studies reported outcomes relating to study workloads (Fig. 10). The remaining 11 studies reported other outcomes that were unique (i.e. specific to each study).

Sixteen studies (15% of all those included in the map) evaluated the use of digital tools for trial recruitment or retention with a focus on individuals from minority or under-served populations (as defined by the study authors). Minority or under-served populations included in studies of recruitment were: Black, African-American and Hispanic people [77, 89, 110, 113, 135, 146, 149, 154]; Maori or Pacific Island populations [162]; Turkish migrants with depressive symptoms [159]; men who have sex with men [107]; transgender women [107]; other people at risk of HIV infection [137]; people with low computer and health literacy [76]; Gulf War veterans [96]; infertile couples [160]; and low-income gay, lesbian and bisexual smokers [77]. One study focussed specifically on the retention of a minority/under-served population: of low-income, vulnerable, smokers with mental illness in psychiatric units [114].

Overall, the most commonly evaluated digital tools in studies on minority or under-served populations were Internet sites for recruitment (reported in 13/16 studies; 81%), television or radio for recruitment, social media for recruitment, and email for recruitment (each reported in 9/16 studies; 56%) (the percentages do not summate to 100 as some studies evaluated more than one tool).

The health-topic area with the largest number of studies that included a focus on minority or under-served populations was health promotion and public health (8/37 studies; 22%) (Table 3). For most health-topic areas the total number of studies was small, but it is notable that only 1 of 17 studies on cancers, and only 2 of 13 studies on circulatory system disorders included a focus on minority or under-served populations.

We did not code participant age in the map keywords. However, inspection of the study publications after the map was completed shows that none of the studies on digital tools for retention included populations of older people or children. In contrast, four of the studies of digital recruitment tools were specifically on populations of older people [125, 143, 152, 155] (although the definition of ‘older’ varied, e.g. range 50 to 75 years [125], mean 69 years [155], median 70 years [152] or range 70 to 93.9 years [143]), whilst three studies were on children [34, 130, 141].

Tools that optimise both recruitment and retention would be desirable, since failure to retain adequate participants in a trial would negate the benefits of achieving good recruitment. In a survey of trial investigators and UK CTUs, the need for strategies to improve retention in trials was identified as being a key research priority [168], whilst the PRioRiTy studies have identified the use of technology in the trial recruitment and retention processes as being within the top 10 research priorities for the future [169]. However, the map shows that most of the studies (81%) have evaluated digital tools only for recruitment. Where studies did utilise digital tools both for recruitment and retention the tools that they used for recruitment and retention were different [73, 79, 80, 97, 104, 106, 126, 139, 162]. This is not surprising, since the approaches for recruitment were primarily to raise awareness whilst the approaches for retention were primarily to provide reminders or to collect data (see Figs. 6 and 8).

We did not specifically seek studies of digital approaches for managing other components of trial management beyond recruitment and retention and so our map keywords do not cover these aspects. However, according to the study publications, several of the studies in the map that evaluated digital tools for both recruitment and retention had included further digital methods for other purposes. These digital methods covered: online patient verification [79, 80]; online randomisation [73, 104]; allocation concealment [73]; automated data collection [73, 104]; information dissemination from investigators to participants [104]; email and website options for participants to contact investigators [139]; data monitoring and tracing [97]; and data-quality checking [73]. A key question is whether the efficiency of clinical trials could be improved by developing a small set of compatible digital tools that would cover all of these aspects of trial management, rather than requiring trial investigators to select separate digital tools to cover recruitment, retention and other aspects of trial management. Further research would be helpful to understand whether (and, if so, how) the different digital tools interact with one another, to ensure that the most efficient combinations of tools can be selected by trial investigators. However, as there are many different possible permutations and combinations of different tools that could be employed in clinical trials for different populations and health conditions, it may be necessary to identify for further research a core set of tools that appear to show the most promise in improving recruitment and retention efficiency.

The appropriateness and efficiency of digital tools for recruitment and retention is likely to be vary across populations and health conditions. The 17 studies on cancers in the map focussed solely on recruitment, with none having investigated digital tools for retention (Fig. 3). A possible explanation could be that retention tools are not required in studies where the primary outcome is survival, or that for other outcomes cancer patients in general are highly motivated, by their need for life-prolonging therapy, to remain in clinical trials as long as possible, reducing the need for retention tools. Whilst a proportion of cancer patients inevitably drop out from clinical trials due to adverse effects of therapy or worsening of their condition, they may no longer be eligible for continuation of the same therapy and so it may not be possible to improve rates of retention in this group. We also noted a relative lack of retention studies for RCTs of circulatory system disorders (Fig. 3). Without an understanding of the underlying mechanisms explaining these observations it is unclear whether the lack of digital retention tools studies in these health conditions represents a major evidence gap, or that retention tools are less useful for certain health conditions.

The map shows that the impact of digital tools for recruitment or retention has mainly been investigated for trial participants who were either patients with a specified health condition or were people eligible for a health promotion intervention. Only one study specifically evaluated digital recruitment of a health professional group: general practitioners being recruited to an online trial to develop and evaluate theory-based interventions to improve antibiotic prescribing [157]. Among the minority or under-served populations (as defined by the study authors), there has been a particular focus on African-American and Hispanic people (eight out of 16 studies on minority or under-served populations), suggesting that these populations are especially challenging to recruit in clinical trials. As with the majority of studies in the map, the research on minority and under-served populations has mainly focussed on recruitment rather than retention. Nevertheless, the authors of a study included in the map commented that they had some difficulty at retaining African American people in their clinical trial of an HIV-prevention intervention in young people [80].

It is unclear why there is an imbalance in the population age distribution in the studies included in the map, i.e. children and older people were represented in some studies of recruitment but not in any of the studies of retention. The age of recruited participants differed between digital and non-digital recruitment methods in a number of the studies (e.g. [78, 83, 103, 107, 129, 140, 152]), indicating that the age of the target population should be taken into consideration when selecting recruitment tools. National statistics show that the proportion of adults who use the Internet (and also social media [170, 171]) is lowest in the over-65 years’ group [171‐173], although the proportion of older people using the Internet has risen steadily in recent years [172, 173], suggesting that some digital tools that may currently be out of reach of older people may become increasingly more accessible.

The studies included in the map evaluated single digital tools, combinations of multiple digital tools or combinations of digital and non-digital tools, with comparators that could be a single digital tool, a single non-digital tool or a combination of multiple digital and/or non-digital tools. The relevance of these digital tool comparisons to clinical trial investigators would depend on whether the investigators’ aim is to supplement existing non-digital recruitment or retention strategies with digital methods; or whether the aim is to employ only digital methods (e.g. if digital tools are considered to be more appropriate or more efficient than standard non-digital recruitment or retention tools, and hence should replace them). Although not specifically captured by our map keywords, we observed that whilst some studies in the map deployed digital tools from the outset of the host trial, others added digital tools after the trial had started, as a response to inadequate recruitment using existing methods.

The problem of how to increase the rates and accuracy of participant recruitment and retention in clinical trials appears well-suited to prospective experimental research, but our systematic map shows that the majority of the primary evaluation studies have been observational in design. A recent paper providing guidance to researchers on the use of social media for recruiting trial participants noted that most of the studies that have shown benefits of social media were observational [174], although observational studies are at increased risk of bias. Direct proof of the effectiveness of digital tools may, therefore, be difficult to establish conclusively unless further experimental studies are conducted. The lack of experimental studies might reflect the idea that SWATs is a relatively new concept [66], and few organisations yet provide funding for these (although this is improving [175]).

The map shows there were many cases where ‘bundles’ of digital and/or non-digital tools were employed for recruitment and/or retention, but the outcomes reported for the digital and non-digital components were often not separable. For example, 12 of the studies in the map employed monetary incentives (e.g. gift vouchers) in addition to their digital tools to encourage participants to join and/or remain in clinical trials. Some studies provided similar monetary incentives to participants in both their digital and non-digital tool study groups (e.g. [104, 106, 114, 126, 158, 162]), whilst other studies provided different monetary incentives for their digital and non-digital tool groups (e.g. [79, 80, 82]). Employing a ‘bundle’ of recruitment or retention tools could have practical relevance (e.g. monetary payment is an integral part of Amazon Mechanical Turk, which was employed as a recruitment tool in one study [82]), but it is important when considering recruitment and retention outcomes to be aware that in some studies estimates of the effectiveness of digital tools could be confounded with the influence of other tools that are present, such as incentives. The authors of one of the studies included in the map acknowledged that they could not rule out the possibility that the observed rates of retention in their study were influenced by the ‘generous’ monetary incentives that they provided [106].

A key challenge when conducting research is to ensure that appropriate outcomes are measured, but the map suggests that a wide range of recruitment and retention outcomes are relevant and it might not be practical for an individual study to assess all of these. The outcomes that appear to have been considered most important by researchers (Fig. 10) are the rate, reach and costs of recruitment and the efficiency and accuracy of recruitment tools. Among these outcomes, accuracy, efficiency and costs of recruitment were reported in only 18%, 24% and 28% of the recruitment studies, whilst hardly any studies assessed the tool users’ attitudes or satisfaction, despite these being important factors that may determine whether a digital tool is likely to used, and work effectively, in practice. For instance, a tool that achieves a high recruitment reach would be of little value if it inaccurately matches people to the trial eligibility criteria, or is considered burdensome or too expensive to use routinely. Where possible, studies evaluating digital tools should, therefore, measure costs, accuracy and efficiency alongside rates of recruitment or retention, and should also capture key process measures, including the attitudes and satisfaction of end-users of the digital tools.

Our study benefitted from comprehensive and systematic methods to identify and characterise the evidence for the effectiveness of digital tools for recruitment and retention in RCTs in health. Our inclusion criteria were necessarily broad, to chart the range of digital approaches that have been evaluated, something that has not been achieved by previous evidence syntheses in this area.

We consulted with relevant stakeholders, via our Advisory Board, to ensure that the scope of the project was as relevant as possible to contemporary practice in trial management. The Board advised us on issues such as the inclusion criteria, and the choice of keywords.

We are publishing our map in its entirety, to help other researchers in this area to understand which areas have been well-researched and where the gaps are in the evidence base. We have identified a number of clear recommendations for researchers that should help to improve the quality and relevance of research in this area and, thus, help to fill these important evidence gaps.

There are some potential limitations to this study that should be acknowledged. Whilst our intention was to map studies that investigated RCTs as their host trial, the host-trial design was not stated explicitly in 15 studies (15%); these were included in the map as the host trial appeared likely to be an RCT in the review team’s judgement. Clearer reporting of study designs would be helpful.

A challenge with developing the keywords was that it is difficult to find keywords that are mutually exclusive. For example, a smartphone App could access an Internet site or social media or advertisements (or none of these). A fine-grained systematic map database that splits out all the possible combinations (e.g. distinguishing between: smartphone App with Internet site; smartphone App with social media; smartphone App with advertisements) would be impractically large and would result in relatively small sample sizes per category. We feel that the granularity of the current map is appropriate since it provides useful information whilst maintaining a manageable breadth of keywords.

We limited our searches to the English language, meaning that the scope of the systematic map is specific to English-language studies. We believe it unlikely that the exclusion of non-English-language studies would make our map unrepresentative, as we are not aware of any reasons why English-language and non-English-language studies would differ systematically in the areas of digital tools research covered.

Due to our eligibility criteria requiring studies to have a comparator, some innovative or more recently developed digital approaches or tools for trial recruitment and/or retention that have not been evaluated against a comparator may not have been identified and included in our map. For example, point-of-care clinical trials using electronic health record systems to carry out the research, when a patient presents to, and is being cared for by, a health professional, can potentially be used to enrol and randomise patients, and to collect outcome data. Studies by D’Avolio et al. [176] and van Staa et al. [177] are examples of point-of-care trials that were excluded from our systematic map due to having no comparator.

Our choice of keywords, agreed with our Advisory Board, aimed to enable us to produce a systematic map with a useful level of detail within a specific timescale. A limitation of our map keywords is that they do not go into detail about the methodology of the host trials and primary evaluation studies. For instance, we did not record systematically whether the studies provided a rationale for why a digital approach was appropriate for their particular setting, although we noticed that the reporting of this appeared to be quite variable. We did not assess whether the trials and studies followed the reporting standards for interventional studies such as the Template for Intervention Description and Replication (TIDieR) Checklist [178], or the Guidelines for reporting embedded recruitment trials [179], or whether the studies strictly meet the definition of a SWAT [66] (many of the studies included in the map were conducted before these guidelines were published). However, further keywords could be added to the map database to capture other information on aspects of study rigor, if required. Further keywords could also be added to address specific research questions of interest (for example, to ascertain the extent to which patients and the public were involved in the conception, design and application of digital tools; or to explore how the selection of digital tools is influenced by host-trial endpoints).