Implementation strategies for telemental health: a systematic review

We included studies meeting the following criteria:

Studies conducted in primary care or standalone psychotherapy service settings, or that involved service users with substance misuse, neuropsychiatry/neurology or dementia diagnoses were excluded.

Studies where the intervention was only delivered to selected participants recruited for the purpose of the study, as opposed to being rolled out across an existing service, were excluded.

Studies with or without a comparator were included. Studies were excluded which reported findings about the extent of implementation of a TMH programme, or described barriers to or facilitators of the implementation of TMH, but did not describe and evaluate an explicit pre-planned strategy designed to achieve more widespread, effective and/or sustained implementation of TMH, or did not report a relevant outcome according to Proctor’s taxonomy.

We also excluded conference abstracts, review articles, editorials and opinion pieces. Papers were excluded if they were published before January 2010 as earlier studies may be less relevant due to changes in both the availability of and familiarity with technology that can be used to support telehealth.

The search strategy included the following:

The full search strategy is provided in Additional file 1: Appendix 1.

All references were de-duplicated in Endnote X9 [13] and then imported into Rayyan [14] for title and abstract screening. Title and abstract screening was conducted by five reviewers (RA, PB, SS, ER, MT), with 100% included references and 25% of excluded references checked by another member of the research team (PB, MY, MW) to ensure inclusion criteria had been applied correctly. Full text screening was conducted by four reviewers (PB, RA, ET, NL), with 100% included references and 25% of excluded references checked by another member of the research team to assess if they agreed with the original decision. All disagreements were resolved by discussion with a third reviewer.

All included references were imported into EPPI-Reviewer 4.0 [15] for data extraction. A data extraction form was created and piloted on a small number of included studies by three reviewers (RA, NSJ, PB), before data for the remaining studies was extracted by three reviewers (NL, ER, ET). All data extraction was checked by another member of the research team (PB, NSJ, RA, JP).

Details on the service setting, study design, characteristics of the clinical population, characteristics of the staff and TMH modalities used were extracted from each study. Details of the implementation strategy used, categorised according to the ERIC compilation of implementation strategies [16] (outlined in more detail below) and implementation outcomes (categorised according to Proctor’s taxonomy [12]) were also extracted.

Quality appraisal was conducted using the Mixed Methods Appraisal Tool (MMAT) [17] for all primary research studies which aimed to answer a research question. MMAT recommendations are to report the percentage of criteria met for each study (e.g. 20% represents 1/5 criteria met, while 80% represents 4/5 criteria met). Studies were considered of low quality if they met 20% or less, medium quality if they met 40–60% of the criteria, and high quality if they met at least 80% of the criteria. The AACODS (Authority, accuracy, coverage, objectivity, date, significance) tool for descriptive studies [18] was used to appraise the quality of studies which did not meet the MMAT criteria of having a clear research question. We describe these studies as “descriptive studies”. Quality assessment was carried out by one member of the research team (NL, ER, ET, or RA) and checked by another reviewer (RA, PB, or NVSJ). Due to the relatively small number of papers included in this review, the results of quality assessment were not used to determine eligibility for inclusion, although quality assessment along with study design were considered during interpretation of findings.

We conducted a framework synthesis [19] to consolidate findings from the included studies, a method which “allows themes or concepts identified a priori to be specified as coding categories from the outset” (p1). We used two pre-established implementation science frameworks as the basis for our framework synthesis, and mapped data from included papers according to headings from these two frameworks. Firstly, we used the overarching categories from the ERIC compilation of implementation strategies [16], to identify and record strategies used to implement TMH (see Table 1 for further details). Secondly, outcomes of studies were categorised according to the Proctor taxonomy of implementation outcomes [12], which consists of the following: acceptability (stakeholders’ perception that the intervention is agreeable), adoption (uptake of the intervention), appropriateness (the perceived compatibility of the intervention), feasibility (the extent a new intervention can be used in a particular setting), fidelity (whether an intervention was implemented as originally prescribed), implementation cost (the cost effect of implementing the intervention), penetration (the integration of the intervention within a service) and sustainability (the extent to which a new intervention is maintained). Details of outcomes and interventions from the framework were used to create summaries of the strategies employed and resulting outcomes in each study. As per recommendations for the use of these outcomes [12], we recorded within the taxonomy both those outcomes reported as resulting from the pre-planned strategies intended to optimise TMH, and outcomes relating to the TMH interventions themselves (as a result of implementation strategies) where reported by the included studies. For example, we recorded acceptability of training reported by clinicians as well as acceptability of the TMH interventions reported by service users, which may have been impacted by the implementation strategy of 'train clinicians'.

Database searches identified 20,858 papers, of which 14,294 were screened by their title and abstract once duplicates had been removed. A total of 338 papers were screened at full text, resulting in 14 studies identified for inclusion in the review. No additional papers were identified from preprint servers or from forward or backward citation searching. The study selection and screening process is summarised in Fig. 1.

Of the 14 included studies, 11 were conducted in the USA, with one each in the UK, Canada, and Australia. Six studies focused only on service users, seven only on staff, and one involved both staff and service users as participants. Four studies used quantitative methods, four were mixed methods studies, two were qualitative studies and four studies were descriptive in nature (did not aim to address a specific research question and instead provided a description of actions and their outcomes). No trials or studies with a comparison group were identified.

The majority (n = 11) of studies explored implementation in community mental health services, and five studies used an established implementation framework to inform their analysis. Further details of included studies can be found in Table 2.

MMAT [17] quality appraisal was conducted for the 10 primary studies which aimed to answer a specific research question (as opposed to purely describing the implementation of TMH). Of these, two studies met 20% of criteria (low quality), three studies met 40% of criteria (medium quality), three studies met 60% or criteria (medium quality), and one study each met 80% and 100% of criteria (high quality). Four descriptive studies were appraised using the AACODS checklist [18], and as all studies met all (3/4) or all but one (1/4) criteria, we considered these studies as appropriate descriptive studies but considered primary studies as more important. A full breakdown of the results of the quality assessment is provided in Additional file 2: Appendix 2.

Each type of implementation strategy in ERIC’s taxonomy was reported in at least one study. The most commonly used strategy was ‘Train and educate stakeholders’, which was identified in nine studies, whilst the least used was ‘Utilise financial strategies’, which was only reported in one study [24]. The mean number of strategies used per study was 3.5, while the most common numbers of strategies used per study was 2 or 3. The implementation strategies reported by each study can be found in Table 3.

Most of the implementation strategies were reported as being associated with good outcomes for TMH implementation, such as improved knowledge for clinicians regarding how to conduct care via TMH, increased acceptability or adoption. However, some barriers to TMH implementation remained, including a lack of staff time, higher administrative burden, or a preference for face-to-face appointments (amongst either staff or service users).

While most studies reported implementation outcomes only in relation to the TMH intervention itself, five studies reported implementation outcomes resulting specifically from the strategies used [23, 25, 28, 31, 33]. Felker et al. [23] used three strategies: ‘provide interactive assistance’, ‘adapt and tailor to the context’, and ‘train and educate stakeholders’, the latter strategy resulted in high ratings of the acceptability, adoption, and appropriateness of the training provided. Lindsay et al. [25] also reported using the same three strategies and found that therapists reported a high degree of satisfaction with the external facilitation provided as part of training. Myers et al. [28] used three strategies: ‘provide interactive assistance’, ‘develop stakeholder interrelationships’ and ‘train and educate stakeholders’ during the implementation of a TMH platform, and found that whilst adoption of TMH increased, there were additional implementation costs in terms of time associated with unpaid site champion roles developed to provide interactive assistance. Puspitasari et al. [31] used five implementation strategies in the rapid adoption of TMH due to the COVID-19 pandemic: ‘use evaluative and iterative strategies’, ‘provide interactive assistance’, ‘adapt and tailor to the context’, ‘develop stakeholder interrelationships’, and ‘train and educate stakeholders’. They found education, training, and ongoing supervision were of particular importance at the start of teletherapy implementation to support clinicians’ successful engagement with the technology. Taylor et al. [33] used two strategies—‘use evaluative and iterative strategies’ and ‘develop stakeholder interrelationships’ to investigate the importance of clinical facilitation for the implementation and sustainability of TMH in perinatal and infant mental health services and concluded that ongoing clinical facilitation is necessary for the sustainability of TMH services.

We had originally aimed to make specific links between strategies and general implementation of TMH outcomes, but as all but one study reported several implementation strategies in varying combinations, this was not possible. Instead, strategies and reported outcomes are presented in Table 4 and outlined for each study in Additional file 3: Appendix 3. Outcomes for each study are categorised according to the taxonomy of implementation outcomes [12].

There was no discernible difference in the implementation strategies used by studies which did or did not use implementation frameworks to guide their approach, as all studies used a variety of different strategies. In addition, we did not observe any commonalities across the outcomes of these studies.

We also compared the use of telemental health strategies between the studies which implemented telemental health before and after the start of the COVID-19 pandemic, but as all studies used a variety of different strategies there were no obvious differences in the types of strategy used or reported outcomes. However, Felker et al. had implemented telemental health prior to the pandemic and concluded that “earlier TMH training efforts likely helped foster staff interest and experience using TMH, and in doing so contributed to the relatively strong TMH adoption rate observed in this health care system during the early months of the COVID-19 pandemic.”

In this study, we have reviewed literature on strategies used to improve implementation of TMH and their associated implementation outcomes. We identified as meeting our inclusion criteria fourteen studies, conducted across five countries. Both staff and service user views and outcomes were represented in these studies. Results indicated that using a combination of different planned implementation strategies could be associated with successful implementation of TMH, although the methodologies of most studies were such that firm causal conclusions were difficult to draw. Whilst we had originally planned to explore links between individual types of implementation strategy and implementation outcomes directly, we were unable to isolate the effects of specific mechanisms as all studies reported outcomes of initiatives that combined multiple implementation strategies and none had a comparison group. We are, however, able to propose some tentative conclusions based on the synthesis of findings from studies which reported outcomes of these strategies. Ongoing support and facilitation, for example, through either technical assistance or ongoing consultation, was common and tended to be strongly linked to successful implementation. Providing initial training and the use of ‘digital champions’ to model best practice, also benefited implementation of TMH.

Other recent studies further support and supplement these conclusions. In our recent rapid realist review of TMH [34], we found that providing staff with training on the use of technology to deliver mental health services, a strategy reported by several studies in the current review, was reported to increase confidence in and uptake of TMH. The rapid realist review also found that adapting the use of TMH to take into account service user preferences was beneficial in removing barriers to accessing TMH [34] (see Table 5 for further discussion). For example, a commonly cited barrier to service user engagement with telemental health has been a lack of choice of modality [5, 35], so that offering a choice of platforms to service users may remove some barriers to telementation implementation.

Research indicates that ERIC strategies are considered suitable to influence different implementation outcomes [36], but there is currently little consensus or evidence regarding which strategies affect specific outcomes. Furthermore, Powell and colleagues [37] argue that implementation strategies should be tailored to the circumstances and context of each change project, as they may be more likely to result in changes to practice. They suggest four different methods to identify appropriate strategies: concept mapping, group model building, conjoint analysis, and intervention mapping. Essentially, this means that the appropriateness and effectiveness of an implementation strategy or implementation support ‘bundle’ may well depend on the context in which a clinical intervention or service delivery mechanism, such as TMH, are introduced to. It logically follows that the same implementation strategy may be very well suited to one context, but redundant in another if it fails to address specific barriers to implementation. Hence selection of implementation strategies should be tailored to the local context of application. This can be illustrated by studies which have reported various staff barriers or challenges to using TMH. These have included: feeling isolated from colleagues, tasks taking more time when working from home, lack of support, concerns around privacy or confidentiality, and concerns around developing and maintaining a therapeutic relationship [2, 38, 39], indicating the importance of tailoring implementation strategies to overcome local barriers. Whilst some studies included in this review evaluated the barriers and facilitators to TMH before using implementation strategies to address these, not all papers reported taking this approach in a systematic manner.

As all studies reported using multiple implementation strategies, we were unable to compare the effectiveness of specific strategies. This could be addressed by future research, for example, cluster randomised controlled trials to formally compare the effectiveness of different implementation strategies linked to specific outcomes at a team or organisational level, although it would also be helpful to evaluate a theoretically informed approach to selecting a bundle of implementation strategies to fit a particular context, using more robust study designs. Inclusion of a control group in future evaluations of the implementation of TMH is critical: without this we will remain unable to establish causal links between the presence or absence of a strategy to support implementation, or bundle thereof, and success of implementation offers an illustration of a clustered randomised evaluation of different interventions to support the implementation of four evidenced-based psychosis treatments. Within this study, what was randomised was not the clinical therapies, but rather the level of implementation support, which was limited in the control arm (provision of treatment manuals) and substantially enhanced in the intervention arm to include toolkits, training, implementation facilitation, and data-based feedback. The primary endpoint was fidelity of treatment delivery, and the trial concluded that the implementation ‘bundle’ was successful in enhancing fidelity across all four studied treatments for psychosis. Similarly designed studies in the context of TMH provision will significantly expand our knowledge regarding how best to deliver it sustainably and effectively.

Beyond controlled evaluations, our understanding of the relative effectiveness and suitability of strategies to support implementation of TMH across different settings can be further enhanced. Future observational studies should offer a detailed description of local (or wider, as appropriate) settings in which TMH is offered, a well-articulated rationale for the selection of strategies (such as those we summarised above) to support implementation, and clear details of what each strategy involves – such that subsequent evidence syntheses can offer a better articulation of which strategies may be better suited to which contexts. We further propose that selection of implementation support interventions should be based on a description and mapping of the barriers and drivers an implementation effort is likely to face, for example carried out using one of the methods proposed by Powell et al. [38].

It is also important to note that most studies identified in this review were conducted in the United States. There is therefore a need to replicate these findings and conduct further research in other countries with different healthcare structures and funding models to generalise the findings from this research.

Our review identified a range of potential implementation strategies to be deployed to improve TMH implementation in routine settings, with evaluations spanning a full range of types of implementation strategy. Although we were unable to identify causal links between implementation strategies and outcomes, findings across the included studies suggest that when implementing TMH, service planners should consider a multi-component implementation strategy. This strategy should be tailored to the local context and designed to address any pre-identified barriers. It is likely that staff training and facilitation support are key factors in the success of implementing TMH.

A strength of our review is that it focused only on studies which implemented TMH as part of their routine service (i.e., not just in a trial), which means findings are more likely to generalise to ‘real world’ settings. This review also integrated data from studies conducted before the COVID-19 pandemic with those conducted during the pandemic, which enables findings to be used to inform future models of service development.

Limitations of this research should be acknowledged. Firstly, as noted earlier, the high heterogeneity of strategies and outcomes reported across studies makes it hard to reach firm judgements about which strategies are linked with effective implementation of TMH. As all studies reported the inclusion of multiple strategies, we were unable to draw conclusions regarding the active ingredients of specific strategies. Secondly, researchers were not blinded to the results of screening and quality assessment during double screening. This was due to the short timeframe in which the review was conducted in order to make its results relevant to the current service context. We limited the scope of our review to the implementation of TMH in specialist mental health services, which typically serve mainly people with severe and enduring mental illness. Our review is therefore unable to provide evidence about TMH implementation in primary care settings, or for other client groups including people with substance misuse (in the absence of a mental illness) or people with dementia or other organic disorders. Finally, included studies were mostly of moderate to low quality, and only five interventions were informed by an established implementation framework.