Protocol for the Process Evaluation of the Online Remote Behavioural Intervention for Tics (ORBIT) randomized controlled trial for children and young people

The ORBIT trial and its BIP TIC intervention have been described in detail previously as part of the main trial protocol [23] (03/01/2019; version 3.0), a brief summary is given to provide context to the process evaluation design. The ORBIT trial is a 10-week, parallel group, single-blind RCT with an internal pilot. ORBIT aims to evaluate the efficacy of an online, remote, therapist-supported and parent-guided behavioural intervention for tics, which was initially developed and piloted in Sweden and called BIP TIC [24]. The comparator is an online, remote, therapist-supported and parent-guided psychoeducation programme for tics. Participants will be recruited from clinics, Patient Identification Centres (PICs) across National Health Service (NHS) Trusts, or from the two study sites involved in the trial (Queen’s Medical Centre (QMC), Nottingham and Great Ormond Street Hospital (GOSH), London), or via a tic disorder charity (Tourettes Action), the ORBIT study website or social media. Participants need to be 9–17 years old and be suspected or confirmed as having Tourette syndrome (TS) or chronic tic disorder (CTD) and must not have had any form of behavioural treatment for tics in the last 12 months or a change in medication for tics in the previous 2 months. Participants will be followed up mid treatment and at 3, 6, 12 and 18 months post-randomization.

Participants will be randomized to one of two groups. The intervention group will receive 10 self-help modules of behavioural therapy delivered over a period of 10–12 weeks, which will be accessed via a secure online platform [24]. The behavioural therapy will follow evidence-based exposure and response prevention (ERP) therapeutic principles, whereby patients learn strategies for managing their tics through allowing premonitory urge sensations to come to the fore and actively tolerate the premonitory urges and suppress their tics. In doing so, the child masters their ability to tolerate the urge, control their tics, and is able to do so for an increasing amount of time in a hierarchical manner. The child also receives education about tics for the family and others, such as teachers, friends and family. Throughout the 10–12 weeks, participants will have access to a therapist and their role is to encourage participants to engage with the treatment content and its homework assignments, and answer any queries that participants may have. The parent components contain information about how to support their child and various coping strategies for themselves. Previous studies have shown that ERP is effective in reducing tics [25, 26], with European clinical guidelines [25] and a National Institute of Health Research Health Technology Assessment Evidence Synthesis [27] recommending that behavioural therapy should be offered as a first-line intervention for tics in CYP. The primary outcome measure is the severity of tics as measured on the Total Tic Severity Score (TTSS) subscale of the Yale Global Tic Severity Scale (YGTSS) [28]. The target sample size for the ORBIT trial is 220 (110 in the intervention arm and 110 in the control arm).

Overall, the ORBIT trial aims to evaluate the clinical effectiveness of an online behavioural treatment for CYP with tics compared to online tic-related education in reducing tics, as measured by the YGTSS TTSS. Furthermore, the trial aims to evaluate the cost effectiveness of the online treatment and to estimate the longer-term impact on patient outcomes and health service costs.

The aim of the ORBIT process evaluation is to understand the causes of the observed behaviour change using data obtained from the RCT, and in particular, to explore the fidelity of intervention delivery, acceptability of the intervention and reasons for observed variation in uptake and use, and to consider the resources and implementation processes required.

Specific objectives are:

The design of this process evaluation is guided by MRC directives on the process evaluation of complex interventions [17]. The MRC outlines three essential components in understanding how outcomes are achieved: implementation, mechanisms of impact and context. The application of these guidelines in the context of the ORBIT trial will be as follows:

MRC guidance on the development and evaluation of complex interventions notes that identifying and developing a theoretical understanding of the likely process of change is a key early task for developing a complex intervention or evaluating one that has already been developed. MRC guidelines stipulate an important component of a process evaluation is to outline the processes of the intervention and the outcomes it aims to achieve by means of a logic model. The logic model for the study is shown in Table 1.

The overall design of the ORBIT process evaluation is a mixed-methods study using purposively sampled qualitative data together with quantitative data from the trial. This will involve semi-structured interviews with children, parents, therapists and supervisors and clinicians, and analysis of online feedback from participants together with data from the online platform, such as total therapist time, number of chapters viewed and number of log-ins.

The schedule of the ORBIT process evaluation procedures is displayed in Fig. 1. In Additional file 1 a populated Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklist is provided [29] and in Additional file 2 a CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions is provided [30].

Ethical approval for the process evaluation was obtained from North West - Greater Manchester Central Research Ethics Committee as part of the ORBIT trial (REC: 18/NW/0079).

Qualitative data will be collected by interviewing participants in the BIP TIC intervention (both CYP and parents, either separately or as a dyad), therapists and supervisors and referring clinicians. Interviews with therapists and supervisors involved in the ORBIT trial will be conducted early in the trial and near the end of recruitment in order to gain an understanding of their experience at different time points. All interviews will be conducted either by telephone or by the WebEx videoconferencing application. In addition, at the end of treatment participants are asked within the BIP TIC platform questions including what the most important thing they have learnt from treatment, how the treatment has helped, if the treatment caused any difficulties to participants, and any other comments they may wish to add. These data will be put into BIP TIC and then will be exported to an excel spreadsheet and content analysis will be performed.

Online data will be collected and recorded from participants throughout the trial. These include the following measures: total therapist time; therapist time specific to each therapist; therapist time specific to each child and parent; total number of characters submitted by child and parent (as part of communication messages via the online system); total number of logins for child and parent; average time between each login (in days) for child and parent; average pages visited per login for child and parent and the five most frequently visited pages per child and parent. These data will be amalgamated and entered into a centralised online database whereby the main researcher will then extract this data for analysis as part of the process evaluation.

Qualitative data will be exported and analysed in QSR International’s NVivo 12 Software [39] and quantitative data will be exported and analysed in SPSS (version 25.0) [40]. Process evaluation data will be analysed autonomously of the main outcome data of the ORBIT trial.

All interviews will be recorded either by the WebEx videoconferencing application or by Dictaphone and then transcribed verbatim. Transcripts will be checked for accuracy against the recordings, with any corrections made as appropriate. Prior to importing transcripts into QSR NVivo 12, any reference to places, clinicians, therapists, and/or family members that may reveal participants’ identity will be redacted, and all participants’ names will be anonymised. The interviewer will take notes during all interviews.

As the process evaluation is a combination of exploration and description, thematic analysis will be used to identify, analyse and report patterns within the transcribed interviews. Thematic analysis is widely used within the field of psychology and is considered the most flexible qualitative analytical process [41]. More broadly, the framework method [42] of analysis will be employed, as it is most commonly used for the thematic analysis of semi-structured interviews [43]. Moreover, Ritchie and Spencer [42] outline four types of research questions that they believe framework analysis can helpfully address: (1) contextual - identifying the form and nature of what exists (e.g. what is the nature of people’s experience?); (2) diagnostic - examining the reasons for, or causes of, what exists (e.g. why are services or programmes not being used?); (3) evaluative - appraising the effectiveness of what exists (e.g. what affects the successful delivery of programmes or services?) and (4) strategic - identifying new theories, policies, plans or actions (e.g. how can systems be improved?). As the process evaluation covers all of these questions, we feel this is the appropriate methodology to use.

Ritchie and Spencer [42] suggest five key stages of framework analysis: familiarisation, identifying a thematic framework, indexing, charting and mapping and interpretation. During the familiarisation stage, the main researcher will immerse himself in the data by listening and/or watching back the interviews, reading transcriptions and studying observational notes whilst listing key ideas and recurring themes. The data will then be analysed to identify key issues, concepts, themes, and sub-themes drawing on both a priori and emergent issues. Next, the transcripts will be coded and indexed into framework categories by systematically applying the thematic framework to each interview. The indexed data will be summarized for each category and organised in chart form. This process will involve working through each framework category, summarizing all data that have been indexed to that category, and then providing a summary for each category for each participant, using headings and subheadings. Consequently, key characteristics of the holistic dataset will be mapped and interpreted. An independent coder will double-code a subset of transcripts to identify emergent patterns and themes relating to participants’, therapists’ and clinicians’ experiences of the ORBIT trial. Charted data will be annotated independently and the findings discussed, which will allow for refinement and amendment of data in an iterative process. Once confidence in the congruity and meaningfulness of interpretation is established between researchers, we will review the remaining interviews to establish whether our understanding has reached acceptability.

The large amount of data collected for the process evaluation encouraged us to use computer-assisted qualitative data analysis software (CAQDAS). The CAQDAS package, QSR NVivo 12, is fully integrated with framework analysis and this will be used to categorise data and document any themes and sub-themes. Online feedback given by participants at the end of therapy will be analysed using content analysis and integrated into the aforementioned framework.

Quantitative data from the online platform will be analysed descriptively by calculating total numbers and percentages and mean and standard deviation or if the data are not normally distributed, the median and range. This will provide information on intervention delivery, including the implementation of different components and fidelity. The independent samples t-test and the chi-squared test will be used to explore any significant differences within the intervention group. Data that are not normally distributed, will be analysed using non-parametric alternatives (i.e. Kruskal–Wallis H and Mann–Whitney U tests), using a significance level of P < 0.05.

Qualitative and quantitative data will be analysed separately and then mixed during analysis in a methodological approach known as triangulation [44]. Both qualitative and quantitative data will be given equal importance, as both sets of data are central to addressing the research questions posited by the process evaluation. A Good Reporting of a Mixed Methods Study (GRAMMS) [45] checklist is provided in Additional file 5.

Qualitative data and preliminary qualitative analysis will be coded synchronously with the analysis of descriptive statistics of participants’ online data. Thus, the descriptive data will aid in the refinement and amendment of questions central to qualitative data collection. In other words, key themes may emerge from the quantitative data, which could then be further explored or clarified from qualitative data, and vice versa. The main researcher will integrate and compare outcomes from the various datasets guided by the triangulation protocol. The aim of this is to create a matrix of converging datasets to assess outcomes where there is agreement or dissonance, and where themes or outcomes emerge in one dataset but not another. Once the matrix of outcome synthesis from the various datasets is finalised, it will be used to emphasise the mechanisms of impact, implementation fidelity and, more broadly, explain the outcomes of the trial.

The process evaluation data will be analysed prior to knowing the main ORBIT trial results, with the two analyses being independent of each other. The ORBIT trial team will be unaware of the findings of the process evaluation until the primary outcomes from the main trial have been analysed. Once both trial and process evaluation analyses are complete, combined qualitative and quantitative data may aid in the development of hypotheses about the potential successful implementation in one context over another and how and why some components were delivered successfully and others were not. Furthermore, the analysis of different components may aid in the identification of causal mechanisms and how and why individual intervention components were more effective than others. Following quantitative analysis of ORBIT trial data, qualitative data from the process evaluation can potentially be used to help explain the outcomes of the trial. Additional analyses can then be conducted to test hypotheses emanating from integration of process evaluation data with trial outcomes, drawing together the findings to understand why the intervention worked (or not), context and implications for further dissemination to improve provision of care for CYP with tics.

Conducting the process evaluation will contribute to explaining the overall findings of the main RCT: the factors underlying positive and negative effects of different aspects of BIP TIC. For example, if there were certain negative outcomes from using BIP TIC, the process evaluation will be an invaluable resource in elucidating whether the intervention was inherently inadequate, if there was a failure of implementation and if this was related to participants (e.g. lack of motivation) or contextual factors (e.g. pre-existing beliefs about online therapy). This would help to improve the intervention progressively.

In contrast, if there were positive outcomes from using BIP TIC, the process evaluation will identify the core components that made the intervention a success. For example, if it was determined that an essential component for promoting participants’ adherence to the intervention was the use of parental support and therapist encouragement, these findings will be crucial to the development and implementation of future digital programmes aimed at CYP with tics.

By collecting data from a range of relevant stakeholders (e.g. parents, children, therapists and supervisors and clinicians) and combining quantitative and qualitative data, we will gain a holistic understanding of the mechanisms underlying the impact of the intervention. Furthermore, the proposed sample size is adequate to capture a comprehensive overview of perspectives, generating rich data and analytical depth.

One potential limitation arising from this is that the majority of participants who drop out of treatment are more likely to refuse to be interviewed, which could lead to a more positive overall evaluation of the intervention. We will attempt to overcome this by making a more concerted effort to recruit participants who drop out of treatment or, if this is not possible, those who complete fewer modules. The main limitation in terms of future implementation is that the environment/context will be heavily influenced by this study being an RCT. It would arguably be more appropriate to conduct a parallel implementation study; however, lack of resources prohibit this.

The trial and recruitment of participants for the process evaluation is ongoing.