A pragmatic randomized control trial and realist evaluation on the implementation and effectiveness of an internet application to support self-management among individuals seeking specialized mental health care: a study protocol

Participants in the DTG will receive treatment as usual with no intervention from the study for the first 3 months. Treatment as usual may include formal or informal mental health care including pharmacological and non-pharmacological treatments. Participants recruited at discharge from clinics or the ED will receive usual recommendations regarding other services and programs that would routinely be offered upon discharge from those settings.

Participants will receive a 3-month subscription to BWW, free of charge. Use of BWW is participant-driven, with the ability to utilize the services at any frequency. A unique BWW access code will be emailed to participants. If registration has not occurred within 3 days of receipt, a follow-up call will be made to remind the participant to log in. For the DTG participants, a study team member will contact them 3 months after enrollment to remind them that their access to BWW will start and the access code will be emailed. Participants randomized to ITG-e will be able to reactivate their same account once allocation has been revealed.

All participants maintain anonymity on the site through a unique non-identifiable user ID. Study participants will have a unique landing page that includes contact information for local crisis services in the event urgent assistance is required. BWW uses automated word recognition software that will alert the Wall Guides when potential safety issues are arising (eg. a user mentions ‘suicide’). The Wall Guides will then monitor that user’s activity and if deemed clinically necessary, will intervene to offer support. In the event that urgent support is required, they will direct them to use the crisis services listed on the landing page.

At time of consent, participants will complete a baseline questionnaire assessing socio-demographics, mental health history, measures of treatment credibility and outcome expectancy, and all outcome measures. Due to the length of time required to complete consent and all baseline questionnaires, baseline data may be collected in 1 of 3 ways depending on recruitment setting and participant preference: 1) by web survey (default); 2) by phone; or 3) in-person hard-copy.

Baseline data will include participant age, gender, ethnicity, education, relationship status, household income, employment status, and living situation.

At baseline, we will collect age at first onset of mental health problems, age at which formal mental health support was first sought, and duration of current episode.

Participants are asked to rate their belief in the credibility of self-help resources to improve mental health with the author-generated statement: “Self-help tools including on-line services or books are helpful for people with mental health problems” on a 4-point scale from ‘definitely agree’ to ‘completely disagree.’ In addition, they will be asked about their outcome expectancy of BWW to improve their mental health. Item #4 from the Credibility and Expectancy Questionnaire [29] will be used, “After having access to BWW for 3 months, how much improvement in your mental health do you think will occur?” Participants are asked to rate their response from 0 to 100 % with options available in 10 % increments. This single item has been shown to correlate strongly with psychotherapy outcomes [30].

REDCap™ will be automated to push web-based surveys containing all outcome measures to participant emails 2 weeks prior to each pre-determined data collection time point. If, after 2 weeks, the participant has not completed the survey, a follow-up phone call will be made and confirmation of intention to complete it will be obtained and the link pushed again, or the questionnaire will be completed over the phone. For the former option, after 1 week, if still not complete a second follow-up phone call will be made and every effort to collect data over the phone will be made. All outcome measures must be completed within ±2 weeks of the intended time point.

BWW will provide a report of each participants’ use at 2 time points during the study: 1) after all participants have been enrolled for 3 months, and 2) study end. This will include total number of logins, average and total time on the site, along with logs of activity such as pages viewed and posts made.

At the end of the BWW access period (month 3 for ITG, month 6 for DTG) participants will be asked to rate BWW overall and indicate if they would recommend it, pay for it, and continue to use it if available.

The primary outcome is the total score on the Recovery Assessment Scale-revised (RAS-r) at 3 months. The RAS-r is a 24-item validated pan-diagnostic consumer-oriented outcome measure [31]. All items are scored on a 5-point scale from ‘strongly disagree’ to ‘strongly agree.’ The choice of the RAS-r reflects the current “recovery era” for mental health policy and services and the increasing focus on consumer achievement of satisfying and fulfilling lives, rather than being symptom free [31]. Secondary outcomes include the five subscale scores of the RAS-r: 1) personal confidence and hope, 2) willingness to ask for help, 3) goal and success orientation, 4) reliance on others, and 5) not dominated by symptoms; symptoms of depression and anxiety measured with the Personal Health Questionnaire-9 item (PHQ-9) and the Generalized Anxiety Disorder Questionnaire-7 item (GAD-7) respectively; quality of life measured with the EuroQOL group’s EQ-5D-5 L [32]; and community integration assessed with the Community Integration Questionnaire (CIQ) [33]. The 15-item CIQ is a brief, reliable measure of a person’s level of integration into the home and community. The overall score ranges from 0 to 29 with a higher score indicating better integration. The CIQ can also be divided into 3 subscales: 1) home integration, 2) social integration, and 3) productivity [33]. Timing of the completion of outcome assessments is outlined in Table 1.

Self-reported health care utilization will be assessed with the Client Service Receipt Inventory [34], adapted for the Ontario context. This measure captures utilization of physician and hospital services as well as community mental health services. Health administrative data will be used to validate reports of physician and hospital care, however, community service use and non-physician visits are not captured in Ontario data. Studies of reliability of self-report data over a 3 month recall period demonstrate approximately 60 % exact match with roughly equal proportions of over and under-reporting [35]. High emotional distress may be associated with over-reporting, and hospital and ED use is typically more accurately self-reported than outpatient care [35]. The CSRI will also assess prescribed medication and adherence. Self-reported out-of-pocket costs for medication, laboratory investigations, mental health services, and physical health services will be captured using validated questions from the Commonwealth Fund Survey of patients with Complex Needs [36].

Baseline descriptive statistics will be generated overall, between treatment groups and between recruitment settings. Employment status and medication use will be described at baseline and 3 months. Prior 3-month health care utilization assessed with the CSRI (and validated with health administrative data when available) will be described at baseline and at 3 months. All primary and secondary outcomes will be described at baseline and 3 months, including the RAS-r and CIQ subscales. In addition, at baseline and 3 months, we will describe the proportion of participants with PHQ-9 and GAD-7 scores of 10 or more, indicative of clinically important levels of depression and anxiety [40], respectively. At 3 months, we will describe clinically important cutoffs on the PHQ-9 and GAD-7 including a reduction of 5 or more points from baseline (clinically meaningful change), reduction in score of 50 % or more from baseline (response), and a score less than 10 (remission) [40]. Adverse event descriptive data will be reported at 3 months, and again at 6 months for the BWW extension group. BWW satisfaction descriptive data will be reported at 3 months for ITG participants and 6 months for DTG participants.

The first exploratory analysis will examine a subset of the ITG group who had a PHQ-9 or GAD-7 score of at least 10 at baseline. Participants will be categorized as ‘responders’ or ‘non-responders’ based on whether or not they achieved at least a 50 % reduction in the PHQ-9 or GAD-7 at 3 months relative to baseline. A logistic regression model will be built to predict whether or not they were a responder using age, gender, education, relationship status, household income, baseline belief in treatment credibility and outcome expectancy, and recruitment setting as predictors. We will also assess the contribution of BWW utilization, outpatient mental health visits, and medication changes during the intervention months as predictors of response.

The combination of program costs, out-of-pocket costs and costs in the health care system will be used to determine the total costs associated with participants in the program. Incremental cost-effectiveness will be assessed using an incremental cost-effectiveness ratio calculated as the difference in costs between DTG and ITG divided by the difference in outcomes between these groups. Confidence intervals around these estimates will be calculated using nonparametric bootstrapping resampling techniques (with 10,000 replications).

We have performed a power calculation assuming a sample size of 1000 participants which is the minimum target recruitment desired by the funder. Using data from other recent studies of internet based mental health interventions [10], we have calculated power assuming a conservative 30 % attrition rate. The minimal detectable difference between the two treatment groups was calculated assuming an ANCOVA analysis with a 0.8 correlation between baseline and 3-month follow-up RAS-r measurements. With a sample size of 700 accounting for attrition, allocated in a 2:1 ratio, using an alpha of 0.05 and power of 0.9, we are able to detect a difference of 1.35 on the RAS-r.

Qualitative interviews will be completed with 5–7 health care providers at each site; 2–4 organizational leaders at each site (for example, clinical managers); and 5–7 health system decision makers in Ontario. These participants will be identified by clinical site leads using a snowball sampling process to identify key informants.