The effect of web-based educational interventions on mental health literacy, stigma and help-seeking intentions/attitudes in young people: systematic review and meta-analysis

We aim to conduct a systematic review and meta-analysis of interventions that target MHL, help-seeking intentions/attitudes, and stigma related to mental illness in young people aged 12 to 25 years. We treat these as separate but potentially related outcomes of intervention programs, while recognizing that MHL is a complex concept that may influence stigma and other factors. However, we distinguish between help-seeking intentions/attitudes and stigma related to mental illness to avoid the confusion and inconsistency that arises when studies measuring different aspects are grouped under the broad term of “MHL” [25].

Research question: How effective are web-based interventions in (1) improving MHL and/or (2) enhancing help-seeking intentions/attitudes and/or (3) reducing stigma related to mental illness among young people aged 10 to 25 years?

This review was conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [26]. To identify eligible studies 7 databases were searched: PubMed, Web of Science, Scopus, ERIC, CINAHL, Cochrane library and Embase. PubMed, Web of Science, Scopus, Embase and Cochrane library were searched directly; the others were accessed via the research platform EBSCOhost. The search was conducted February 2023 and search results were limited to English language articles published between April 1975 and 2023. We limited our search to articles published between April 1975 (the year when WHO launched its first global program on mental health) and February 2023 (the date when we conducted our search), as we assumed that web-based interventions on mental health were unlikely to exist before 1975. To avoid missing any publication, we also examined reference lists of all included studies as well as review articles. Unpublished data and grey literatures, including dissertations, congress abstracts, and patents, were not included in the current meta-analysis. In addition, we removed duplicate citations. The search term was adapted for each database. See the supplementary material 1 for an example of the search string used in PubMed. Articles identified by the database search were screened to assess relevance to the aims. In addition, Google Scholar and selected reference lists were also searched to identify additional studies of interest. This review is a registered PROSPERO review: CRD42023397689.”

We included studies that met the following criteria: (1) they addressed young people aged 10 to 25 years old; (2) they were randomized controlled trials, non-randomized controlled trials, experimental or before-and-after studies; (3) they delivered a web-based intervention program; (4) they had a control group or provided an intervention as treatment as usual; (5) they assessed the help-seeking intentions/attitudes and/or the mental health-related stigma and/or MHL directly through the self-report of young people, instead of relying on information from caregivers or teachers; (6) they were published in English; and (7) they reported mean changes and their standard deviations (SDs) of outcomes for both intervention and control group or presented required information for calculating of those effect sizes.

We excluded studies that met any of the following criteria: (1) they were qualitative studies, as they were not suitable for meta-analysis; (2) they had no information about participants’ age or affiliation to the educational system (so it could not be inferred whether they were under aged); (3) they did not report results (e.g., abstracts of registered clinical trials); (4) they did not have random allocation; (5) they had an observational design; or (6) they had no control group.

Two authors (A.N and M.H) independently reviewed the relevant titles and abstracts. They obtained and scrutinized the full text of the chosen studies with another author’s (GH.G) help. They resolved any disagreements by discussion or referral to a fourth author (A.RF). Some studies were excluded as irrelevant after full-text review.

The data extracted included: first author’s name, publication year, study population, sample size, participants’ sex, number of subjects in each group, age range and average age of participants, trial design, type of intervention, time of post-test, mean and SD mental health literacy, stigma and help-seeking intentions/attitudes after intervention in each group. The authors converted SEs or interquartile ranges to SDs using appropriate formulas if data were reported as such.

The present study uses the QuADS tool [27] to assess the quality of nine systematic reviews of mixed- or multi-method studies on different aspects of web-based interventions for MHL, help-seeking intentions/attitudes, and stigma among young people. The QuADS tool consists of 12 criteria that cover four domains: (a) research aim/s, (b) research setting and target population, (c) study design, and (d) analytic method. Each criterion is scored from 0 to 3, and the total score is the sum of these scores. The total score ranges from 0 to 36, and indicates if the study is excellent (above 80%), good (between 50% and 80%) or low (below 50%) in quality.

The mean change (S.D) were used to estimate the overall effect size of the intervention. If S.D was not reported in each of individual study; it was calculated using following formula: S.Dchange = square root [(S.Dbaseline2 + S.Dfinal2)-(2 × R × S.Dbaseline × S.Dfinal)]. A correlation coefficient of 0.8 was considered as R-value of the above-mentioned formula [28]. Where standard error of mean (S.E.M) was only reported, standard deviation (S.D) was calculated by S.D = S.E.M × √n (n is the number of participants in each group). Finally, in studies which only reported outcome measure in graphic form, data extraction was performed by using GetData Graph Digitizer 2.24 [29].

Estimates of effect sizes were expressed based on standardized mean difference (SMD) and 95% CI from the random-effects model. The SMD was calculated by dividing the mean difference by the pooled standard deviation of the intervention and control groups. Assessment of between-study heterogeneity was carried out using Q test and I-square (I2) test [30]. Pre-defined subgroup analysis based on time of post-test, mean age, study setting, and study quality was conducted to detect potential sources of heterogeneity. The sensitivity analysis was conducted by the one-study remove (leave-one-out) approach, to explore the impact of each study on the pooled effect size. Publication bias was evaluated using visual assessment of funnel plots and Egger’s weighted regression tests. All statistical analyses were performed using Stata software version 12 (StataCorp. College Station, Texas, USA). P < 0.05 was considered as statistically significant.

We were able to use data from five reports studies [31, 32, 35, 36], including 2,195 participants, to assess the efficacy of interventions on MHL (see Fig. 2). The results of random-effect analysis showed that interventions have a significant effect on MHL [SMD = 0.70, 95% CI = (0.16, 1.25)]. A remarkable between-study heterogeneity was observed (I2 = 96.3%, P < 0.001), By performing the analysis, it was found that other design methods (those that were not RCTs) were the source of heterogeneity(p-value > 0.10 AND I² < 50%).

The results of the subgroup analysis showed a significant difference so that studies that were RCTs got lower scores than other designs. Also, in the studies that have been done in school setting, they get a lower score than other settings (supplementary material 3).

By performing sensitivity analysis, it was found that by removing each of the studies, there is no significant change in the overall result (supplementary material 4).

There was no publication bias either with Funnel’s or with Egger’s test (P = 0.057) (supplementary material 5).

We were able to use data from nine reports [33‐35, 37‐39], with 2389 participants, to evaluate the effectiveness of interventions on help-seeking intentions/attitudes. Help-seeking intentions/attitudes refer to the willingness or readiness to seek professional help for mental health problems. We performed a meta-analysis using a random-effects model to pool the standardized mean differences (SMDs) of help-seeking intentions/attitudes between intervention and control groups. Figure 3 shows the forest plot of the meta-analysis results. The overall pooled SMD was 0.48, with a 95% confidence interval (CI) of (− 0.50, 1.46). The p-value was 0.340. This means that the interventions did not improve help-seeking intentions/attitudes significantly.

The studies showed significant heterogeneity (I² = 99%, P < 0.001). We performed subgroup analysis to explore the sources of heterogeneity. We used four factors: time of post-test, mean age, study setting, and study quality. These factors were chosen based on our hypothesis that the effectiveness of interventions may vary depending on how long they last, how old the participants are, where they are delivered, and how well they are conducted. There was no significant difference between subgroups based on study quality(p < 0.001), mean age (p < 0.001) and study setting (p < 0.001), but based on time of post-test (p-value > 0.10 AND I2 < 50%). There was a significant difference in that the reports collected immediately after the intervention showed higher help-seeking intentions/attitudes. The results of the subgroup analysis are shown in supplementary material 3.

We performed sensitivity analysis to examine the influence of each study on the overall pooled estimate. We omitted each study one by one and recalculated the pooled estimate and its 95% CI. The results of the sensitivity analysis are shown in supplementary material 4. None of the studies had a large influence on the overall pooled estimate, as none of them changed it substantially or made it statistically significant when omitted. This suggests that our result is robust and not driven by any single study.

We assessed publication bias using Funnel and Egger tests. The results are shown in supplementary material 5. There was no evidence of publication bias (P = 0.080), indicating that there was no systematic tendency for studies with positive or negative results to be more or less likely to be published.

We were able to use data from 7 reports [36‐38], including 1,053 participants, to evaluate the effectiveness of interventions in reducing stigma (see Fig. 4). The results of the random effect analysis showed that the interventions had a non-significant effect on reducing the stigma associated with mental disorders. [SMD = -0.20, 95% CI = (-0.48, 0.08)]. A significant heterogeneity was observed between the studies (I² = 80.3%, P < 0.001), by performing the analysis, it was determined that the studies that measured the impact of the intervention in different countries(P = 0.891), mean ages (p = 0.874), time of post-test(p = 0.474) and study quality(p = 0.874) were a heterogeneous source. The results of the subgroup analysis showed no significant difference between subgroups based on age, time of post-test, the countries or quality assessment (supplementary material 3). By performing sensitivity analysis, it was found that removing any study did not cause a significant change in the overall result (supplementary material 4). There was no publication bias with the Funnel or Egger test (P = 0.332) (supplementary material 5).

Web-based interventions can increase the knowledge and understanding of mental health and mental disorders among young people who may face barriers to access and engage with traditional mental health services. Web-based interventions can also complement existing services by providing information, support, and referral options for young people who experience mental health problems or who are at risk of developing them. However, web-based interventions may not be successful in improving the attitudes and behaviors of young people toward themselves or others with mental health problems. Web-based interventions should be tailored to the specific needs and preferences of young people, and should be evaluated for their effectiveness and acceptability. Involving young people in the design and evaluation of web-based interventions can ensure that they are relevant, informed, and user-friendly. Furthermore, web-based interventions should be part of a broader and coordinated strategy to improve the mental health outcomes of young people.

Only articles in English were considered, which left out research published in other languages. Also, while we tried to do a comprehensive search, we might have missed some studies, including evidence from the gray literature, because we did not search for specific mental illnesses (e.g., schizophrenia) or other sources (e.g., dissertations, reports). This may introduce publication bias, as studies with positive or negative results may be more or less likely to be published in peer-reviewed journals. Also, as many of the studies were complex interventions with several parts it was hard to separate which parts influenced our target outcomes specifically. Another limiting aspect relates to outcome categorization. The studies used different tools to measure the same constructs, which makes them not easily comparable. We tried to assign each outcome to the best category to reduce heterogeneity from differences in outcome definition. Moreover, the overall risk of bias assessment showed that most studies have high or moderate risks, while few have low risk. The rather unfavorable assessment is partly due to the fact that studies do not report on all assessed aspects. We suggest that intervention studies follow reporting guidelines to overcome this information gap and offer more reliable results. Finally, given that our research question was broad and aimed to examine the effects of web-based interventions on various outcomes related to mental health literacy, stigma, and help-seeking intentions/attitudes behavior in youth, we anticipated that there would be a limited number of RCTs on this topic and excluding non-RCTs reduces the power and generalizability of our findings, so we included all RCT and non-RCT articles. However, this may introduce heterogeneity and bias in the pooled estimates, as non-RCTs are more prone to confounding, selection bias, and performance bias than RCTs, which may lead to overestimation or underestimation of the intervention effects [46]”.