http://orcid.org/0000-0002-6590-1606
Figures
Abstract
Background
Although the effectiveness of therapist-guided internet-based cognitive behaviour therapy (ICBT) for treating depression has been well documented, knowledge of outcome predictors and risk factors associated with lower treatment response is limited, especially when the treatment has been conducted within a naturalistic clinical setting. Identification of such factors is important for clinicians when making treatment recommendations.
Methods
Data from a large cohort (N = 1738) of adult outpatients having been treated with ICBT for depression at an outpatient psychiatric clinic were analysed. A multilevel modelling approach was used to identify patient and treatment variables associated with the speed of recovery during treatment using weekly measurements of the Montgomery Åsberg Depression Rating Scale Self-Rated (MADRS-S).
Outcomes
Adhering to the treatment, perceiving it as credible and working full-time emerged as predictors of a faster pace of recovery and were also associated with a lower level of depression at the end of treatment. Higher pre-treatment depression and sleep problems were associated with a greater improvement rate, but predicted higher depression after treatment. Having a history of psychotropic medication was associated with both slower improvement and higher post-treatment depression.
Conclusion
Perceived credibility of ICBT is a strong predictor of treatment response. Assessing patient beliefs and expectations may be a useful aid for clinicians when identifying those who are more or less likely to benefit from ICBT. Helping patients improve expectations prior to treatment may be an important goal for clinicians during the initial assessment phase.
Citation: El Alaoui S, Ljótsson B, Hedman E, Svanborg C, Kaldo V, Lindefors N (2016) Predicting Outcome in Internet-Based Cognitive Behaviour Therapy for Major Depression: A Large Cohort Study of Adult Patients in Routine Psychiatric Care. PLoS ONE 11(9): e0161191. https://doi.org/10.1371/journal.pone.0161191
Editor: Emilio Russo, University of Catanzaro, ITALY
Received: May 3, 2016; Accepted: August 1, 2016; Published: September 12, 2016
Copyright: © 2016 El Alaoui et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Study data are stored at the repository of Karolinska Institutet. With respect to the legal framework regulating access to research data in Sweden, the data are not freely accessible due to regulations regarding personal integrity in research, public access and privacy; each request is therefore assessed by the Karolinska Institutet ethics committee, and approval of data access can be given after this assessment. To request data, contact Karolinska Institutet by e-mail at info@ki.se, or at Karolinska Institutet, SE-171 77, Stockholm, Sweden.
Funding: This study was supported by a research grant (ALF 20130413, to Nils Lindefors) from Stockholm County Council, a public institution that had no role in study design, data collection, analysis, data interpretation, manuscript writing, or decision to submit for publication. All authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Major depression is one of the most common mental disorders and is related to lower quality of life and higher risk of suicidal behaviours [1–3]. Cognitive behavioural therapy (CBT) is an effective treatment for depression [4]. To increase accessibility of CBT, online methods of treatment delivery such as therapist-guided internet-based CBT (ICBT) has been recognised as a promising administration format yielding effect sizes at par with conventional face-to-face CBT [5, 6]. However, more knowledge on outcome predictors of ICBT is needed and identifying such determinants can be of high value to clinicians when making treatment recommendations or deciding how to proceed with an ongoing treatment. Given that ICBT is a relatively new treatment format, previous prediction research and risk factors for low response mainly concerns conventional face-to-face CBT. Within clinical and sociodemographic domains, some factors appear to be relatively stable predictors across studies. For example, social support seems to be a major protective factor (e.g. being in a relationship has been associated with a better response [7, 8]) while higher illness severity has been found to predict a worse outcome in some studies [7, 9–11] but not in others [12, 13]. There are fewer predictor studies on ICBT for depression. In regard to the prognostic value of pre-treatment illness severity, findings are inconsistent. In a study from 2004 on outcome predictors of ICBT for depression with 71 participants [14], a weak negative relationship between outcome and those who have had repeated episodes of depression was found, suggesting in line with previous depression research [15] that the level of illness severity is related to treatment outcome. On the other hand, one study found no relation between initial level of depression and outcome [16] and in other studies it has been linked to better response [17, 18]. With regard to substance use, a study from 2014 that included 770 of patients (also included in the present study) treated for depression with ICBT did not detect any effect of problematic substance use on depression outcomes after having excluded patients with substance use disorder [19]. Some pharmacotherapy trials in the treatment of depression have found that substance use disorder was associated with worse outcomes [20, 21] whereas other studies have not identified this relationship [22]. Several studies on ICBT have identified both the level of adherence and treatment credibility as important process measures related to treatment success in self-help interventions such as ICBT [23, 24] where perceived credibility of ICBT [25, 26] and adherence to either face-to-face CBT or ICBT [25–29] has been associated with treatment outcome. A recent study of outcome predictors of ICBT for social anxiety disorder found that patients’ expectations of treatment success and the perceived credibility of ICBT is strongly associated with the rate of improvement, where higher levels of perceived credibility predicts a steeper rate of improvement [30].
This study investigated a large cohort of patients having been treated at a clinical unit for internet treatment, which is part of the Psychiatric Clinic Southwest at Karolinska University Hospital Huddinge, Sweden. The clinical effectiveness of ICBT has been evaluated, not only for depression but also for panic disorder and social anxiety disorder [31–33]. No large-scale study of this size (N = 1738) has previously assessed determinants of symptomatic improvement of ICBT for depression within a clinical psychiatric setting.
Methods
Design of the Study
This was a longitudinal study investigating adult patients (N = 1738) who had been treated for depression as part of routine care at the internet psychiatry clinic between 2010 and 2014 were included in the study. Patients could either be referred to the clinic by their general practitioner or register via a self-referral system online. After an online screening procedure, patients were invited to the clinic for a structured diagnostic interview, including the Mini International Diagnostic Interview (MINI) [34], which was performed by a psychiatrist or supervised resident physician. The general guidelines for assessing suitability of ICBT were that patients had to fulfil criteria for major depression according to DSM-IV, be able to write and read in Swedish, not undergo concurrent psychotherapy during the ICBT period and be ≥18 years old and residing in the city county. The following guidelines were followed as exclusion criteria: having severe depression (clinician rated MADRS ≥ 35) and/or moderate to high risk of suicide where monitoring is required; newly (i.e. during the last month) initiated antidepressant medication (ADM); suspected bipolar depression; combinations of comorbidity with high impacts on the complexity of clinical management; low motivation for the internet-format, apathy severe enough to manage self-help, or unable to concentrate on the treatment material; patients with psychosis; in need of drug or alcohol addiction treatment; reading and/or writing difficulties, including language difficulties. Approximately 50% of those who sought treatment were included, and these would typically initiate treatment within one or two days after the psychiatric assessment. The intervention lasted for three months, during which patients were granted access to web-based treatment modules in a step-wise manner and were guided by a licensed psychologist who would provide online support and regular feedback on homework assignments and general progress. Data was collected both during a structured diagnostic interview with a psychiatrist prior to treatment, and from weekly online measurements before, after and during treatment, which have previously been documented as a valid administration format [35]. This research has been approved by the Regional Ethical Review Board in Stockholm, Sweden (no 2011/2091-31/3).
Primary Outcome Measure
The primary outcome measure was the Montgomery Åsberg Depression Rating Scale Self-Rated (MADRS-S) [36]. MADRS-S measures nine clinical characteristics of depression and the total score scale range is 0 to 54. The test-retest reliability of MADRS-S has been shown to be high (r = .80–.94) [36] and Cronbach’s alpha for the study sample at baseline was .80.
Candidate predictor variables
A total of 38 candidate predictors were analysed in order to identify those having a significant effect on the rate of improvement over the course of treatment. Due to the large number of candidate predictors, these were grouped and screened in separate domains prior to inclusion in a combined regression model: socio-demographic variables (6 variables), clinical characteristics (11 variables), comorbidity (9 variables), family history of mental illness (10 variables) and treatment related factors (2 variables).
Within the social-demographic domain, the following variables were analysed: age; gender; relationship status (dichotomized as being single or not); level of education rated on a 7-point scale (1 = less than 7–9 years in school; 2 = 7–9 years in school; 3 = incomplete vocational or secondary school; 4 = vocational school; 5 = secondary school; 6 = university, started but not completed studies; 7 = completed university studies); employment status dichotomized as working full-time or not; and having children.
The clinical characteristics domain included the Clinical Global Impression–Severity Scale [37] (CGI-S) which was rated on a 7-point scale; the AUDIT [38] and the DUDIT [39] which were used to screen for alcohol problems or drug misuse; sleep problems measured with the insomnia severity index [40, 41], duration of illness (years since onset), history of psychotropic medication, history of inpatient psychiatric care, history of depression, attempted suicide, currently on medication, and currently on psychotropic medication.
Presence (yes/no) of any comorbid illness was assessed during the diagnostic interview prior to treatment using the MINI. These included panic disorder, agoraphobia, social anxiety disorder, mild depressive episode, moderate depressive episode, severe depressive episode, recurrent mild depressive episode, recurrent moderate depressive episode, recurrent severe depressive episode, recurrent depression without current symptoms and dysthymia.
Having a family history of mental illness was also assessed during the diagnostic interview prior to treatment. The following set of variables were assessed: having a family history of dependence / substance abuse, bipolar disorder, depression, minor depression, neuropsychiatric condition, anxiety, panic disorder, psychosis, social anxiety disorder, social anxiety disorder-like symptoms, suicide attempts or suicide completed.
Finally, treatment process factors included two variables. First, a measure of perceived treatment credibility measured during the second week in treatment, measured with the credibility/expectancy scale [42] where patients’ attitudes to the credibility of the treatment and expectancy regarding treatment effectiveness were rated on a 10-point scale (0 = not at all to 10 = very much). Second, treatment adherence was measured at post-treatment, reflecting the degree of use of the ICBT program operationalized as the total number of activated treatment modules.
Statistical Analyses
Longitudinal multilevel modelling of repeated measurements was used to identify the variables that had a significant influence on the rate of improvement between pre- and post-treatment. These models were built using the linear mixed-effects models procedure in SPSS version 22. Multilevel modelling was considered to be a suitable analytical approach for several reasons. For example, a multilevel model is able to make use of all available data, rendering it a full intent to treat analysis, thus not omitting cases with incomplete observations from the analysis (where occurrence of missing values lead to more missing values, e.g. through list-wise deletion). Second, it takes into account the hierarchical nature and dependency of the data (i.e., repeated measurements are nested within patients). Of primary interest was the effect of predictor variables on the rate of change (i.e. the slope of the regression line).
Due to the relatively large amount of candidate predictors, all variables were first grouped into separate domains as described above. Within each domain, a separate multilevel model was developed in a stepwise manner where variables were successively evaluated as to whether they significantly had an effect on the rate of symptomatic improvement, a procedure previously adopted by Fournier et al. [43] and Amir et al. [44]. In the first step (Step 1), a model which included all candidate predictors within a specific domain, and their interaction with time (predictor × time), was tested. Only those variables that had an interaction with time at the p < .20 level were included in a new model (Step 2). In Step 3, those variables that met the p < .10 level were kept and finally assessed in Step 4, at the p < .05 level. This procedure was repeated for all predictor domains. Finally, all predictors that were significant at the p < .05 level were combined in a final model which consequently tested the predictive value of each predictor from all domains while controlling for the effects of the other variables.
All models contained two levels, where the first level was the repeated MADRS-S measurements from each patient, and the second level consisted of the patients with their individual characteristics. Since the primary interest in this study was the total change over the course of treatment (i.e. improvement between pre- and post-treatment assessment, as opposed to weekly change) and status at post-treatment, the time variable was re-centred to post-treatment (i.e. the value at the intercept reflects level of depression at post-treatment) and re-coded the values to range between -1 and 0. This approach has previously been adopted by Smits and colleagues [45]. Also, all predictor variables were standardized prior to analysis in order to facilitate comparison of slope coefficients for the different predictors [45].
Ethics Statement
The study, including its consent procedure, was approved by the Regional Ethical Review Board in Stockholm, Sweden (no 2011/2091-31/3). This research was conducted as a retrospective cohort study; therefore, active informed consent was not required by the ethics committee. However, all participants were provided with a choice of participation through an opt-out methodology. Since passive consent does not violate the option of providing choice and increases the likelihood of having a representative sample, this approach is considered to be an efficient procedure for registry data [46, 47].
Results
Sample Description
Patient characteristics are presented in Table 1.
Attrition and Adherence
Table 2 presents the number of observations for each measurement occasion of the outcome variable (MADRS-S) used in the analyses.
As shown in Table 2, 1447 patients provided post-treatment scores on the main outcome measure, whereas 291 patients had missing data on this measurement occasion. There was a significant difference in baseline level of depression between patients with no missing data at post-treatment (M = 22.14, SD = 6.51) and patients with missing post-treatment scores (M = 23.42, SD = 7.43) conditions; t(1736) = -2.99, p < .05.
The mean level of adherence was 7.21 (SD = 3.09) activated treatment modules out of 10. The median level was 8 with and the interquartile range (IQR) was 5.
Predictors of symptomatic change
The results of the final multilevel model predicting symptomatic improvement are presented in Table 3. The outcome variable “Level of depression” reports the estimated change over the study period in terms of MADRS-S scores, predicting that patients will improve by about 7.89 points and that after treatment patients will have an estimated level of depression of about 13.51 points on the MADRS-S. The effect of each predictor on the rate of improvement (i.e. interaction of predictor × time) is illustrated in Figs 1–5. In order to depict the effects of the predictors, continuous predictor variables were categorized into a “high” and “low” group, where a “high” score was operationalized as one standard deviation above the mean and a “low” score as one standard deviation below the mean. The division in groups regarding the levels “high” and “low” of the predictors were used for the graphic illustrations. In the analysis, however, all data from all participants was used.
According to the final model, working full-time, high treatment adherence and ratings of perceived treatment credibility predicted both greater rate of improvement as well as lower post-treatment level of depression (Figs 1–3).
Initial level of depression predicted a greater rate of improvement but also a higher level of post-treatment scores. Neither gender nor relationship status were related to individual differences in rate of improvement, although being single was associated with a higher post-treatment level of depression.
Within the domain of clinical characteristics, patients with a history of psychotropic medication showed both a slower rate of improvement as well as a higher level of post-treatment depression (Fig 4). Increasing severity of self-reported sleep problems prior to treatment was related to a faster rate of improvement (Fig 5) but also higher post-treatment depression.
Neither the presence of any of the co-morbid diagnoses tested nor having a family history of mental illness was related to the rate of improvement.
Discussion
Main Findings
The aim of this study was to identify patient and treatment characteristics associated with symptomatic improvement in a large cohort of depressive patients after ICBT within a naturalistic clinical setting. The degree to which patients adhered to the treatment had the strongest overall relationship with the rate of improvement. The average adherence in the study was about 70%, which is similar to the median adherence of 69% reported in a recently published review of ICBT trials of treating depression [48]. However, since adherence only can be fully measured after the intervention, it might be of more use to clinicians to identify pre-treatment variables–or process variables measured early in treatment–that have prognostic value. Thus, treatment credibility was identified as the strongest prognostic variable, an observation in line with previous research having identified a significant association between treatment credibility and the effectiveness of ICBT [25, 26]. The findings of the relationship between perceived credibility of the treatment and patients’ expectancies–including expectations of the effects of treatment–could indicate the involvement of a placebo effect. The question then concerns whether the observed treatment response is caused by the intervention or by patient’s expectations. However, a patient is not a passive receiver of therapy, but rather actively interacting with the treatment content to produce changes in behaviour and cognitions. It is logical then, that expectations may have a moderating effect on the relationship between being exposed to ICBT and responding to treatment; a placebo effect could therefore be regarded as an integral part of every therapeutic intervention, indicating a central role of perception in treating depression. It may be noted that a meta-analysis of the placebo effect in antidepressant trials reported that the placebo effect accounted for as much as 68% of the treatment response [49]. In the case of ICBT, it is reasonable to assume that adherence and credibility are correlated, and that the relationship between credibility and outcome may be mediated by adherence. Further, although having a correlation between the credibility and outcome might not explain an overall placebo effect, it could still be that those who have low credibility make proper improvements.
We found no significant relationship between gender and rate of improvement, which is in line with previous predictor studies on ICBT for depression where no association could be detected between gender and therapeutic outcome [14, 50]. Since this study had a relatively large cohort, it might have increased the possibility of identifying this relationship.
Also in line with research on conventional treatments for depression and the protective effect of social relationships (e.g. being married has been linked to a better outcome [7, 8]), we found that being single was associated with a worse end state. Further, having a history of psychotropic medication predicted both a worse end state and a slower improvement rate, whereas working full-time emerged as a protective factor. Although sleep problems were associated with a slightly steeper improvement rate (perhaps since addressing sleep problems is addressed during treatment in a module specifically devoted to this problem area), we also found that these patients were more depressed after treatment which is in line with previous findings that patients suffering from both conditions more seldom achieve sufficient relief [51]. In line with a previous ICBT study on the predictive role of alcohol use on treatment outcome [19], our multilevel model found no significant relationship between alcohol use and rate of improvement when controlling for other factors within the same model. Difficulties in detecting such a relationship might be partly explained by the fact that patients who have a significantly problematic alcohol use were generally excluded from the treatment and thus from the study sample.
Study Limitations
The present study was not a randomized controlled trial; therefore, we could not compare the proportion of patients with non-response with a control group of patients receiving no treatment for depression. However, there are a number of studies on the natural course of depression who report that a significant proportion of patients within primary care generally do not recover spontaneously [52, 53]. Also, in a recent study on ICBT for depression, patients receiving treatment as usual (TAU) improved significantly less compared to the ICBT group [54]. Another limitation concerns the external validity, since approximately 50% of patients who sought treatment were excluded from ICBT treatment; the results are therefore based on predictor variables of a biased and non-random sample which may have implications for which populations these results may be generalized to.
Some of the exclusion criteria such as low motivation, severe apathy, and difficulty concentrating are related to basic criteria for diagnosing depressive conditions. It may seem counter-intuitive that some of these factors were used to diagnose patients and to exclude patients at the same time. However, to propose ICBT it is essential with an overall assessment of the ability and motivation to manage the specific treatment format. Hence, once a depressive diagnosis has been established, the clinician must also make an informed overall evaluation regarding the severity of the condition and the ability of the patient to manage the treatment. Mostly, exclusion of depression patients is made on the same basis as for face-to-face CBT, i.e. when non-psychological treatment is more indicated (high severity), or the patient has other problems that need to be investigated more (e.g. suspicious bipolar II disorder) or specifically treated (e.g. PTSD). However, there is a component for the position to ICBT that places additional demands on the patient, namely the ability to concentrate on reading and understanding written text. When this ability is unclear we often have patient samples read the paper module or read the first section of the module to see whether the patient regards the treatment form as suitable.
We also found a significant difference in baseline level of depression between patients who dropped out versus those who provided post-treatment scores on the MADRS-S. However, the mean difference was only -1.3 (95% CI: -2.13 to -0.44) and we believe that the assumption used in the multilevel models (i.e. that the data is missing at random, MAR, not missing completely at random, MCAR) is valid and that this difference is ignorable and have not likely biased the predicted estimates. Further, this analysis indicates MAR on the grounds that we found an observed variable that significantly explains missingness. Had the difference been greater, it had still been in accordance with the MAR assumption as long as the observed variable (pre-value) is included in the analysis model used in the study (which it is in the present study).
Conclusion
Several factors of importance for ICBT treatment outcome have been identified. Results indicate that assessing patients’ perceived treatment credibility of ICBT may be highly informative with regard to probable treatment response. These findings may provide clinicians with supportive information when considering patients for ICBT.
Author Contributions
- Conceptualization: SEA BL EH CS VK NL.
- Data curation: SEA BL EH CS VK NL.
- Formal analysis: SEA BL EH CS VK NL.
- Funding acquisition: NL.
- Investigation: SEA BL EH CS VK NL.
- Methodology: SEA BL EH CS VK NL.
- Project administration: SEA BL EH CS VK NL.
- Resources: SEA BL EH CS VK NL.
- Software: SEA BL EH CS VK NL.
- Supervision: NL.
- Validation: SEA BL EH CS VK NL.
- Visualization: SEA BL EH CS VK NL.
- Writing – original draft: SEA BL EH CS VK NL.
- Writing – review & editing: SEA BL EH CS VK NL.
References
- 1. Beautrais AL, Joyce PR, Mulder RT, Fergusson DM, Deavoll BJ, Nightingale SK. Prevalence and comorbidity of mental disorders in persons making serious suicide attempts: a case-control study. Am J Psychiatry. 1996;153(8):1009–14. Epub 1996/08/01. pmid:8678168.
- 2. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime Prevalence and Age-of-Onset Distributions of DSM-IV Disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):593–602. pmid:15939837
- 3. Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, et al. Disability and quality of life impact of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatr Scand Suppl. 2004;(420):38–46. Epub 2004/05/07. pmid:15128386.
- 4. Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev. 2006;26(1):17–31. pmid:16199119.
- 5. Hedman E, Ljotsson B, Lindefors N. Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert review of pharmacoeconomics & outcomes research. 2012;12(6):745–64. pmid:23252357.
- 6. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PloS one. 2010;5(10):e13196. Epub 2010/10/23. pmid:20967242; PubMed Central PMCID: PMC2954140.
- 7. Jarrett RB, Eaves GG, Grannemann BD, Rush AJ. Clinical, cognitive, and demographic predictors of response to cognitive therapy for depression: a preliminary report. Psychiatry research. 1991;37(3):245–60. pmid:1891508.
- 8. Sotsky SM, Glass DR, Shea MT, Pilkonis PA, Collins JF, Elkin I, et al. Patient predictors of response to psychotherapy and pharmacotherapy: findings in the NIMH Treatment of Depression Collaborative Research Program. The American journal of psychiatry. 1991;148(8):997–1008. pmid:1853989.
- 9. Simons AD, Gordon JS, Monroe SM, Thase ME. Toward an integration of psychologic, social, and biologic factors in depression: effects on outcome and course of cognitive therapy. Journal of consulting and clinical psychology. 1995;63(3):369–77. pmid:7608349.
- 10. Hoberman HM, Lewinsohn PM, Tilson M. Group treatment of depression: individual predictors of outcome. Journal of consulting and clinical psychology. 1988;56(3):393–8. pmid:3397431.
- 11. Thase ME, Simons AD, Cahalane J, McGeary J, Harden T. Severity of depression and response to cognitive behavior therapy. The American journal of psychiatry. 1991;148(6):784–9. pmid:2035722.
- 12. Hollon SD, DeRubeis RJ, Evans MD, Wiemer MJ, Garvey MJ, Grove WM, et al. Cognitive therapy and pharmacotherapy for depression. Singly and in combination. Arch Gen Psychiatry. 1992;49(10):774–81. pmid:1417429.
- 13. Shapiro DA, Barkham M, Rees A, Hardy GE, Reynolds S, Startup M. Effects of treatment duration and severity of depression on the effectiveness of cognitive-behavioral and psychodynamic-interpersonal psychotherapy. Journal of consulting and clinical psychology. 1994;62(3):522–34. pmid:8063978.
- 14. Andersson G, Bergstrom J, Hollandare F, Ekselius L, Carlbring P. Delivering cognitive behavioural therapy for mild to moderate depression via the Internet: Predicting outcome at 6-month follow-up. Verhaltenstherapie. 2004;14(3):185–9. pmid:WOS:000225018800003.
- 15. Hamilton KE, Dobson KS. Cognitive therapy of depression: Pretreatment patient predictors of outcome. Clin Psychol Rev. 2002;22(6):875–93. Pii S0272-7358(02)00106-X pmid:WOS:000177307300003.
- 16. Proudfoot J, Goldberg D, Mann A, Everitt B, Marks I, Gray JA. Computerized, interactive, multimedia cognitive-behavioural program for anxiety and depression in general practice. Psychol Med. 2003;33(2):217–27. pmid:12622301.
- 17. Ruwaard J, Schrieken B, Schrijver M, Broeksteeg J, Dekker J, Vermeulen H, et al. Standardized web-based cognitive behavioural therapy of mild to moderate depression: a randomized controlled trial with a long-term follow-up. Cognitive behaviour therapy. 2009;38(4):206–21. pmid:19221919.
- 18. Spek V, Cuijpers P, Nyklicek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med. 2007;37(3):319–28. pmid:17112400.
- 19. Gajecki M, Berman AH, Sinadinovic K, Andersson C, Ljotsson B, Hedman E, et al. Effects of baseline problematic alcohol and drug use on internet-based cognitive behavioral therapy outcomes for depression, panic disorder and social anxiety disorder. PloS one. 2014;9(8):e104615. pmid:25122509; PubMed Central PMCID: PMC4133364.
- 20. Davis LL, Wisniewski SR, Howland RH, Trivedi MH, Husain MM, Fava M, et al. Does comorbid substance use disorder impair recovery from major depression with SSRI treatment? An analysis of the STAR*D level one treatment outcomes. Drug and alcohol dependence. 2010;107(2–3):161–70. pmid:19945804.
- 21. Howland RH, Rush AJ, Wisniewski SR, Trivedi MH, Warden D, Fava M, et al. Concurrent anxiety and substance use disorders among outpatients with major depression: clinical features and effect on treatment outcome. Drug and alcohol dependence. 2009;99(1–3):248–60. pmid:18986774.
- 22. Watkins KE, Paddock SM, Zhang L, Wells KB. Improving care for depression in patients with comorbid substance misuse. The American journal of psychiatry. 2006;163(1):125–32. pmid:16390899.
- 23. Geraghty AW, Wood AM, Hyland ME. Attrition from self-directed interventions: investigating the relationship between psychological predictors, intervention content and dropout from a body dissatisfaction intervention. Soc Sci Med. 2010;71(1):30–7. pmid:20400220.
- 24. Aydos L, Titov N, Andrews G. Shyness 5: the clinical effectiveness of Internet-based clinician-assisted treatment of social phobia. Australasian psychiatry: bulletin of Royal Australian and New Zealand College of Psychiatrists. 2009;17(6):488–92. pmid:20001373.
- 25. Hedman E, Andersson E, Ljotsson B, Andersson G, Andersson E, Schalling M, et al. Clinical and genetic outcome determinants of internet- and group-based cognitive behavior therapy for social anxiety disorder. Acta psychiatrica Scandinavica. 2012;126(2):126–36. pmid:22320999.
- 26. Nordgreen T, Havik OE, Ost LG, Furmark T, Carlbring P, Andersson G. Outcome predictors in guided and unguided self-help for social anxiety disorder. Behaviour research and therapy. 2012;50(1):13–21. Epub 2011/12/03. pmid:22134140.
- 27. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. The British journal of psychiatry: the journal of mental science. 2009;195(5):440–7. pmid:19880935.
- 28. Rapee RM, Abbott MJ, Baillie AJ, Gaston JE. Treatment of social phobia through pure self-help and therapist-augmented self-help. The British journal of psychiatry: the journal of mental science. 2007;191:246–52. pmid:17766766.
- 29. Hilvert-Bruce Z, Rossouw PJ, Wong N, Sunderland M, Andrews G. Adherence as a determinant of effectiveness of internet cognitive behavioural therapy for anxiety and depressive disorders. Behaviour research and therapy. 2012;50(7–8):463–8. pmid:22659155.
- 30. El Alaoui S, Ljotsson B, Hedman E, Kaldo V, Andersson E, Ruck C, et al. Predictors of symptomatic change and adherence in internet-based cognitive behaviour therapy for social anxiety disorder in routine psychiatric care. PloS one. 2015;10(4):e0124258. pmid:25893687; PubMed Central PMCID: PMC4404057.
- 31. El Alaoui S, Hedman E, Kaldo V, Hesser H, Kraepelien M, Andersson E, et al. Effectiveness of Internet-Based Cognitive-Behavior Therapy for Social Anxiety Disorder in Clinical Psychiatry. Journal of consulting and clinical psychology. 2015. pmid:26009780.
- 32. Hedman E, Ljotsson B, Kaldo V, Hesser H, El Alaoui S, Kraepelien M, et al. Effectiveness of Internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord. 2014;155:49–58. pmid:24238951.
- 33. Hedman E, Ljotsson B, Ruck C, Bergstrom J, Andersson G, Kaldo V, et al. Effectiveness of Internet-based cognitive behaviour therapy for panic disorder in routine psychiatric care. Acta psychiatrica Scandinavica. 2013. pmid:23406572.
- 34. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. The Journal of clinical psychiatry. 1998;59 Suppl 20:22–33;quiz 4–57. Epub 1999/01/09. pmid:9881538.
- 35. Hedman E, Ljótsson B, Rück C, Furmark T, Carlbring P, Lindefors N, et al. Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behavior. 2010;26:736–40.
- 36. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta psychiatrica Scandinavica. 1994;89(1):21–8. Epub 1994/01/01. pmid:8140903.
- 37.
Guy W. Clinical Global Impressions. ECDEU Manual. Rockville, MD: NIMH: US Dept of Health and Human Services; 1976. p. 217–22.
- 38. Allen JP, Reinert DF, Volk RJ. The alcohol use disorders identification test: an aid to recognition of alcohol problems in primary care patients. Preventive medicine. 2001;33(5):428–33. pmid:11676584.
- 39. Berman AH, Bergman H, Palmstierna T, Schlyter F. Evaluation of the Drug Use Disorders Identification Test (DUDIT) in criminal justice and detoxification settings and in a Swedish population sample. European addiction research. 2005;11(1):22–31. pmid:15608468.
- 40. Bastien CH, Vallieres A, Morin CM. Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep medicine. 2001;2(4):297–307. pmid:11438246.
- 41. Thorndike FP, Ritterband LM, Saylor DK, Magee JC, Gonder-Frederick LA, Morin CM. Validation of the insomnia severity index as a web-based measure. Behavioral sleep medicine. 2011;9(4):216–23. pmid:22003975.
- 42. Borkovec TD, Nau SD. Credibility of analogue therapy rationales. Journal of behavior therapy and experimental psychiatry. 1972;3(4):257–60.
- 43. Fournier JC, DeRubeis RJ, Shelton RC, Hollon SD, Amsterdam JD, Gallop R. Prediction of response to medication and cognitive therapy in the treatment of moderate to severe depression. Journal of consulting and clinical psychology. 2009;77(4):775–87. pmid:19634969; PubMed Central PMCID: PMC2810269.
- 44. Amir N, Taylor CT, Donohue MC. Predictors of response to an attention modification program in generalized social phobia. Journal of consulting and clinical psychology. 2011;79(4):533–41. pmid:21707134; PubMed Central PMCID: PMC3207502.
- 45. Smits JA, Hofmann SG, Rosenfield D, DeBoer LB, Costa PT, Simon NM, et al. D-cycloserine augmentation of cognitive behavioral group therapy of social anxiety disorder: prognostic and prescriptive variables. Journal of consulting and clinical psychology. 2013;81(6):1100–12. pmid:23937345.
- 46. Clark AM, Jamieson R, Findlay IN. Registries and informed consent. The New England journal of medicine. 2004;351(6):612–4; author reply -4. pmid:15295059.
- 47. Cassell J, Young A. Why we should not seek individual informed consent for participation in health services research. Journal of medical ethics. 2002;28(5):313–7. pmid:12356960; PubMed Central PMCID: PMC1733638.
- 48. Andrews G, Hobbs MJ, Newby JM. Computerised cognitive behaviour therapy for major depression: a reply to the REEACT trial. Evidence-based mental health. 2016;19(2):43–5. pmid:26993366.
- 49. Rief W, Nestoriuc Y, Weiss S, Welzel E, Barsky AJ, Hofmann SG. Meta-analysis of the placebo response in antidepressant trials. J Affect Disord. 2009;118(1–3):1–8. pmid:19246102.
- 50. Warmerdam L, Van Straten A, Twisk J, Cuijpers P. Predicting outcome of Internet-based treatment for depressive symptoms. Psychotherapy research: journal of the Society for Psychotherapy Research. 2013;23(5):559–67. pmid:23848944.
- 51. Blom K, Jernelov S, Kraepelien M, Bergdahl MO, Jungmarker K, Ankartjarn L, et al. Internet treatment addressing either insomnia or depression, for patients with both diagnoses: a randomized trial. Sleep. 2015;38(2):267–77. pmid:25337948; PubMed Central PMCID: PMC4288608.
- 52. Gledhill J, Garralda ME. The short-term outcome of depressive disorder in adolescents attending primary care: a cohort study. Social psychiatry and psychiatric epidemiology. 2011;46(10):993–1002. pmid:WOS:000294911000007.
- 53. Vuorilehto MS, Melartin TK, Isometsa ET. Course and outcome of depressive disorders in primary care: a prospective 18-month study. Psychol Med. 2009;39(10):1697–707. pmid:WOS:000270260500014.
- 54. Hallgren M, Kraepelien M, Ojehagen A, Lindefors N, Zeebari Z, Kaldo V, et al. Physical exercise and internet-based cognitive-behavioural therapy in the treatment of depression: randomised controlled trial. The British journal of psychiatry: the journal of mental science. 2015;207(3):227–34. pmid:26089305.