Background
Harmful alcohol use is a major contributor to the global burden of disease [
1] and is considered to be the main cause of nearly 4% of global mortality [
2]. The magnitude of this burden partly results from the wide treatment gap, which represents the difference between the prevalence of harmful alcohol use and the number of individuals receiving treatment for harmful alcohol use [
3]. The development and use of innovative treatment options (e.g. Internet-based interventions) could narrow the treatment gap for harmful alcohol use.
Internet-based interventions are seen as an attractive option for people who meet harmful alcohol use criteria and who have relatively mild conditions [
4‐
7]. Moreover, these interventions have been found effective in addressing harmful drinking behaviour and improving quality of life (e.g. [
7‐
9]; for a review see: [
10]). There are also indications that Internet-based alcohol interventions are cost-effective [
11].
However, there is notable heterogeneity in treatment outcomes, which several recently published studies have demonstrated. Postel and colleagues [
9] found that three months after baseline, 32% of the alcohol E-therapy participants had not reached a drinking level within the British Medical Association (BMA) drinking guideline limits (no more than 21 standard glasses per week for men, 14 standard glasses per week for women). Riper et al. [
7] conclude that after six months, the majority (83%) of the participants in their ‘Drinking Less’ Internet-based self-help program still consumed more alcohol than the BMA guideline recommends. A study by our research group [
8] found that 71% of the self-help program participants had an unsuccessful treatment outcome six months after baseline.
A number of studies have explored clinical outcome predictors of face-to-face alcohol therapy. These studies have studied the predictive potential of a large number of possible baseline predictors regarding alcohol consumption, other substance use, psychosocial functioning, and demographic characteristics. The research results are mixed: while some authors have identified relevant predictors, other authors have not been able to replicate this. Adamson and colleagues concluded in a recent review [
12] that attempts to synthesize findings on patient predictors of alcohol treatment outcome were also hampered by lack of agreement of the best measure for predictor variables.
For the purpose of the current study, a literature search on PubMed / MEDLINE (1980–2011) using as search term the title words (alcohol OR drink* OR substance *use*) AND (predict* OR outcome* OR treatment) resulted in 5041 articles. The abstracts of potentially relevant articles were screened and those that were considered relevant for our literature overview were retrieved, to identify studies in which the same baseline and/or outcome variables were used as available in our dataset). Based on expert advice, 5 more articles were added to our literature database. A brief overview of the findings reported in 17 publications with the highest relevance to our literature review is presented below.
A number of studies have found a negative relationship between the severity of drinking problems at baseline and clinical outcome [
13‐
15]. McKay & Weiss [
16] however report a positive relationship between baseline drinking problems and clinical outcome. Age of first alcohol consumption, overall duration of alcohol problems and number of previous quit attempts have been linked to treatment outcome [
13]. With regard to psychosocial functioning, several measures have been found to predict intervention outcome: self-efficacy [
17,
18], motivation to change [
18‐
21], internal locus of control, coping skills, low levels of experienced stress, concern from partners or peers, and a stable social environment [
13,
16,
22‐
24]. Social problems and psychopathology are found to negatively correlate with successful outcome [
13,
16,
22,
25]. Particular demographic characteristics, such as age, sex, education level, being of foreign origin, and general socioeconomic status have been linked to clinical outcome [
18,
22,
26,
27], although these findings have not always been replicated [
16,
22,
28]. To date, only one paper by Riper and colleagues [
29] has assessed which baseline variables predict clinical outcome in Internet-based alcohol interventions. The authors concluded that being female and highly educated were correlated with receiving benefits from an Internet-based self-help intervention.
All in all, it is difficult to define a core set of predictors that should be included in a model aiming to predict treatment outcome. Thus, a large number of possible predictors will be considered for inclusion in the current analysis. Interactions between the possible predictors will also be taken into account, with the aim to test whether a valid predictive model, which can be used as a screening or decision-support tool, can be found. While it is generally assumed that a large sample size will be needed in order to construct and test a model comprising a large number of predictors (and possibly an even larger number of interactions among these predictors), this is not necessarily true [
30]. In the current study, a classification tree analysis will be performed using recursive partitioning. Using this data-driven technique, it is feasible to analyze multi-dimensional data in a dataset with a limited sample size [
31]. This is an important advantage of recursive partitioning over generalized linear modelling regression analysis. Recursive partitioning can be used to identify variables that are of relevance to future research, but also to create data-driven, evidence-based treatment decision support tools [
30]. For example, Swan and colleagues [
32] identified relevant variables when examining the heterogeneity of their outcomes from a smoking cessation intervention using recursive partitioning. Others [
33] have used recursive partitioning in an analysis of pregnant women’s responses to substance use questions, which resulted in a three-item Substance Use Risk Profile-Pregnancy scale. In the current study, recursive partitioning is used in an analysis of data from a randomized controlled trial (RCT) performed in the Netherlands, comparing the effectiveness of Internet-based therapy and Internet-based self-help for harmful alcohol use. Results from this study have been published elsewhere [
8]. The current analysis will be performed in order to test whether a screening instrument with acceptable sensitivity and specificity can be developed.
Discussion
The most relevant classification variables to predict treatment outcome (6 months post-randomization) were whether a participant lived alone (living alone) and interpersonal sensitivity (measured in a subscale of the BSI). Participants living alone had a relatively low probability of positive treatment outcome, whereas participants who lived with others and scored high on interpersonal sensitivity had a relatively high probability of positive treatment outcome. Participants in a shared living condition and low score on interpersonal sensitivity had a moderate probability of positive treatment outcome. With the exception of the BSI global severity index, the three subgroups did not differ significantly on any of the other baseline measures, after Bonferroni correction.
It is remarkable that from 46 predictors found in the literature, only five remain candidate predictors for the recursive partitioning procedure after univariate regression analysis. The exclusion criterion for predictors (p ≥ 0.15) can even be considered lenient. Against a conventional significance level of α = 0.05, living alone would have been the only significant predictor (p = 0.02) out of the 46 tested predictors. This indicates that either the dataset in this analysis is different from other harmful alcohol use treatment datasets used to explore outcome predictors (e.g. due to a difference between face-to-face and Internet-based interventions), or it might indicate methodological flaws in some other studies (e.g. insufficient correction for multiple testing which would result in many false positive test results in explorative studies).
The reported results were moderately robust against small fluctuations in the sample based on which the classification tree was constructed. The classification tree predicts above chance level: when making conservative assumptions, the instrument has a high specificity, and when the assumptions are more progressive, a high sensitivity is obtained. However, the utility of this screening instrument as a stand-alone decision tool is limited, considering the low sensitivity under the conservative assumption, and the low specificity under the progressive assumption.
Limitations
The results of this study should be considered in light of its limitations. Only those limitations related to the current recursive partitioning analysis will be discussed; limitations regarding the RCT and interpretation of its clinical results have been discussed elsewhere [
8].
The sample size in the RCT had sufficient power to draw conclusions for the main research questions (regarding effectiveness of the interventions). However, for secondary explorative analyses of subgroups performed in the current study, the sample size was somewhat small. Although recursive partitioning does not use significance tests (and therefore no concept of power applies to guide a power or sample size analysis) [
58], it is generally conceived that a sample size of 100–150 is the minimum for making recursive partitioning worth attempting [
59]. Based on this view, the sample size of
n = 136 in the current study is just about the required minimum. In order to achieve this sample size, data from IT and IS participants had to be pooled. The underlying assumption of this pooling is that the relation between predictors and outcome is comparable for these two interventions. Some support for this assumption may be inferred from results from project MATCH. This project was an 8-year, multi site, $27-million investigation that studied which types of alcoholics respond best to which forms of treatment. The results with regard to patient matching in this study suggest that triaging clients to a particular treatment, at least based on the attributes and treatments studied in project MATCH, will not assure treatment success as previously believed [
60]. This means that the baseline matching variables of the project MATCH sample do not differentiate between which form of treatment will be most effective for a specific client, and thus that the relation between these predictors and outcome is comparable for the different interventions. To what extend these findings in project MATCH can be transferred to low-intensity Internet-based alcohol interventions is a matter of debate.
Recursive partitioning is primarily a data driven approach. Debate remains as to whether recursive partitioning is prone to over-fitting the data or not. Either way, the resulting classification tree is always one of the possible solutions rather than the only solution to fit data. However, as in this study a univariate regression analysis was performed to empirically support the selection of candidate predictor variables, the current classification tree was the only possible solution when following this procedure. Another common critique on recursive partitioning is its sensitivity to small changes in the data used. The robustness of the presented model is assessed based on a resampling approach and was found to be moderately stable. A methodologically stronger approach would have been to use two separate datasets, the first to construct the classification tree, and the second to evaluate the model and to calculate the statistics presented in Tables
2 and
3. Therefore a validation of the model in a new sample would be desirable before future use of the presented model is considered.
The current study is performed using data from only one trial on Internet-based alcohol interventions, and in this trial, many of eligible participants refused to participate. Although compared to all 832 people who were eligible to participate, the participants who were included reported somewhat higher, but not significantly higher baseline AUDIT scores (
p = 0.11, see [
8]), generalizations beyond this study population are only possible to a limited extent. A number of factors may play a role in successful outcome resulting from an intervention. Treatment outcome itself is one of these factors, however other factors impact a participant’s recovery process over time. This study did not allow for disentanglement of treatment effects and other effects (e.g. natural recovery rates). Given this fact, the current classification tree should in no way be regarded as a causal model of treatment response, but merely as the unique outcome of the recursive partitioning approach taken in combination with the current dataset.
Strengths
The main strength of the study is the thorough statistical approach. A selection of possible predictors was made based on the literature on outcome predictors in alcohol treatment studies. The recursive partitioning software was used in such a manner that the formulation of small, clinically irrelevant subgroups was prevented. The robustness of the presented tree (Figure
2) was tested using a leave-one-out jackknifing approach, in which it was shown that in the majority of the resampled datasets, the same classification tree was formed. The presented classification tree was tested by classifying actual cases in the bootstrapped samples of the dataset.
Implications and future research
If the results presented in this paper find themselves replicated or extended by future studies, this ultimately could lead to the development of an evidence based intervention allocation decision support system. This system could be helpful for problem drinkers contemplating whether participation in an Internet-based intervention would be profitable to them. Currently, it is often reported that Internet-based interventions lead to a favourable treatment outcome for some of the participants, but not for others. This is also the case for face-to-face addiction treatment. An instrument with predictive validity to profile those that will likely have favourable treatment outcome after addiction treatment interventions is therefore highly needed [
61]. The results of the current study contribute to the development of such an instrument for Internet-based alcohol interventions.
However, before the two variables living alone and interpersonal sensitivity can be used in a profiling instrument, a necessary step for future research would be to test the presented model on an alternative dataset. Although the robustness of the presented tree has been tested using resampling in this study, applying the tree on a different dataset (but with the predictor and outcome variables measured) would further support its validity if the predictive validity of the two variables can be replicated. If this validation would be successful, an advantage of the two currently presented predictors (living alone, interpersonal sensitivity) is that they are easy to measure as they are based on only a few self-report items, which would make the development of a self-report decision support system more practically feasible.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
MB, MWJK, GMS designed the RCT from which the data reported in this article stems; MB, MWJK, GMS conceived the study presented in this manuscript. MB, MWJK planned and performed the statistical analysis for this manuscript; MB, MWJK, GMS drafted the submitted manuscript. All authors read and approved the final manuscript.