Level of Agreement Between Problem Gamblers’ and Collaterals’ Reports: A Bayesian Random-Effects Two-Part Model

The data used in this study were derived from an intervention study (study protocol: Nilsson et al. 2016) investigating the effects of involving CSOs in internet-delivered treatment for problem gambling. The study involves problem gamblers and their CSOs, who could be their spouse, parent, sibling, or friend. The participants were recruited nationwide via the Swedish National Gambling Helpline, by referrals from health-care professionals who were informed about the study, and through advertisements on Facebook and Google. The gamblers had to score 8 or greater on the Problem Gambling Severity Index (PGSI) to be included in the study.¹

The participants signed up on a website and were asked to provide information about gambling, psychological well-being, and relationship satisfaction, as well as background data, such as age, gender, and employment status. All the information was collected through surveys conducted online. After being enrolled in the study, the participants were randomized as a dyad into one of two treatment conditions: one involving only the gambler and one where both members of the dyad received treatment. The treatments were based on cognitive behavioral therapy, and they were internet-based, with therapist support being provided via email and telephone (c.f., Carlbring and Smit 2008). Data are also included from a pilot study (Nilsson et al. 2018) conducted prior to the randomized controlled trial (RCT). The pilot and the RCT data were pooled, since the recruitment and screening processes were identical, and recruitment into the RCT followed immediately after the completion of the pilot study. The data collection for the pilot study lasted from March 2015 to August 2015, while the RCT data collection lasted between September 2015 and December 2017.

During registration, all the gamblers and their CSOs filled out a TLFB report covering the last 30 days. The TLFB instrument is commonly used to collect information about behaviors linked to addiction. Created in order to monitor daily drinking and problem drinking history (Maisto et al. 1979), TLFB has also been adopted for use in other addiction fields, including gambling (Weinstock et al. 2004). The TLFB instrument resembles a calendar, and users are instructed to record the number of days spent gambling and the amount of money lost to gambling each day throughout a 30-day period. The Banff consensus statement (Walker et al. 2006) stipulates that net losses and time spent on gambling are the most important aspects of gambling behavior. Only the baseline TLFB was used, which was answered prior to randomization, so as to avoid the issue of the agreement being influenced by the treatment. The level of agreement on some secondary variables, namely gambling debt and years spent gambling was also investigated.

ICCs were calculated in order to estimate the level of agreement between the gamblers’ and the CSOs’ TLFB reports. ICCs represent a common method of assessing agreement in dyadic research (Kenny et al. 2006). However, it is important to note that most studies that calculate ICCs rely on a model that assumes a normal (Gaussian) response distribution. This assumption is highly unlikely to hold in the case of gambling losses. Indeed, TLFB data concerning gambling losses are frequently heavily right-skewed, and it is not unexpected to see some participants report losses that are several orders of magnitude above the median. Moreover, the average differences between dyads, as well as the discrepancies within dyads, are both likely to be skewed, and the variance is likely to increase for larger reported losses. A further complication stems from the fact that some participants reported no losses. Figure 1a shows how the reports of gambling losses usually appear, that is, right-skewed, with some persons losing large amounts of money, and some reporting no losses. Figure 1b shows that in the presence of zeros, a log(y + c) transformation (whereby a small constant c, is added prior to the transformation in order to get rid of the zeros) does little to remedy the problem; in this paper c = 1 was used.

A flexible model that allows for both the inclusion of zeros and the skewed continuous responses is a two-part model² (Neelon et al. Smith 2016; Olsen and Schafer 2001). A two-part model splits the model into two distinct parts, and it models the presence of zeros using a binary model, while it models the actual non-zero losses using a skewed continuous distribution, such as the gamma or lognormal distribution. By including random effects, it is possible to model the variance both within and between the dyads by fitting each part of the two-part model as a two-level generalized linear mixed-effects model (GLMM). This enables a thorough assessment of the level of agreement between gamblers and their CSOs, since ICCs can be calculated for the reported zero losses (binary part), the level of agreement for dyads that report losses (the conditionally positive continuous part of the model), and the overall distribution, which includes reports of both zeros and losses. In addition, the two GLMMs can be linked via correlated random effects (Liu et al. 2010). For the dyads’ reports concerning gambling losses, correlated random effects represents a reasonable assumption, meaning that it is likely that dyads with higher reported losses (continuous part of the model) would exhibit a lower likelihood of reporting a zero (in the binary part of the model). However, in this study, only a small number of participants reported zero losses, which raised the question as to whether including the correlation between random effects would introduce more bias than simply ignoring the correlation. For this reason, results derived from both models are reported, and the models’ performances are assessed using Monte Carlo simulations. It was also investigated whether there were any systematic differences between the gamblers’ and the CSOs’ reports. To allow for comparisons with other gambling studies (e.g., Petry et al. 2006), Spearman’s rho is also reported.

In order to investigate whether the level of agreement differed as a function of the type of relationship the CSO had with the gambler (for instance, if they were a parent or a partner), a model was fit wherein both the random effects and the parameters of the GLMM’s distribution family were allowed to be different for each CSO relationship type. This allows for different variances both within and between the dyads as a function of the type of relationship. For the gamma distribution, this meant that the shape parameter was allowed to differ as a function of the relationship type.

The variance components derived from a GLMM are typically given on the link scale. However, since the focus of this paper was in the level of agreement in the actual TLFB reports, the ICCs were calculated on the observed data scale, which was achieved by calculating the variances both within and between the dyads on the original data scale by integrating over the random effects for all the posterior samples using the methods described in de Villemereuil et al. (2016). The difference between the levels of agreement for the CSOs who were parents or partners could then be compared using the posterior samples by taking the difference.³

In order to validate the performance of the proposed two-part GLMM in terms of assessing the level of agreement concerning gambling losses, a small Monte Carlo simulation was performed and each simulation involved 5000 replications. Relative bias for the posterior medians were investigated, as well as how well the 95% credible intervals (CI) recovered the true values, under assumptions similar to the data observed in this study. The results were also contrasted the results obtained from a model assuming a Gaussian response distribution, that is, a classical two-level LMM. Furthermore, the two-part model was fit with and without a correlation between the binary and continuous parts of the model, when the true data generating process included a correlation. The focus was on the difference in the estimated ICCs from parents and partners.

All computations in this study were performed using R v3.5.1 (R Core Team 2018). The models were fitted using the brms v2.4.0 package (Bürkner 2017), which fits models using the Hamiltonian Monte Carlo (HMC) algorithm via the probabilistic programming language Stan (Carpenter et al. 2017; RStan v2.17.3). Uncertainty in the estimates was summarized using the 95% percentile CIs and point estimates by means of the posterior medians. The different models were compared using leave-one-out cross-validation (LOO-CV) or 10-fold CV (Vehtari et al. 2017). The models’ goodness-of-fits were also investigated using posterior predictive checks (Gelman et al. 2014; Gelman et al. 1996), whereby simulation predictions from a fitted model are compared to the observed data in order to see whether any obvious discrepancies can be detected between that fitted model and the observed data.

The Monte Carlo simulations were performed using resources provided by the Swedish National Infrastructure for Computing (SNIC) at the High-Performance Computing Center North (HPC2N), Umeå, Sweden.

An example of the code used to fit the model and run the simulation can be downloaded from the Open Science Framework https://​osf.​io/​ec5b9/​.

Participants belonging to the same dyad could sign up, and therefore complete the baseline measures, at different times. Since the TLFB instrument is intended to capture losses during the last month, differences in the completion time within the dyads were investigated. In total, 21/154 (14%) of the dyads had an absolute difference of more than 1 week, while 84/154 (55%) of the dyads signed up on the same day. In order to retain as many participants as possible, while still ensuring that the ratings within the dyads covered approximately the same period of time, it was decided to exclude those dyads that had registration dates that differed by more than 7 days; this decision was made prior to analyzing any data (Fig. 2 shows the participant flow). Table 1 presents the background information for the overall sample and split per CSO type.

Both the posterior predictive checks and the 10-fold CV indicate the gamma response distribution provided a better fit than the lognormal distribution, as well as a substantively better fit than the Gaussian distribution, as shown in Table 2. Figure 3 shows the posterior predictive distribution comparing the two-part gamma model to the Gaussian LMM.⁴

Overall, there was a fair level of agreement⁵ between the CSOs and the gamblers concerning the amount of money lost when gambling, as shown in Table 3. Table 4 breaks down the results per CSO type and illustrates that the reports obtained from CSOs who were partners tended to exhibit better agreement with the gamblers’ reports when compared to the reports obtained from CSOs who were parents and CSOs who were neither parents nor partners. The difference between the partner CSOs and the parent CSOs was most clear for the losses part, with a smaller difference being seen in the level of agreement regarding the overall reports. However, all the models had point estimates that indicated that the partner CSOs exhibited better agreement than the other CSOs, and the findings based on Spearman’s rho in Table 5 show the same result.

The two-part gamma GLMM was also used to investigate whether there were any systematic differences in the CSOs’ and the gamblers’ reports. The results presented in Table 6 show that a larger proportion of the CSOs reported no losses when compared to the gamblers (14% vs. 6%).

Table 7 shows that the results were inconclusive with regard to the existence of any systematic differences in the average daily losses, with no indication of any large differences being found. Furthermore, within almost half of the dyads, the CSOs reported higher losses than their respective gamblers. Indeed, the proportion of CSOs with higher reports was .43, 95% CI (.34, .52).

Overall, there was excellent agreement in terms of the years spent gambling, ICC = .79, 95% CI (.71, .85), with equal point estimates being found for parents and partners, ICC = .76 versus .76, albeit with the 95% CI (− .16, .17) of the difference indicating that important differences could not be ruled out. There was fair agreement overall regarding the amount of gambling debt, ICC = .57, 95% CI (.46, .65). However, the parent CSOs showed higher agreement when compared to the partner CSOs, ICC_diff = − .39, 95% CI (− .62, − .17), with more inconclusive results being found when comparing to the CSOs who were neither partners nor parents, ICC_diff = − .30, 95% CI (− .60, .32). The results for both years gambling and gambling debt are shown in Table 8.

Sensitivity analyses were performed to check whether the results were impacted by (1) excluding those dyads with a perfect agreement, (2) including dyads with sign up dates differing by more than 7 days, and (3) including only those dyads with a maximum absolute difference in sign-up dates of 1 day. None of these model decisions had a major impact on the results. The sensitivity analyses are detailed in the online appendix.⁶

Figures 4 and 5 show that, although the relative bias for the ICCs obtained from the Gaussian model was seldom more than ± 20%, the estimates exhibited low precision, and the 95% CIs had coverage probabilities much smaller than .95, whereas the two-part gamma GLMM yielded both low relative bias and appropriate coverage probabilities. As anticipated, the binary part of the model showed some bias, although the CIs had good coverage rates. This only had a minor impact on the ICCs for the overall outcomes. Ignoring the true correlation between the random effects in the two-part model led to increased bias and worse coverage rates for the overall ICCs, although it is worth noting that the inferences regarding the ICCs for only the losses part of the two-part model worked well in all the scenarios.

The power to detect a difference between the ICCs of the partner CSOs and the parent CSOs was slightly higher for the two-part gamma GLMM when compared to the Gaussian LMM, as shown in Fig. 6.