Prediction models for clustered data with informative priors for the random effects: a simulation study

At the model development phase, data containing measures of the predictor(s) and outcome of interest are collected for the purpose of estimating the model parameters. In empirical applications, selection of relevant predictor(s) is often performed first. In this study, we assume that relevant predictors were selected already and directly focus on parameter estimation.

We consider a simple logistic regression model with one predictor measured on the subject level and random effects at the cluster level. Let x_ij be the predictor and y_ij be the observed binary outcome of subject i (i = 1, …, n_j) from cluster j (j = 1, …, J), and p_ij = p(y_ij = 1) be the latent underlying risk for the observed binary outcome. A random effects logistic regression model can be expressed as:

where the logit (i.e., log-odds) of the latent underlying risk of the outcome logit(p_ij) is equivalent to the linear predictor LP_ij = β₀ + β₁x_ij + u_j. The model can alternatively be written in the form of:

The linear predictor consists of the regression parameter β₁ for the predictor, the average (fixed) intercept β₀ and the cluster specific random effect u_j. The random effects are assumed to have a normal distribution with mean zero and variance \( {\sigma}_u^2 \).

Within the frequentist approach, parameters of a logistic regression prediction model are estimated via maximum likelihood (ML). In our study, functions from the R package ‘lme4’ were used [9].

Within the Bayesian framework, parameters are expressed in the form of distributions rather than fixed values. Before observing the data, prior distributions that contain the plausible values for the model parameters need to be specified [7]. The prior distributions are subsequently updated with observed data, resulting in posterior distributions. The posterior can be derived either analytically or by sampling methods. In this study, Markov chain Monte Carlo (MCMC) sampling was used.

Prior distributions needed to be specified for parameters β₀, β₁ and \( {\sigma}_u^2 \) in model (2.1). Priors for the regression parameters β₀ and β₁ were assumed to be normally distributed, and prior for the variance \( {\sigma}_u^2 \) had an inverse gamma distribution. When there is no a priori evidence available, one often uses weakly informative priors. A common choice for a normal distribution is to fix the mean at zero and take a large variance (here we used 1000 for the variance). For an inverse gamma distribution, small values are often assigned to the hyperparameters (here we used 0.001 for both hyperparameters).

Estimates for the parameters of interest are provided by the MCMC samples from the posterior distribution after convergence. In this study, we used OpenBUGS (via R package ‘BRugs’ [10]) to carry out the Bayesian analyses.

Let x_sc be the predictor, y_sc be the observed binary outcome and p_sc be the latent underlying risk of the observed outcome for subject s (s = 1, …, n_c) from new cluster c (c = 1, …, C). The prediction model developed and estimated from model development data is applied to calculate the risk of outcome for each subject in new clusters. The predicted risk of outcome \( {\widehat{p}}_{\mathrm{sc}} \) is compared to the true latent underlying risk p_sc and the observed outcome y_sc for the evaluation of the predictive performance.

In the frequentist approach, prediction for new clusters is usually based on a model where point estimates for the regression parameters are incorporated and the random effect term is substituted with mean 0 (i.e., removed). This leads to the predicted linear predictor.

where \( {\widehat{\beta}}_0^{ML} \) and \( {\widehat{\beta}}_1^{ML} \) are the estimated regression coefficients using maximum likelihood estimation and x_sc contains values of the predictor from subjects in new clusters. Accordingly, the predicted risk for the binary outcome in new clusters can be written as

In the Bayesian approach, by MCMC sampling, we obtain the posterior distribution for the parameters β₀, β₁ and \( {\sigma}_u^2 \). In this study, instead of using summarized point estimates, the full posterior for the parameters is exploited for prediction. To explain the Bayesian prediction, consider Table 1 in which a small part of the MCMC output is listed.

The sampled values for parameters from iteration k, denoted by \( {\overset{\sim }{\beta}}_0^{(k)} \),\( {\overset{\sim }{\beta}}_1^{(k)} \) and \( {{\overset{\sim }{\sigma}}_u^2}^{(k)} \), are presented in the left hand part of the table. Predictions made for subjects in a new cluster can be found in the right hand part of the table. As the model development clusters and the new clusters are assumed to be exchangeable, i.e., originated from the same metapopulation, random effects for all clusters are assumed to have the same normal distribution \( N\left(0,{\sigma}_u^2\right) \). For new cluster c, in each iteration, the predicted random effect \( {\widehat{u}}_c^{(k)} \) can be drawn from distribution\( N\left(0,{{\overset{\sim }{\sigma}}_u^2}^{(k)}\right) \). For each subject s (s = 1, …., n_c) from cluster c, the estimated linear predictor is henceand the predicted risk can be obtained by.

Eventually, a predicted risk distribution based on all K iterations is available for each subject. In this paper, in order to compare the results between the Bayesian models and the frequentist model, we used the median of the predicted risk distribution as the summarized predicted risk \( {\widehat{p}}_{\mathrm{sc}} \), resulting in a single estimate per subject.

By applying the Bayesian approach, we can maintain the random effect term in the prediction model, which takes uncertainty due to variance between clusters into account. To improve the predictive ability of the Bayesian random effects model, one may include cluster specific expert knowledge as prior for the random effects on new clusters. Demonstration of how this prior knowledge was incorporated into a Bayesian prediction model can be found in the following section.

In the previous section, for each new cluster c in each iteration k, the predicted random effect \( {\widehat{u}}_c^{(k)} \) was sampled from the entire random effects distribution \( N\left(0,{{\overset{\sim }{\sigma}}_u^2}^{(k)}\right) \). This would add more uncertainty to the prediction compared to the frequentist model that substituted the random effects with the mean 0. However, if there is information available about the position of a new cluster relative to other clusters, we may sample a value for \( {\widehat{u}}_c^{(k)} \) from part of the distribution rather than the whole distribution.

Consider again the example of hospitals where some hospitals are known to be better at treating a particular disease than others. When we have no clue about the relative risk of death for a disease from a particular hospital regarding other hospitals, we sample a random effect for the hospital from the entire random effects distribution. However, if an expert is capable of using hospital level information to judge whether a hospital would provide below or above average chance of survival, we could sample the random effect from only the lower or upper half of the distribution [11]. Suppose the expert says the hospital will provide a below average probability of survival, the random effect will accordingly be sampled from the lower half of the distribution (see the first plot in Fig. 1). Further, if the expert is more precise about the relative position of the hospital with regard to other hospitals, the random effect can also be drawn from a smaller area, such as one third or one fifth of the distribution (see the second and third plots in Fig. 1).

It is expected that more precise prior knowledge leads to better predictions, under the assumption that an expert provides information that matches the true value. It is however likely that experts sometimes provide suboptimal judgments that deviate from the true quantity. Incorporation of discrepant expert opinion into the prediction model was therefore investigated as well. All investigations were done through simulations. Details of the simulation studies are presented in the next section.

One set of data containing 5000 subjects (n = 5000) was generated for model development. The number of clusters was set to 50, and the number of subjects per cluster was 100. Each cluster consisted of equal numbers of subjects. For each subject, one continuous predictor was sampled from the normal distribution N(0, 1) and allocated to the subjects in all further analyses. The true value for the regression parameter β₁ was set to 1.5. The random effects for clusters were sampled from a normal distribution with cluster variance 0.822. This value corresponds to an intraclass correlation coefficient (ICC) of 0.20, calculated from the formula \( {\sigma}_u^2/\left({\sigma}_u^2+{\pi}^2/3\right) \) where the error variance has a fixed value π²/3 in a logistic regression model [3]. The latent underlying risk of the outcome for each subject was calculated by computing the linear predictor. The observed outcome was subsequently sampled from a Bernoulli distribution using its underlying risk. By adjusting the value of the fixed intercept β₀, we set the overall prevalence of the dataset approximately to 50%. Finally, one new dataset was simulated by the exact same setting and used to evaluate the different prediction models. The simulation study thus contained one dataset for model development and one dataset for prediction. The R code for data simulation is provided in Additional file 2: Appendix B. It is worth noting that each time if we replicate a simulation study using the same settings with a different seed, we get different samples for the model development and prediction datasets, as there is randomness involved in the sampling process. Comparisons of the relative performance between models were hence carried out only within the same sets of development and prediction data.

Five models were estimated using model development data and applied to predict in new data. In the frequentist model (denoted FREQ), ML estimates of the regression parameters \( {\widehat{\beta}}_0^{ML} \) and \( {\widehat{\beta}}_1^{ML} \) were incorporated, and the random effect term was replaced with 0. Within the Bayesian approach, a prediction model using weakly informative priors for the random effects was first specified and denoted BAYES.WI. Three more prediction models were subsequently constructed where cluster specific expert judgments were incorporated as prior information for the random effects. Optimal expert judgment was defined as choosing the truncated area from the random effects distribution which contained the true value of the random effect. Three scales were specified for the random effects distribution on the basis of varying degrees of precision of the expert opinion. In each scale, the distribution was divided into equal sized truncated areas (i.e., equal proportions). The model that used low informative priors which contained half of the distribution was denoted BAYES.LI. Similarly, the model that used medium informative priors which contained one third of the distribution was denoted BAYES.MI, and the model that used highly informative priors which contained one fifth of the distribution was denoted BAYES.HI. For each Bayesian prediction model, two posterior chains were sampled and thinned by the interval 10 (i.e., taking every 10th observation). Within each chain 100 samples were saved after convergence was reached, leading to 200 posterior samples in total for prediction in each subject from the new clusters.

Impact of including expert opinion that deviated from the true random effect value as prior information was explored in models BAYES.LI, BAYES.MI and BAYES.HI. We defined discrepant expert judgment as selecting a truncated area that was next to the ‘correct’ area that contained the true value of the random effect. In the BAYES.LI condition, it was straightforward, since the random effects distribution was divided into two equal sized areas. When the true value of the random effect was located in one area, the other area would be chosen. However, in the BAYES.MI and BAYES.HI conditions where the random effects distribution was divided into more than two equal sized areas, choosing a truncated area that was next to the ‘correct’ area indicated that the discrepant expert opinion was still relatively close to the true value (see Fig. 1). Suppose the true value was at the middle 1/5 truncated area of the distribution, the discrepant expert opinion would be selecting the second 1/5 or the fourth 1/5 truncated area from the left hand side. In other words, discrepant expert opinion with more precision (i.e., 1/3, 1/5) was less off from the true value in comparison to the discrepant expert opinion with least precision (1/2). The percentage of new clusters that incorporated discrepant expert opinion was set to 10, 30% or 50%.

In principle, one could also include prior knowledge in the frequentist model. We performed simulations where optimal expert opinion was incorporated as a fixed effect in the frequentist model in the default scenario. First, in the phase of model development, the true random effects were used to create the additional predictor representing the prior knowledge. For instance, when the true random effect for a specific cluster was among the upper half of the true random effects distribution, expert opinion for this cluster was then coded into 1 when the lower half was the reference category (coded 0). Likewise, expert opinion was coded using a categorical variable with 3 or 5 levels by placing the true random effects for the model development clusters in the correct tertiles and quintiles, with the second tertile and the third quintile as the reference category respectively. The frequentist models with inclusion of expert opinion coded into 2, 3, and 5 categories were denoted FREQ.2, FREQ.3 and FREQ.5 respectively.

The resulting prediction models with expert opinion were used to predict the outcomes in the new clusters. Again, the expert knowledge (i.e., the scores on the categorical variable) for each new cluster was obtained by placing the true random effect for the new cluster in the correct quantiles of the true random effects for the model development clusters.

The predictive ability of the models was assessed by Brier scores, model discrimination and calibration. Brier score was computed as the mean of the squared difference between the observed binary outcomes and the predicted risks. Discriminative ability was assessed by the concordance index (C-index) which equals the area under the ROC curve. Model calibration was evaluated using the calibration slope. The ideal value for the calibration slope is 1, which represents perfect prediction. The further the calibration slope deviates from 1, the worse the model is calibrated [1]. Since in the simulation research, true latent underlying risks of outcome are available, the calibration was computed as the agreement between predicted risks and true risks. The calibration slope was the linear regression coefficient for the predicted risk as the independent variable, and the true risk as the dependent variable. It is worth noting that in empirical data, true risks are not available. Model calibration can hence be assessed by using the observed binary outcome as the dependent variable and the estimated linear predictor as the independent variable in a logistic regression analysis [1, 12]. Brier scores were computed at the subject level (overall) for all models. Model discrimination and calibration were measured both at subject level (overall) and cluster level (within cluster). The within cluster measures were summarized in mean and standard deviation over all clusters. Calibration of the models was further visualized in calibration plots where the predicted risks of outcome were plotted against the true risks.

In order to check the influence of prevalence, ICC, sample size and strength of the predictor on prediction, eight sensitivity analyses were carried out. Each sensitivity analysis consisted of one new simulation study that had default simulation settings except for the specific feature that was investigated. Impact of the between cluster variance was evaluated by comparing the default ICC value 0.20 to 0.05 and 0.50. Impact of the prevalence was evaluated by comparing the default prevalence value 50 to 10% and 25%. To examine the effect of smaller sample sizes on prediction, we reduced in one sensitivity analysis the number of clusters, resulting in n = 2000 subjects in total ( J = S = 20, n_j = n_c = 100), and in another sensitivity analysis the number of subjects per cluster, resulting in n = 1000 subjects in total ( J = S = 50, n_j = n_c = 20). Finally, by changing the value for the model parameter β₁ from 1.5 (default) to 0.5 and 3.0, we explored the influence of a weaker or a stronger subject level predictor.