The online version of this article (doi:10.1186/1471-2288-13-19) contains supplementary material, which is available to authorized users.
All authors (W Bouwmeester, JWR Twisk, TH Kappen, WA van Klei, KGM Moons, Y Vergouwe) state that they have no conflict of interests.
Study design and analysis: WB, YV. Drafting manuscript: WB. Study design and reviewing the manuscript: JWRT, KGMM, THK, WAvK, YV. All authors read and approved the final manuscript.
When study data are clustered, standard regression analysis is considered inappropriate and analytical techniques for clustered data need to be used. For prediction research in which the interest of predictor effects is on the patient level, random effect regression models are probably preferred over standard regression analysis. It is well known that the random effect parameter estimates and the standard logistic regression parameter estimates are different. Here, we compared random effect and standard logistic regression models for their ability to provide accurate predictions.
Using an empirical study on 1642 surgical patients at risk of postoperative nausea and vomiting, who were treated by one of 19 anesthesiologists (clusters), we developed prognostic models either with standard or random intercept logistic regression. External validity of these models was assessed in new patients from other anesthesiologists. We supported our results with simulation studies using intra-class correlation coefficients (ICC) of 5%, 15%, or 30%. Standard performance measures and measures adapted for the clustered data structure were estimated.
The model developed with random effect analysis showed better discrimination than the standard approach, if the cluster effects were used for risk prediction (standard c-index of 0.69 versus 0.66). In the external validation set, both models showed similar discrimination (standard c-index 0.68 versus 0.67). The simulation study confirmed these results. For datasets with a high ICC (≥15%), model calibration was only adequate in external subjects, if the used performance measure assumed the same data structure as the model development method: standard calibration measures showed good calibration for the standard developed model, calibration measures adapting the clustered data structure showed good calibration for the prediction model with random intercept.
The models with random intercept discriminate better than the standard model only if the cluster effect is used for predictions. The prediction model with random intercept had good calibration within clusters.
Additional file 1: Table S1: Simulation results in a domain with ICC = 5%, Pearson correlation X1 and random effect 0.4. Apparent performance. Table S2 Simulation results in a domain with ICC = 15%, Pearson correlation X1 and random effect 0.4. Table S3 Simulation results in a domain with ICC = 30%, Pearson correlation X1 and random effect 0.0. Table S4 Simulation results in a domain with ICC = 30%, Pearson correlation X1 and random effect 0.4. Table S5 Simulation results in a domain with ICC = 5%, Pearson correlation X1 and random effect 0.0, outcome incidence 3% in 1000 patients. (DOC 83 kb) (DOC 84 KB)12874_2011_939_MOESM1_ESM.doc
Authors’ original file for figure 112874_2011_939_MOESM2_ESM.pdf
Authors’ original file for figure 212874_2011_939_MOESM3_ESM.pdf
Authors’ original file for figure 312874_2011_939_MOESM4_ESM.pdf
Steyerberg EW: Clinical prediction models; a practical approach to development, validation, and updating. 2009, New York: Springer
Bouwmeester W: Reporting and methods in clinical prediction research: a systematic review. Prediction models: systematic reviews and clustered study data. 2012, Utrecht: Igitur archive
Goldstein H: Multilevel statistical models. 1995, London: Edward Arnold
Twisk JWR: Applied multilevel analysis. 2006, New York: Cambridge University Press CrossRef
Guo G, Zhao H: Multilevel modeling for binary data. Annu Rev Sociol. 2000, 26: 441-462. 10.1146/annurev.soc.26.1.441. CrossRef
Liu I, Agresti A: The analysis of ordered categorical data: an overview and a survey of recent developments. Sociedad de Estadistica e Investigacion Operative Test. 2005, 14: 1-73.
Skrondal A, Rabe-Hesketh S: Prediction in multilevel generalized linear models. J R Stat Soc A Stat Soc. 2009, 172: 659-687. 10.1111/j.1467-985X.2009.00587.x. CrossRef
Van OR, Lesaffre E: An application of Harrell's C-index to PH frailty models. Stat Med. 2010, 29: 3160-3171. 10.1002/sim.4058. CrossRef
Scott AJ, Holt D: The effect of Two-stage sampling on ordinary least squares methods. J Am Stat Assoc. 1982, 77: 848-854. 10.1080/01621459.1982.10477897. CrossRef
R Development Core Team: R: a language and environment for statistical computing. 2008, Vienna, Austria: R Foundation for Statistical Computing, Ref Type: Computer
Bates D, Maechler M: lme4: Linear mixed-effects models using S4 classes. 2009, 14-12-2009. R CRAN Project. Accessible via http://lme4.r-forge.r-project.org/. Ref Type: Computer Program
Harrell F: Design. 2009, 18-9-2009. R CRAN Project. Accessible via http://biostat.mc.vanderbilt.edu/s/Design. Ref Type: Computer Program
Hedeker D, Gibbons R, Davis J: Random regression models for multicenter clinical trial data. Psychopharmacol Bull. 1991, 27: 73-77. PubMed
Turrell G, Sanders AE, Slade GD, Spencer AJ, Marcenes W: The independent contribution of neighborhood disadvantage and individual-level socioeconomic position to self-reported oral health: a multilevel analysis. Community Dent Oral Epidemiol. 2007, 35: 195-206. 10.1111/j.1600-0528.2006.00311.x. CrossRefPubMed
- Prediction models for clustered data: comparison of a random intercept and standard regression model
Jos WR Twisk
Teus H Kappen
Wilton A van Klei
Karel GM Moons
- BioMed Central
Neu im Fachgebiet AINS
Meistgelesene Bücher aus dem Fachgebiet AINS
Mail Icon II