The online version of this article (doi:10.1186/1471-2288-14-5) contains supplementary material, which is available to authorized users.
The authors declare that they have no competing interests.
DK, ES and YV designed the study. PP participated in the collection of data and organisation of the databases from which this manuscript was developed. DK and YV analysed the data and wrote the first draft of the manuscript. All authors contributed to writing the manuscript and read and approved the final manuscript.
The discriminative ability of a risk model is often measured by Harrell’s concordance-index (c-index). The c-index estimates for two randomly chosen subjects the probability that the model predicts a higher risk for the subject with poorer outcome (concordance probability). When data are clustered, as in multicenter data, two types of concordance are distinguished: concordance in subjects from the same cluster (within-cluster concordance probability) and concordance in subjects from different clusters (between-cluster concordance probability). We argue that the within-cluster concordance probability is most relevant when a risk model supports decisions within clusters (e.g. who should be treated in a particular center). We aimed to explore different approaches to estimate the within-cluster concordance probability in clustered data.
We used data of the CRASH trial (2,081 patients clustered in 35 centers) to develop a risk model for mortality after traumatic brain injury. To assess the discriminative ability of the risk model within centers we first calculated cluster-specific c-indexes. We then pooled the cluster-specific c-indexes into a summary estimate with different meta-analytical techniques. We considered fixed effect meta-analysis with different weights (equal; inverse variance; number of subjects, events or pairs) and random effects meta-analysis. We reflected on pooling the estimates on the log-odds scale rather than the probability scale.
The cluster-specific c-index varied substantially across centers (IQR = 0.70-0.81; I 2 = 0.76 with 95% confidence interval 0.66 to 0.82). Summary estimates resulting from fixed effect meta-analysis ranged from 0.75 (equal weights) to 0.84 (inverse variance weights). With random effects meta-analysis – accounting for the observed heterogeneity in c-indexes across clusters – we estimated a mean of 0.77, a between-cluster variance of 0.0072 and a 95% prediction interval of 0.60 to 0.95. The normality assumptions for derivation of a prediction interval were better met on the probability than on the log-odds scale.
When assessing the discriminative ability of risk models used to support decisions at cluster level we recommend meta-analysis of cluster-specific c-indexes. Particularly, random effects meta-analysis should be considered.
Gelman A, Hill J: Data analysis using regression and multilevel/hierarchical models. 2007, Cambridge: Cambridge University Press
Duchateau L, Janssen P: The Frailty Model. 2008, New York: Springer
Van Oirbeek R, Lesaffre E: Assessing the predictive ability of a multilevel binary regression model. Comput Stat Data Anal. 2012, 56 (6): 1966-1980. 10.1016/j.csda.2011.11.023. CrossRef
Collaborators MCT, Perel P, Arango M, Clayton T, Edwards P, Komolafe E, Poccock S, Roberts I, Shakur H, Steyerberg E, et al: Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ. 2008, 336 (7641): 425-429. CrossRef
Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JD, et al: Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 2008, 5 (8): e165-10.1371/journal.pmed.0050165. CrossRefPubMedPubMedCentral
Edwards P, Arango M, Balica L, Cottingham R, El-Sayed H, Farrell B, Fernandes J, Gogichaisvili T, Golden N, Hartzenberg B, et al: Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. Lancet. 2005, 365 (9475): 1957-1959. CrossRefPubMed
R Development Core Team: R: A Language and Environment for Statistical Computing. 2011, Vienna, Austria: R Foundation for Statistical Computing, ISBN 3-900051-07-0, URL http://www.R-project.org/
Therneau T, original Splus->R port by Lumley T: survival: Survival analysis, including penalised likelihood. R package version 2.36-9. 2011, http://CRAN.R-project.org/package=survival,
Quade D: Nonparametric partial correlation. Volume No. 526. 1967, North Carolina: Institute of Statistics Mimeo, Volume 526
Higgins JPT, Thompson SG, Spiegelhalter DJ: A re-evaluation of random-effects meta-analysis. J R Soc Health Series A. 2009, 172 (1): 137-159. CrossRef
Lumley T: rmeta: Meta-analysis. R package version 2.16. 2009, http://CRAN.R-project.org/package=rmeta,
Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009, New York: Springer CrossRef
Gönen M, Heller G: Concordance probability and discriminatory power in proportional hazards regression. Biometrika. 2005, 92 (4): 965-970. 10.1093/biomet/92.4.965. CrossRef
Nam BH, D'Agostino RB: Discrimination Index, the Area under the ROC Curve. Goodness-of-Fit Tests and Model Validity. 2002, Boston: Birkhauser, 267-279. CrossRef
Efron B, Tibshirani R: An Introduction to the Bootstrap. 1993, Boca Raton, FL: CRC press CrossRef
- Assessing discriminative ability of risk models in clustered data
David van Klaveren
Ewout W Steyerberg
- BioMed Central
Neu im Fachgebiet AINS
Meistgelesene Bücher aus dem Fachgebiet AINS
Mail Icon II