Abstract
Reliable estimation methods for non-linear mixed-effects models are now available and, although these models are increasingly used, only a limited number of statistical developments for their evaluation have been reported. We develop a criterion and a test to evaluate nonlinear mixed-effects models based on the whole predictive distribution. For each observation, we define the prediction discrepancy (pd) as the percentile of the observation in the whole marginal predictive distribution under H0. We propose to compute prediction discrepancies using Monte Carlo integration which does not require model approximation. If the model is valid, these pd should be uniformly distributed over [0, 1] which can be tested by a Kolmogorov–Smirnov test. In a simulation study based on a standard population pharmacokinetic model, we compare and show the interest of this criterion with respect to the one most frequently used to evaluate nonlinear mixed-effects models: standardized prediction errors (spe) which are evaluated using a first order approximation of the model. Trends in pd can also be evaluated via several plots to check for specific departures from the model
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
L.B. Sheiner and J.L. Steimer, Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol. 40 (2000) 67-95
Aarons L., Karlsson M.O., Mentré F., Rombout F., Steimer J.L., van Peer A., and Cost B15 experts. Role of modelling and simulation in phase I drug development. Eur. J. Pharm. Sci. 13:115–122 (2001).
N.H. Holford, H.C. Kimko, J.P. Monteleone and C.C. Peck, Simulation of clinical trials. Annu. Rev. Pharmacol. Toxicol. 40 (2000) 209-234
L.J. Lesko, M. Rowland, C.C. Peck and T.F. Blaschke, Optimizing the science of drug development: opportunities for better candidate selection and accelerated evaluations in humans. J. Clin. Pharmacol. 40 (2000) 803-814
H.C. Kimko and S.B. Duffull, Simulation for Designing Clinical Trials: A Pharmacokinetic—Pharmacodynamic Modeling Prospective. New York: Marcel Dekker (2003).
Y. Yano, S.L. Beal and L.B. Sheiner, Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28 (2001) 171-192
M. Davidian and D.M. Giltinan, Nonlinear Models for Repeated Measurement Data. London: Chapman and Hall (1995).
E.F. Vonesh and V.M. Chinchilli, Linear and Nonlinear Models for the Analysis of Repeated Measurements. New York: Marcel Dekker (1997).
J.C. Pinheiro and D.M. Bates, Mixed-Effect Models in S and Splus. New York: Springer Verlag (2000).
L. Aarons, Software for population pharmacokinetics and pharmacodynamics. Clin. Pharmacokinet. 36 (1999) 255-264
A.J. Boeckmann, L.B. Sheiner and S.L. Beal, NONMEM users guide. San Francisco: NONMEM Users group at University of California (1994).
F. Mentré and R. Gomeni, A two-step iterative algorithm for estimation in nonlinear mixed-effect models with an evaluation in population pharmacokinetics. J. Biopharm. Stat. 5 (1995) 141-158
R.D. Wolfinger and X. Lin, Two Taylor-series approximation methods for estimation in nonlinear mixed-effects models with an evaluation in population pharmacokinetics. Comput. Stat. Data Anal. 25 (1997) 465-490
A.E. Gelfan, Model determination using sampling-based methods. In: Markov Chain Monte Carlo in Practice.. Boca Raton: Chapman & Hall (1996) pp. 145-161
F. Mentré and M. E. Ebelin. Validation of population pharmacokinetic/pharmacodynamic analyses: review of proposed approaches. In The Population Approach: Measuring and Managing Variability in Response Concentration and Dose. Office for official publications of the European Communities, Brussels, 1997, pp. 141–158.
Food and Drug Administration. Guidance for Industry: Population Pharmacokinetics (available at http://www.fda.gov/cder/guidance/index.htm 1999).
A. Gelman, J.B. Carlin, H.S. Stern and D.B. Rubin, Bayesian Data Analysis. London: Chapman and Hall (1995).
P.J. Williams and I.E. Ette, Determination of model appropriateness. In: In Simulation for Designing Clinical Trials: A Pharmacokinetic–Pharmacodynamic Modeling Prospective.. New York: Marcel Dekker (2003) pp. 73-104
M. Evans, Comments of asymptotic distribution of P values in composite null models by J. M. Robins, A. van der Vaart and V. Ventura.. J. Am. Stat. Assoc. 95 (2000) 1160-1163
M.J. Bayarri and J.O. Berger, P values for composite null models. J. Am. Stat. Assoc. 95 (2000) 1127-1142
J.M. Robins, A. Vaart van der and V. Ventura, Asymptotic distribution of P values in composite null models (with discussion). J. Am. Stat. Assoc. 95 (2000) 1143-1172
N. Lange and L. Ryan, Assessing normality in random effects models. Ann. Statist. 17 (1989) 624-642
A.E. Gelfand, D.K. Det and H. Chang, Model determination using predictive distributions with implementation via sampling-based methods. In: In Bayesian Statistics 4.. Oxford: University Press (1992) pp. 147-167
N.G. Best, K.K.C. Tan, W.R. Gilks and D.J. Spiegelhalter, Estimation of Population Pharmacokinetics using the Gibbs sampler. J. Pharmacokinet. Biopharm. 23 (1995) 407-435
F. Mesnil, F. Mentré, C. Dubruc, J.P. Thénot and A. Mallet, Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J. Pharmacokinet. Biopharm. 26 (1998) 133-161
A. Mallet, A maximum likelihood estimation method for random coefficient regression models. Biometrika 73 (1996) 645-646
E. Comets, K. Ikeda, P. Hoff, P. Fumoleau, J. Wanders and Y. Tanigawara, Comparison of the pharmacokinetics of S-1, an oral anticancer agent, in Western and Japanese patients. J. Pharmacokinet. Pharmacodyn. 30 (2003) 257-283
A. Gelman and X.L. Meng, Model checking and model improvement. In: In Markov Chain Monte Carlo in Practice.. Boca Raton: Chapman & Hall (1996) pp. 189-201
R.B. D’Agostino and M.A. Stephens, Goodness-of-Fit Techniques. New York: Marcel Dekker (1986).
P. McCullagh and J.A. Nelder, Generalized Linear Models. London: Chapman & Hall (1989).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mentré, F., Escolano, S. Prediction Discrepancies for the Evaluation of Nonlinear Mixed-Effects Models. J Pharmacokinet Pharmacodyn 33, 345–367 (2006). https://doi.org/10.1007/s10928-005-0016-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10928-005-0016-4