Skip to main content

Advertisement

Log in

Two bootstrapping routines for obtaining imprecision estimates for nonparametric parameter distributions in nonlinear mixed effects models

  • Published:
Journal of Pharmacokinetics and Pharmacodynamics Aims and scope Submit manuscript

Abstract

When parameter estimates are used in predictions or decisions, it is important to consider the magnitude of imprecision associated with the estimation. Such imprecision estimates are, however, presently lacking for nonparametric algorithms intended for nonlinear mixed effects models. The objective of this study was to develop resampling-based methods for estimating imprecision in nonparametric distribution (NPD) estimates obtained in NONMEM. A one-compartment PK model was used to simulate datasets for which the random effect of clearance conformed to a (i) normal (ii) bimodal and (iii) heavy-tailed underlying distributional shapes. Re-estimation was conducted assuming normality under FOCE, and NPDs were estimated sequential to this step. Imprecision in the NPD was then estimated by means of two different resampling procedures. The first (full) method relies on bootstrap sampling from the raw data and a re-estimation of both the preceding parametric (FOCE) and the nonparametric step. The second (simplified) method relies on bootstrap sampling of individual nonparametric probability distributions. Nonparametric 95% confidence intervals (95% CIs) were obtained and mean errors (MEs) of the 95% CI width were computed. Standard errors (SEs) of nonparametric population estimates were obtained using the simplified method and evaluated through 100 stochastic simulations followed by estimations (SSEs). Both methods were successfully implemented to provide imprecision estimates for NPDs. The imprecision estimates adequately reflected the reference imprecision in all distributional cases and regardless of the numbers of individuals in the original data. Relative MEs of the 95% CI width of CL marginal density when original data contained 200 individuals were equal to: (i) −22 and −12%, (ii) −22 and −9%, (iii) −13 and −5% for the full and simplified (n = 100), respectively. SEs derived from the simplified method were consistent with the ones obtained from 100 SSEs. In conclusion, two novel bootstrapping methods intended for nonparametric estimation methods are proposed. In addition of providing information about the precision of nonparametric parameter estimates, they can serve as diagnostic tools for the detection of misspecified parameter distributions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Reference

  1. Bustad A, Terziivanov D, Leary R et al (2006) Parametric and nonparametric population methods: their comparative performance in analysing a clinical dataset and two Monte Carlo simulation studies. Clin Pharmacokinet 45(4):365–383

    Article  PubMed  Google Scholar 

  2. Mallet A, Mentre F, Steimer JL et al (1988) Nonparametric maximum likelihood estimation for population pharmacokinetics, with application to cyclosporine. J Pharmacokinet Biopharm 16(3):311–327

    Article  CAS  PubMed  Google Scholar 

  3. Jelliffe R, Schumitzky A, Van Guilder M (2000) Population pharmacokinetics/pharmacodynamics modeling: parametric and nonparametric methods. Ther Drug Monit 22(3):354–365

    Article  CAS  PubMed  Google Scholar 

  4. Mallet A (1986) A maximum likelihood estimation method for random coefficient regression models. Biometrika 3:645–656

    Article  Google Scholar 

  5. Savic RM, Kjellsson MC, Karlsson MO (2009) Evaluation of the nonparametric estimation method in NONMEM VI. Eur J Pharm Sci 37(1):27–35

    Article  CAS  PubMed  Google Scholar 

  6. Baverel PG, Savic RM, Wilkins JJ et al (2009) Evaluation of the nonparametric estimation method in NONMEM VI: application to real data. J Pharmacokinet Pharmacodyn 36(4):297–315

    Article  PubMed  Google Scholar 

  7. Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 7:1–26

    Article  Google Scholar 

  8. Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78:461–470

    Article  Google Scholar 

  9. Efron B, Tibshirani R (1993) An introduction to the bootstrap. Chapman & Hall, New York

    Google Scholar 

  10. Rubin D (1981) The Bayesian bootstrap. Ann Stat 9:130–134

    Article  Google Scholar 

  11. White DB, Walawander CA, Tung Y et al (1991) An evaluation of point and interval estimates in population pharmacokinetics using NONMEM analysis. J Pharmacokinet Biopharm 19(1):87–112

    CAS  PubMed  Google Scholar 

  12. Savic RM, Baverel PG, Karlsson MO (2008) A novel bootstrap method for obtaining uncertainty measurement around the nonparametric distribution. www.page-meeting.org/?abstract=1390

  13. Lindbom L, Ribbing J, Jonsson EN (2004) Perl-speaks-NONMEM (PsN)–a Perl module for NONMEM related programming. Comput Methods Programs Biomed 75(2):85–94

    Article  PubMed  Google Scholar 

  14. Beal S, Sheiner LB, Boeckmann A (1989–2006) NONMEM user’s guides. Icon Development Solutions, Ellicott City, Maryland, USA

  15. Sheiner LB, Beal SL (1981) Some suggestions for measuring predictive performance. J Pharmacokinet Biopharm 9(4):503–512

    Article  CAS  PubMed  Google Scholar 

  16. Karlsson MO, Savic RM (2007) Diagnosing model diagnostics. Clin Pharmacol Ther 82(1):17–20

    Article  CAS  PubMed  Google Scholar 

  17. Leary RH, Chittenden J (2008) A nonparametric analogue to POSTHOC estimates for exploratory data analysis. www.page-meeting.org/?abstract=1343

  18. Savic RM, Karlsson MO (2007) Importance of shrinkage in Empirical Bayes estimates for diagnostics and estimation: problems and solutions. www.page-meeting.org/?abstract=1087

  19. Savic RM, Karlsson MO (2009) Evaluation of an extended grid method for estimation using nonparametric distributions. AAPS J 11(3):615–627

    Article  CAS  PubMed  Google Scholar 

  20. Savic RM, Karlsson MO (2009) Importance of shrinkage in empirical bayes estimates for diagnostics: problems and solutions. AAPS J 11(3):558–569

    Article  CAS  PubMed  Google Scholar 

  21. Antic J, Laffont CM, Chafaï D et al (2009) Comparison of nonparametric methods in nonlinear mixed effects models. Comput Stat Data Anal 53:642–656

    Article  Google Scholar 

  22. Pearson K (1920) Notes on the history of correlation. Biometrika 13(1):25–45

    Article  Google Scholar 

  23. Lindbom L (2006) Development, application and evaluation of statistical tools in pharmacometric data analysis. Doctoral thesis of Lars Lindbom from the Division of Pharmacokinetics and Drug Therapy, Department of Pharmaceutical Sciences, Uppsala University, Sweden

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Paul G. Baverel.

Appendices

Appendix 1: Description of full and simplified nonparametric bootstrap methods suitable for nonparametric distribution and their execution within the PsN program

Full nonparametric bootstrapping method suitable for nonparametric distribution

It is composed of 5 consecutive steps, some of them sharing some aspects with the standard nonparametric bootstrap procedure used to assess nonparametric CIs of parametric estimators [7].

The following notations are introduced and will be used throughout the appendix:

  • D—original dataset

  • J—total number of individuals included in original dataset D

  • M—final nonparametric model for which parameter imprecision is to be estimated

  • n—number of bootstrap samples

  • B 1–n —bootstrapped datasets corresponding to the bootstrap samples from 1 to n

  • P1n—parametric population parameter vectors corresponding to the bootstrap datasets B1n

  • NPD—nonparametric distribution which is a collection of J support points and population joint density (JD)

  • IPD1J—individual probability density vectors associated with support point 1 to J

Fig. 3
figure 3

Graphical representation of the partition of the population joint density (JD) of a NPD into individual probability densities (IPDs) corresponding, respectively to step 3 and step 1 of the full and simplified nonparametric bootstrapping techniques described in Appendix 1. (a) depicts the JD associated with the CL support point estimates in a population of five individuals. (b) illustrates the partition of JD according to the contributions of each individual to the height of the spike for each support point. Finally, (c) shows the IPDs characterizing each individual distribution across the support points. It should be noted that an infinitesimally small individual probability was not possible to display in (b) and (c), IPDs being usually defined at J unique support points of the original NPD. Each color of (b) and (c) represents a unique individual of the population

Step 1: bootstrapped datasets and parametric estimation

Original dataset D containing J distinct individuals, from which J unique support points were defined by applying FOCE-NONP to the final model M, is repeatedly sampled with replacement n times to create n bootstrapped datasets B 1n. Re-estimation of the parameters of this model with each of the n bootstrapped datasets (B 1n) using FOCE gives rise to n vectors of parametric population parameter estimates P 1n.

Step 2: nonparametric estimation of original dataset D based on parametric distribution P1–n

The number of unique individuals assigned in each dataset B 1n is lower than J. The number of unique support points is given by the number of unique individuals in the dataset used to estimate support points location. In order to assure that the number of support points is adequate for each NPD across replicates, the procedure of defining support points needs to be based on the original dataset D, and not on the bootstrapped datasets B 1n. For that reason, each set of parametric population parameters P 1n obtained in step 1, are subsequently used to estimate J support points given D by applying the nonparametric step in NONMEM.

In this manner, n sets of NPD 1n are obtained, each of them being defined at J number of unique support points determined by individual data and the parameter vectors P 1n.

Step 3: partition of NPD into individual probability densities (IPDs)

Each JD vector of NPD 1n obtained in step 2 consists, as illustrated in Eq. 6, of J individual probability density vectors (IPD 1–J), each defined at the same support points. A graphical representation of this step is illustrated in Appendix Fig. 3 for J = 5.

$$ JD = \sum\limits_{i = 1}^{J} {IPD_{i} } $$
(6)

IPD1J is composed of a vector of individual probabilities associated with J support points. Entries of this vector, we denote as IPD ik , are computed from Eq. 7:

$$ IPD_{ik} = {\frac{{JD_{k} \times L_{ik} }}{{\sum\nolimits_{k = 1}^{J} {JD_{k} \times L_{ik} } }}} $$
(7)

where IPD ik designates the ith individual probability (1 ≤ i ≤ J) associated with support point k (1 ≤ k ≤ J), JD k represents the population joint density associated to support point k, and L ik is the ith individual likelihood associated with support point k. The individual likelihood can be obtained from NONMEM by requesting individual objective function values (iOFV) after fixing parameter estimates to the values defining support point k. L ik estimates are then derived from Eq. 8:

$$ L_{ik} \propto e^{{ - {\frac{{iOFV_{k} }}{2}}}} $$
(8)

The IPD ik elements of each individual vector IPD 1–J sum to 1/J and the sum of the IPD ik elements across all J subjects add to one. The sum of the IPD ik elements obtained for support point k equals the population JD associated with support point k (Eqs. 6 and 7).

After the transformation, each NPD obtained in step 2 is now represented as a vector of J support points, coupled to a square matrix T of dimension (J, J) with the ith column representing the IPD of individual i, and the kth row representing the contribution of each individual to the population joint density (JD) defined at support point k.

Step 4: Bootstrapping of individual probability densities (IPDs) according to bootstrap sample key of B1–n and subsequent re-assembling into NPDboot1–n

From the n sets of NPD 1n, n matrices T 1n are obtained. From matrix T m, (1 ≤ mn), IPD 1J (column elements) are sampled with replacement according to the same scheme as for bootstrapped dataset B m obtained in step 1. As a result, a new matrix, Tboot m , is obtained. Summing up the row arrangements of IPD ik in Tboot m for each support point produces a new JD, which, when associated to the vector of J support points gives rise to a new NPD, NPDboot m .

In this manner, the procedure results in n sets of NPDboot 1n based on information from individuals contained in dataset B 1n but defined at J number of unique support points.

Step 5: Construction of nonparametric confidence intervals around NPD

The n sets of NPDboot 1n are finally used to construct nonparametric CIs around the original NPD. Imprecision in any aspect of the NPD can be obtained through the n sets of NPDboot 1n but in this presentation we focus on the 95% CI of the cumulative marginal parameter distribution, where the 2.5th and the 97.5th percentiles of NPDboot 1n are taken respectively as the lower and the upper boundary of the nonparametric 95% CI around NPD.

Simplified nonparametric bootstrap method adapted for nonparametric distribution

The simplified nonparametric bootstrapping method intended for estimating imprecision in NPD is designed based on three consecutive steps. The approach used to quantify imprecision remains the same as for the full, i.e., derive a distribution around NPD by bootstrapping techniques, ensuring that the entire distribution of J unique support points of the original NPD is retained throughout the process.

Unlike the full methodology, the original dataset D is not bootstrapped. The permutation and recombination process is only performed on the individual probability density vectors obtained by prior partitioning of the original NPD. As no new parameter estimation is involved in this procedure, a substantial shortening in run time is consequently achieved.

Step 1: partition of NPD into individual probability density functions

The first step of the simplified nonparametric bootstrapping procedure resembles the third step of the full, except that this time, the partition of the population JD into J individual density vectors IPD 1J described in Eq. 6 and depicted in Appendix Fig. 3 is performed on the original NPD. Since the original dataset D is not formerly altered, each IPD 1J vector shares the same J support points as the original NPD. Elements of IPD 1J (IPD ik ) are derived from Eqs. 7 and 8.

As a result, from a single set of original NPD, a J × J matrix T is obtained, with row vectors representing individual contributions of belonging to a given support point, and column vectors corresponding to individual probability densities across the original J support points. The attributes of matrix T prevail, i.e., each column element sums up to 1/J, whereas the sum of each row arrangement of IPD ik matches up to the estimate of JD associated with the support point k.

Step 2: bootstrapping of individual probability densities (IPD1–J ) and subsequent re-assembling into NPDboot1–n

From single matrix T, individual vectors IPD 1J are sampled with replacement n times. It results in n sets of bootstrapped matrices Tboot 1n. Collection of the row entries of each Tboot 1n adds into n sets of new JD values for a given support point. By repeating this calculation for the entire distribution of J original support points, n sets of NPDboot 1n are obtained.

Step 3: construction of nonparametric confidence intervals around NPD

From NPDboot 1n, nonparametric 95% CIs around original NPD are read out by applying the percentile approach described in step 5 of the full version.

Execution of the full and simplified nonparametric bootstrapping procedures adapted for nonparametric distribution in the PsN program

The basic command line to execute the full resampling-based procedure in PsN, with a user-defined number of bootstrap samples set to n = 100, is the following:

$$ {\text{nonpb runX}}.{\text{mod}} \, {-}{\text{samples}} {=} 100\,{-} {\text{nonpb}}\_{\text{version}}{=}2 $$

To execute the simplified resampling-based procedure, the change to the command line is as follow:

$$ {\text{nonpb runX}}.{\text{mod}}\,{-}{\text{samples}}{=}100\,{-}{\text{nonpb}}\_{\text{version}}{=}1 $$

It is noteworthy that runX.mod represents a parametric model file (although a nonparametric model file will run but take relatively more CPU time) and that the corresponding output file runX.lst, and the original dataset are required in the same directory unless specified otherwise with the function –lst = …/dirX/runX.lst. It should also be noted that both versions are compatible with NONMEM 7 and that PsN provides not only the nonparametric 95% CI, but any relevant statistics corresponding to the distribution of NPD across replicates, including the mean, median, 90, 50, and 25% CIs.

Appendix 2: Statistical metrics of imprecision around population estimates of the NPD

This procedure follows the second step of the simplified method that resulted in n sets of NPDboot 1n.

Standard errors of first level random effects moments (expected value, and variance)

From NPDboot 1n, n sets of expected values E 1n (η) are calculated according to Eq. 1. Standard error (SE) of E(η) is then taken to be the standard deviation (SD) of E 1n (η) based on Eq. 9:

$$ SE\left( {E\left( \eta \right)} \right) = SD\left( {E_{1 - n} \left( \eta \right)} \right) = \sqrt {{\frac{{\sum\nolimits_{i = 1}^{J} {\left[ {E_{i} \left( \eta \right) - \mu \left( {E_{1 - n} \left( \eta \right)} \right)} \right]^{2} } }}{n - 1}}} $$
(9)

where μ(E 1n(η)) represents the arithmetic mean of the distribution across replicates of E 1n(η).

Similarly, from NPDboot 1n, SE of each nonparametric variance (Ω kk ) and covariance (Ω kl ) component of the matrix V depicted in Eq. 2 is computed.

Variance–covariance matrix of the distribution of first level random variables

Diagonal elements of the symmetrical variance–covariance matrix (U) depicted in Eq. 10 are calculated as the variance (w 2) of E 1n(η), \( \Upomega_{{kk_{1 - n} }} \), and \( \Upomega_{{kl_{1 - n} }} \). For off-diagonal elements of U, correlation coefficients cor(X 1n ,Y 1n) are estimated based on the Pearson method [22], X 1n and Y 1n denoting either E 1n(η), \( \Upomega_{{kk_{1 - n} }} \), or \( \Upomega_{{kl_{1 - n} }} \) distribution of replicates (formula not shown). From cor(X 1n ,Y 1n), w 2(X 1n), and w 2(Y 1n), covariance elements of matrix U are collected according to Eq. 10:

$$ U = \left[ {\begin{array}{*{20}c} {w^{2} X = w^{2} \left( {X_{1 - n} } \right)} & {w^{2} X,Y = w^{2} Y,X} \\ {w^{2} X,Y = cor\left\{ {X_{1 - n} ,Y_{1 - n} } \right\} \times \sqrt {w^{2} \left( {X_{1 - n} } \right) \times w^{2} \left( {Y_{1 - n} } \right)} } & {w^{2} Y = w^{2} \left( {Y_{1 - n} } \right)} \\ \end{array} } \right] $$
(10)

Accordingly, correlation matrix, inverse and eigenvalues of matrix U could be derived to further characterize the imprecision of nonparametric population estimates. These measurements can help to assess ill-conditioning and instability of the model by evaluating the condition number and potential strong correlations in parameter imprecision estimates [23].

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baverel, P.G., Savic, R.M. & Karlsson, M.O. Two bootstrapping routines for obtaining imprecision estimates for nonparametric parameter distributions in nonlinear mixed effects models. J Pharmacokinet Pharmacodyn 38, 63–82 (2011). https://doi.org/10.1007/s10928-010-9177-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10928-010-9177-x

Keywords

Navigation