01.12.2013 | Research article | Ausgabe 1/2013 Open Access

# Bayesian methods to determine performance differences and to quantify variability among centers in multi-center trials: the IHAST trial

- Zeitschrift:
- BMC Medical Research Methodology > Ausgabe 1/2013

## Electronic supplementary material

## Competing interests

## Authors’ contributions

## Background

## Methods

### Frequentist IHAST methods

### Bayesian methods in general

### Bayesian methods applied to the IHAST trial

_{ jk }denote the number of subjects assigned to treatment j in center k and X

_{ ijk }be the values of the covariates for the i

^{ th }subject in the j

^{ th }treatment group at the k

^{ th }center (i = 1,…,n

_{ jk }, j = 1,2, k = 1,…,30). Let y

_{ ijk }= 1 denote a good outcome (GOS = 1) for i

^{ th }subject in j

^{ th }treatment in center k and y

_{ ijk }= 0 denote GOS >1 for the same subject. Also let $\underset{\xaf}{\beta}$ be the vector of covariates including the intercept μ and coefficients β

_{ 1 }to β

_{ 11 }for treatment assignment and the 10 standard covariates given previously. Conditional on the linear predictor ${\underset{\xaf}{x}}_{i}^{T}\underset{\xaf}{\beta}$ and the random center effect δ

_{ k , }y

_{ ijk }are Bernoulli random variables. Denote the probability of a good outcome, y

_{ ijk }= 1, to be p

_{ ijk }. The random center effects (δ

_{ k, }k = 1,…,30) conditional on the value σ

_{ e }are assumed to be a sample from a normal distribution with a mean of zero and sd σ

_{ e . }This assumption makes them exchangeable: δ

_{ k }| σ

_{ e }~ Normal (0, σ

^{ 2 }

_{ e }). The value σ

_{ e }is the between-center variability on the log odds scale. The point estimate of σ

_{ e }is denoted by s. The log odds of a good outcome for subject i assigned to treatment j in center k are denoted by θ

_{ ijk }= logit(p

_{ ijk }) = log(p

_{ ijk }/(1 – p

_{ ijk })) (i = 1,…,n

_{ jk }, j = 1,2, k = 1,…,30).

_{ 1 }to β

_{ 11 }are coefficients to adjust for treatment and 10 standard covariates that are given previously and in Appendix A.1.

_{ e }

^{ 2 }is informative and is specified as an inverse gamma distribution (see Appendix A.3) using the expectations described earlier. Values of σ

_{ e }close to zero represent greater homogeneity of centers.

_{ k }) are calculated. A guideline based on interpretation of a Bayes Factor (BF) [14] is proposed for declaring a potential outlier “outlying”. Sensitivity to the prior distribution is also examined [19].

### Specific bayesian methods to determine outlying centers

_{ k }, is greater than 3.137σ

_{ e }in absolute value (Appendix A.4)

_{.}The corresponding prior probability of a specific center being an outlier is 0.0017: Pr(center k is an outlier) = 2 *Φ(−3.137)[22], where Φ(z) is the standard normal distribution function.

_{ k }and σ

_{ e }[22]. The Bayes factor is also calculated for each of the 30 centers to quantify and interpret the strength of evidence. The BF for center k is defined as follows:

_{ k }is less than 0.316 then there is “substantial evidence” for center k being outlying [14]. Similarly if the BF for there being at least one outlying center is less than 0.316 there is substantial evidence for at least one outlying center.

### Bayesian methods regarding other determinants of outcome

_{ k }= n

_{ 1k }+ n

_{ 2k }and classify centers as either very large (n

_{ k }≥ 69 subjects; 3 centers, 248 subjects), large (56 ≤ n

_{ k }≤ 68 subjects; 4 centers, 228 subjects), medium (31 ≤ n

_{ k }≤ 55 subjects, 7 centers, 282 subjects)) and small (n

_{ k }< 31 subjects, 16 centers, 242 subjects). To determine if geographic location predicted outcome, IHAST centers were categorized post hoc as being either North American (US and Canada, 22 centers, 637 subjects) or non-North American (Europe, Australia, New Zealand, 8 centers, 363 subjects). To determine if there was evidence of “learning” over the entire course of the study, outcomes of the first 50% of subjects enrolled in the study (all centers) were compared with outcomes from the second 50% of subjects enrolled (all centers). Similarly, within each center, the outcomes of first 50% subjects were compared to the second 50%.

## Results

### Frequentist analysis

_{ k }) are plotted against the proportion of good outcome for each center and 95% and 99.8% exact binomial confidence intervals are provided. The horizontal line on the funnel plot represents the overall weighted fixed effect good outcome rate (66%). Centers outside of the 95% and 99.8% confidence bounds are identified as outliers. Accordingly, using this method, IHAST centers 26 and 28 would be identified as outliers, performing less well than the rest of the centers, with good outcome rates of 51% and 42%, respectively. However, importantly, patient and center characteristics are not taken into account in this plot.

### Bayesian analysis

^{ th }subject assigned the j

^{ th }treatment in center k is:

_{ k }is the random center effect. The posterior means of the center effects along with 95% CI’s are given in Figure 2.

_{ e }) is s = 0.538 (95% CI of 0.397 to 0.726) which is moderately large. The horizontal scale in Figure 2 shows ± s, ±2 s and ±3 s. Outliers are defined as center effects larger than 3.137σ

_{ e }and posterior probabilities of being an outlier for each center are calculated. Any center with a posterior probability of being an outlier larger than the prior probability (0.0017) would be suspect as a potential outlier. Centers 6, 7, 10 and 28 meet this criterion; (0.0020 for center 6, 0.0029 for center 7, 0.0053 for center 10, and 0.0027 for center 28). BF’s for these four centers are 0.854, 0.582, 0.323 and 0.624 respectively. Using the BF guideline proposed (BF < 0.316) the hypothesis is supported that they are not outliers [14]; all BF’s are interpreted as “negligible” evidence for outliers.

### Subgroup analysis

### Sensitivity analysis

_{ e }, for each of 15 models fit. For the first four models, when non important main effects of race, history of hypertension, aneurysm size and interval from SAH to surgery are in the model, s is around 0.55. The point estimate s is consistently around 0.54 for the best main effects model and the models including the interaction terms of the important main effects. In conclusion, the variability between centers does not depend much on the covariates that are included in the models.

_{ e }= 0.538, 95% credible interval for σ

_{ e }0.397 to 0.726). No center was declared an outlier and no center-specific or other subgroups were associated with outcome. Sensitivity analyses give similar results.

## Discussion

_{ 26 }= 57, n

_{ 28 }= 69) were identified as outliers by the funnel plot but with the Bayesian approach leading to shrinkage, and also adjustment for covariates they were not declared as outliers. Funnel plots do not adjust for patient characteristics. After adjusting for important covariates and fitting random effect hierarchical Bayesian model no outlying centers were identified.

_{ e }are used.

## Conclusion

## Statistical appendix

### A.1. List of potential covariates

### A. 2. Deviance Information Criterion (DIC)

### A. 3. Justification and Description of Prior Distributions

### A. 4. Calculating the Prior Probability of Being an Outlier

^{ -1 }[0.5 + ½ * (0.95

^{ 1/n })][22]. For example, when comparing 30 centers, n = 30 and m is 3.137 and the prior probability of being outlier for a specific center is 0.0017.