Inference of synergy/antagonism between anticancer drugs from the pooled analysis of clinical trials

To test our statistical methodology, on May 7, 2010, we searched PubMed with the following search key: cancer phase II clinical trial overall response rate. From the list of returned abstracts we selected in order of appearance the first 1,000 clinical trials. This number was chosen to balance the effort of manually extracting the required data from the PubMed abstracts and the desire to include as many trials as possible. Following an initial assessment of our methodology with that subset of clinical trials, on August 9, 2011, we searched PubMed again to extract new reports between this date and the previous search. This resulted 163 additional trials adding to a total of 1,163 trials. The reason for the latter search was to allow us to investigate more recent trends. We did not found any significant differences from the analysis of the initial set of 1,000 trials and the final set of 1,163 trials. The complete list of trials is reported in the Additional file 1. Our primary measure for treatment success was the clinical overall response rate (ORR), defined as the percentage of patients whose cancer shrinks (partial response) or disappears (complete response) after treatment. Recognizing the limitations of comparing response rates for each cancer type across separate trials, we chose the overall response rate as the main outcome measure. This choice was based on the assumption that most phase II trials used standard RECIST response criteria, and were powered for a clinically relevant response rate that could lead to a “go no-go” decision for a phase III study. Overall, 184 agents were tested as single agents or in combination in the collected trials.

Each clinical trial reports the number of patients with an overall response (n) and the trial sample size (N). For a given combination tested in a given trial, n is modeled as a random variable following the binomial distribution P n | N , p = N n p n 1 − p N − n, where p is an unknown parameter representing the probability that a patient manifest a partial or complete response to the treatment. The Bayesian posterior distribution of p is given by a beta distribution P p | n , N = 1 B α , β p α − 1 1 − p β − 1, where α=n and β=N-n and B(α,β) is the beta function (Additional file 2). For a given combination tested in multiple trials, we take into account that p may be different due to the use of different dose, schedule or cancer subtype on each trial. To account for these differences, we constructed a statistical methodology that models the existence of multiple classes of trials (among those testing the same combination) with different values of p (Additional file 2). In our dataset, there were 166 combinations tested in two or more trials. When pooling the clinical trials by the combination tested, in 142 of these combinations the data indicates that all trials are statistically equivalent as determined by the Bayesian method. In these cases we pooled together the data from clinical trials testing the same combination even though some were conducted in different cancer subtypes. For the remaining 24 combinations, there are significantly different response rates depending on the cancer type. In this latter case the Bayesian method returns two or more groups, each containing one or more cancer types. When each group was represented by only one trial we removed those trials from our search of synergistic/antagonistic combinations. Otherwise we removed the trials in the group with lowest number of trials. The excluded trials are indicated in the Additional file 3. These trials were removed because the reported ORR was inconsistent with the report by trials testing the same combination in the same cancer type. When all trials are statistically equivalent, p follows a beta distribution with α = ∑ _i n _i and β = ∑ _i (N _i  − n _i ), where the index i runs over all trials testing the combination. The expected probability of response rates is computed from the mean of the beta distribution mean(p) = α/(α + β) = ∑ _i n _i /∑ _i N _i , and the associated ORR is computed as ORR₀ = 100%×mean(p).

In the absence of agent interactions, the probability that a patient responds to a treatment based on two agents equals one minus the probability that he/she does not respond to either treatment: q _ij  = 1 − (1 − p _i )(1 − p _j ), where p _i and p _j are the response probabilities for each agent when used as a single agent. The probabilities p _i were estimated using trials where the agents were tested as single agents. In a trial where N patients were treated with the two agents i and j, we expected n responses with a binomial probability distribution P n | N , q ij = N n q ij n 1 − q ij N − n. We estimated the probability that there was synergy as the probability to obtain as many or more responses given a non-interacting agents hypothesis: p synergy , ij = ∑ m = n N P m | q ij , N. Similarly, we estimated the probability that there was antagonism as the probability to obtain as many or less responses, given a non-interacting hypothesis: p antagonism , ij = ∑ m = 0 n P m | q ij , N. Finally, the expected ORR under the assumption of non-interacting agents was defined as ORR_1,ij  = 100%mean(q _ij ) = 100%[1 − (1 − mean(p _i ))(1 − mean(p _j ))]. The simulations to test the null (non-interacting) model approach were performed as follows. Given a sample size N, uniformingly sampled values between 0 and 1 were generated for the probability of response to drug 1, p ₁, the probability of response to drug 2, p ₂, and the probability to respond to the combination of drug 1 and 2, p ₁₂. Using the probabilities p ₁ and p ₂, the probability of response to the combination of drug 1 and 2 in the absence of drug interactions, q ₁₂ = 1 − (1 − p ₁)(1 − p ₂), was computed. Then, n ₁, n ₂ and n ₁₂ responses out of N attempts were generated using a binomial model with probability of success p ₁, p ₂ and p ₁₂, respectively. Using the generated number of responses, the posterior means p _0,1 = n ₁/N, p _0,2 = n ₂/N and p _0,12 = n ₁₂/N were computed. Finally, using q _0,12 = 1 − (1 − p _0,1)(1 − p _0,2) and n ₁₂ as input, p _synergy,12 and p _{antagonism,12} were computed. This procedure was repeated 10,000,000 times. The manifestation of synergy/antagonism was determined by comparing the expected ORR as obtained from the null model ORR₁ = 100%q ₁₂ with the ORR as observed ORR₀ = 100%p ₁₂. Specifically, the probability for synergy p _synergy,12 was calculated as the fraction of times that ORR₀>ORR₁ and p _synergy,12≥0.05. Similarly, the probability for antagonism p _synergy,12 was calculated as the fraction of times that ORR₀<ORR₁ and p _{antagonism,12}≥0.05. To compute these fractions we divided the [0–1] interval in bins of size 0.01.

In the 2-agent approximation model the ORR for each agent combination c is derived from parameters quantifying the response to a single agent and the interaction between two agents. These combinations are constructed out of N _a agents. The combinations are specified using the agent to combination matrix _. s _ci , where s _ci  = 1 if agent i is part of the combination c and s _ci  = 0 otherwise. The parameter h _i denotes the response rate to agent i when tested as a single agent. The parameters J _ij are introduced to quantify corrections due to agent interactions. The probability p _c to respond to a combination c is written as p c = ∑ i = 1 N a s ci h i + ∑ i = 1 N a − 1 ∑ j = i + 1 N a s ci s cj J ij and the associated overall response rate is ORR _c  = 100%×p _c . To understand the equation for p _c , it is better to focus on the case when the drugs do not interact. In this case p c = 1 − ∏ i = 1 N a 1 − h i s ci = ∑ i = 1 N a s ci h i − ∑ i = 1 N a − 1 ∑ j = i + 1 N a s ci s cj h i h j. Therefore, in the absence of interactions the equation of p _c can be written as we postulated above with J ij 0 = − h i h j. The difference between an observed J _ij and the non-interacting value Δ J ij = J ij − J ij 0 is used to quantify synergy (when positive) and antagonism (when negative). The 2-agent model can be rewritten as p c = ∑ m = 1 N v A cm x m, where N _v is the number of variables (unknown hs and Js), the x _m are variables representing the hs and Js parameters, and A _cm is the variable to a combination matrix (A _cm  = 1 if variable m appears in the equation for combination c and A _cm  = 0 otherwise). The variables x _m were estimated by solving the corresponding least-squares problem, with some additional considerations to account for the variability in the empirical estimates of p _c (Additional file 2). The 2-agent approximation to the ORR was given by ORR_2,ij  = 100%[∑ _i s _ci mean(h _i ) + ∑   _i <j s _ci s _cj mean(J _ij )].

The kolmogorov-smirnov test [10] was used to determine the probability that two lists of ORRs were generated from the same distribution.

The response rate to a combination of agents may depend on the dose of each agent, the treatment schedule, and the cancer subtype. We developed a statistical methodology that determines if a set of trials testing the same combination is statistically equivalent, or if there are trials with statistically different response rates. To this end, we developed a Bayesian method that takes into account the variations in the estimated response rates due to finite sample sizes. This Bayesian method determines whether the number of responses reported in a set of clinical trials are consistent with a unique response rate, up to variations determined by the finite sample sizes, or whether the trials cluster in two or more groups with statistically significant response rates. In our dataset, there were 166 agent combinations that were tested by two or more clinical trials. In 142 combinations the observed response rates were statistically equivalent. Only in 24 combinations there is evidence of trials with statistically different response rates (Additional file 3).

We note that the observation of statistical equivalence does not imply that the response rates do not depend on dose or cancer type. By necessity, we made the assumption that in most trials within the dataset studied, the agent doses are standard based on standard maximal tolerated doses defined by prior phase I trials. Response rates were also comparable because similar agent combinations or combinations of similar drug class were usually tested in the same cancer subtype. We recognized that targeted therapies were often used to treat patients selected on the basis of a companion biomarker. For example, within the studied dataset, there were 32 trials testing trastuzumab as a single agent or in combination. In all cases the patients enrolled were Her2/neu+, the target of trastuzumab. Although we assumed that within the studied dataset the trials associated with the same agent combination are statistically equivalent, we acknowledged that in some instances the results that follow are biased towards specific cancer subtypes. For example, all the interactions with trastuzumab are relevant in the context of Her2/neu+patients and interactions with radiation are relevant in the context of localized disease. These biases will be reported when necessary.

The Bayesian method also reports the posterior mean of the response rate associated with every single agent and combination in our dataset. To investigate trends as a function of the number of agents tested, we stratified the combinations by the number of agents tested. For a given number of agents, we collected the posterior mean response rates of all combinations testing that many agents. From this list we determined the median and the lower and upper bound for the 90% confidence interval, as the combination with posterior response rate higher/lower than 50%/50%, 5%/95% and 95%/5% of the samples, respectively. We observed that the median over the posterior mean ORRs increased with increasing the number of agents in the combination (Figure 1). For single agents, the median ORR was below 30% and increased to above 70% for treatments with five or more agents. Similar trends were observed when restricting the analysis to trials in breast cancer (171 clinical trials), indicating that these observations hold for specific cancer subtypes as well. The ORR 90% confidence interval for combinations testing a given number of agents was quite broad. Therefore, not all combinations testing more agents outperformed those testing lesser agents.

Among the 514 combinations tested, 85 included novel monoclonal antibodies, allowing us to test whether targeted therapies are improving response rates. As expected, based on prior randomized trials, the distribution of posterior mean ORRs for trials testing monoclonal antibodies was shifted toward higher ORRs relative to the distribution for trials not including monoclonal antibodies (54% vs 46%, p-value of 0.0042 based on a Kolmogorov-Smirnov test). Furthermore, the median over posterior ORRs was higher for combinations including monoclonal antibodies independently of the number of agents in the combination (Figure 1), with the exception of combinations with four agents.

We then focused on agent combinations resulting in trials with ORRs that were significantly different than what expected from a null model with no interactions between the agents tested. From the clinical point of view, a combination was deemed synergistic when its ORR was significantly higher than what was expected from the null model of non-interacting agents, and antagonistic when significantly lower. The null model for non-interacting drug combinations is mathematically described in the methods section. In simple words, in the absence of drug interactions the probability that a patient responds to a two-drug combination is equal to the probability that he/she responds to at least one drug in the combination. The probability that a patient responds to at least one drug in the combination can be estimated using as input the trials testing the drugs as single agents. Once we have an estimate of the probability that a patient responds to the two-drug combination in the absence of interactions, we can determine the probability to observe as many or more responses in a trial with a given sample size (p _synergy ), the probability to observe as many or less responses (p _antagonism ) and the expected response rate in the absence of drug interactions ORR₁.

The ability of this approach to uncover synergistic/antagonistic interactions will depend on the sample sizes and the degree of synergy/antagonism. This is illustrated in Figure 2a-c for simulated data with sample sizes 100, 500 and 1,000. The red areas highlight the region where the number of observed responses is consistent with the null model of non-interacting agent. In turn, the white areas represent cases where our approach correctly determines that the two-drug combination is either synergistic or antagonistic. As expected, our ability to discriminate from the null model increases as the sample size increases. Of course, in addition to the statistical significance for synergy/antagonism we should focus on the effect size, i.e. how much the observed response rate (ORR₀) deviates from what expected from the null model for non-interacting drugs (ORR₁).

Figure 2d shows the classification of the two-agent combinations in our dataset into synergistic (p _synergy  < 0.05), antagonistic (p _antagonism  < 0.05) or non-interacting (p _synergy ≥0.05 and p _antagonism ≥0.05). Table 1 and 2 report the two-agent combinations with evidence for synergy and antagonism, respectively. About half of the predicted synergistic combinations are the standard of care in specific cancer types (e.g., docetaxel+doxorubicin in breast cancer), indicating that our analysis captures the current trends in cancer treatment (Table 1). The remaining synergistic combinations should be further studied to evaluate their potential to improve cancer treatment. In contrast, only one antagonistic drug combinations is currently used as standard of care for the corresponding cancer subtypes (Table 2).

We tested the hypothesis that synergy was more common in combinations using monoclonal antibodies, a class of targeted therapies. However, only 1 out 15 combinations in the list of synergistic 2-agent combinations included at least one monoclonal antibody (Table 1, rituximab+alpha-interferon). Although a small sample size, requiring future validation, these data support a lack of significant enrichment of synergy by the addition of a monoclonal antibody relative to other agent combinations (p-value = 0.99, Fischer’s exact test). This observation may also indicate that synergy is as common between chemotherapeutic agents as between a chemotherapeutic agent and a monoclonal antibody.

Understanding that the analysis assessing clinical synergy is limited by the availability of clinical trials testing each agent as a single agent and the two agents in combination, we performed a 2-agent approximation. A new agent is often added to an existing regimen that already includes two or more agents, without testing the new agent in combination with each agent within the existing regimen. Therefore, we estimated the response rate of a combination of two agents from a collection of trials where these agents appeared as part of a combination with more than two agents. We developed a model for the ORR as a function of parameters characterizing the single agent and 2-agent responses (see Methods). A leave-one-out-cross-validation scheme was used to evaluate the performance of the 2-agents approximation. For each combination, the empirically estimated ORR was removed from the dataset and then estimated using the 2-agent approximation. Figure 3 shows a scatter plot of the mean ORR as predicted by the 2-agent approximation (ORR₂) as a function of the mean ORR empirical estimates (ORR₀). We noticed that in most cases the predicted ORRs were lower than the empirical estimates, as can be deduced from the higher density of points in the lower-right triangle in Figure 3. Indeed, in 77, 76 and 69% of the combinations with 1, 2 and 3 or more agents, respectively, the predicted ORR was lower than the empirical estimate (ORR₂<ORR₀). Therefore, the 2-agent approximation can be used to estimate a lower bound to the ORR. More precisely, with about 75% confidence, the actual ORR is higher than what predicted by the 2-agent approximation. In practical terms this means that when the 2-agent approximation predicts a high ORR, with about 75% confidence we should expect as high or higher responses rates. On the other hand, if the 2-agent approximation predicts a low ORR, the prediction become less informative, since likely the ORR will actually be higher. Using the 2-agent approximation we estimated the ORR of all 2-agent combinations, provided that the two agents appeared together in at least one trial within our dataset (Additional file 4). We noted again that these predictions were likely underestimating the actual ORR. We also noted that these predictions were subject to the same biases discussed above. If an agent has been tested in a specific cancer subtype, then the predicted ORRs for combinations using that agent are expected to be valid within that subtype.