Insufficient evaluation of adverse events is not a proof of safety. Reply to LG Hemkens, U Grouven, R Bender, et al. [letter]

To the Editor: We read the letter from Hemkens et al. with interest [1]. We find it flawed with respect to scientific method, as well as inappropriate in its suggestion of a potentially more detailed statistical method. First, we need to comment that in our analysis we have taken the highest level of evidence available for the relationship between malignancy and use of insulin glargine (A21Gly, B31Arg, B32Arg human insulin), randomised controlled trials, and put it in the public domain. In our article, we observe (but do not interpret) the data, and only state that the use of insulin glargine in the circumstances of the studies included is not associated with any reduction or increase in malignancy rates.

The most fundamental error in the letter of Hemkens et al. [1] is that the main argument is based on the position that there is some probability that the use of insulin glargine increases the rate of cancer. If this was the case then there would be a hypothesis, however weak, to test. Such a hypothesis cannot be drawn from a series of papers on a possible increased risk of cancer linked with the use of insulin glargine published in Diabetologia [2‐5], as has been noted by other independent authorities [6‐8]. The authors’ own exploratory analysis [3], suggesting some relationship based on insulin dose, is evidently flawed, as was highlighted in an editorial published in the Lancet by a senior clinical trials' statistician [8]. That editorial also cautioned against the more general use of population observational analyses due to their unknown prescribing biases [8].

This is important because, with no overall prior probability that the use of insulin glargine increases or decreases malignancy rates, our own data cannot be criticised for including or excluding any particular relative risk (offering a ‘proof’), as the authors attempt to do in their letter [1]. It is only possible to say that overall insulin glargine is not likely to increase the malignancy rate by more than 36% or reduce it more than 40%. In particular, since there is no prior probability of direction of change, the likelihood of any effect from insulin glargine clusters around our central estimate of risk (risk reduction of 10%) given that such a probability will have a close to normal distribution over the group of studies we include.

This is now the strongest statement available for discussing a potential increased risk of cancer (and hypothesis-setting for further testing, if deemed necessary). In addition, as noted above, this would be a stronger argument than the modified probabilities arising from the observational data, which have no direction of likelihood.

The authors of the letter [1] make a point of commenting on our data on breast cancer. Their difficulty arises from results on the use of insulin glargine vs no insulin (not other insulins) in observational studies, which have been noted by the authors of these studies themselves to be likely to be subject to statistical and methodological uncertainties [3, 4]. It is of course likely that people with diabetes with other co-existing illnesses will be more likely to use insulin and, speculatively, cancer centres will be in more sophisticated hospitals, in which modern insulins are more likely to be available for therapy. Again, we make no claim that our data on breast cancer suggest any change in malignancy risk but, equally obviously, it is important to put these data in the public domain in the event that others use them for the purposes of a meta-analysis.

We comprehensively discussed the limitations of our data, pointing out, as do Hemkens and colleagues in their letter [1], the problem of small numbers of events, which is partly due to the short duration of many of the studies available. However, the quality of these studies is high; commercial studies are generally of high-conduct quality due to continuing external inspection of investigators and a significant probability of external review by national drug regulators.

In our opinion, Hemkens and colleagues are wrong in suggesting alternative statistical approaches to combining data from different studies. We note that in their own study they seem to apply statistical techniques to data regardless of the validity of these approaches [3, 8, 9]. Suggesting the application of tests of non-homogeneity to multiple studies with very small numbers of events is wholly inappropriate—non-homogeneity testing is simply not powerful enough to be useful in this situation. Sensitivity analyses can be useful (indeed we break down the data by type of diabetes and by duration of study, showing that the findings are consistent), but it is wrong to suggest their wider application when the number of events is already marginal, as stochastic effects are likely to appear.

There is no data discrepancy between our paper and the short communication from Rosenstock and colleagues [10]. In that article, Rosenstock and colleagues explicitly describe that 20 events were reported as serious adverse events on insulin glargine (a relative risk reduction of 37%), and we also report 20 such events. Table 2 of the article by Rosenstock et al. [10] clearly presents both serious and non-serious adverse events, as is made clear in the footnote to the table.

We accept entirely that the observational data that we presented in the article are weak, just as we note that the observational study of Hemkens and colleagues on the same topic is weak. We make no other claim than pointing out that the data presented are consistent with our higher-level data. We would have been correctly criticised for having access to these observational data in the sanofi-aventis database and not placing them in the public domain.