Flaws in design, analysis and interpretation of Pfizer's antifungal trials of voriconazole and uncritical subsequent quotations

We now report problems with the design and analysis of Pfizer's trials on its new antifungal agent, voriconazole. We identified two eligible trials for our systematic review of voriconazole [4]. They were both large, sponsored by Pfizer, and published in The New England Journal of Medicine in 2002 [5, 6]. As with our previous review [3], the first authors declined or were unable to answer our questions and referred to Pfizer, which in this case provided a response.

The first trial was a non-inferiority trial that compared voriconazole to liposomal amphothericin B as empirical treatment of fever of unknown origin in neutropenic cancer patients, using a composite endpoint with 5 separate outcomes [5]. Voriconazole was inferior to liposomal amphothericin B according to the authors' pre-specified criteria, and even significantly inferior according to the pre-specified analysis plan, which staff at the FDA pointed out in a subsequent letter to the journal [7]. More patients died in the voriconazole group and a claimed significant reduction in so-called "breakthrough" fungal infections in favour of voriconazole disappeared when infections arbitrarily excluded from analysis were included. The authors defined breakthrough fungal infections as those confirmed more than 24 hours post-enrolment. The reason for a 24-hour cut-point to exclude baseline fungal infections from analysis was not explained. We searched the references provided as justification for this cut-point in the article but could not find any relevant information. We have not seen such a cut-point in any of the more than 70 other trials of antifungal therapy we have reviewed [1‐4, 8], and in a later study of caspofungin, the same first author now used a 48-hour cut-point for the same outcome, in the same journal, and again without any justification or explanation [9].

We believe the use of arbitrary cut-points creates bias. When we included baseline infections that persisted despite treatment, we found 15 vs. 23 infections (P = 0.27) [4], whereas the trial report noted 8 versus 21 infections (P = 0.02) [5]. Our analysis is not only unbiased, it is also the clinically relevant one, as patients with baseline infections are part of the clinical reality when patients are treated on suspicion of a fungal infection.

The abstract was highly misleading. Despite the fact that voriconazole was clearly inferior to amphothericin B, it concluded that "Voriconazole is a suitable alternative to amphothericin B preparations" and referred to "breakthrough" fungal infections and a significant difference in nephrotoxicity, arbitrarily defined as a 1.5-fold increase of baseline s-creatinine values, which disappeared completely when the usual definition of a 2-fold increase was used [5]. Twenty-nine versus 32 patients had a 2-fold increase, whereas 43 versus 80 patients experienced a 1.5-fold increase (P < 0.001).

The second trial was flawed by design. It compared voriconazole to amphothericin B deoxycholate in the treatment of invasive Aspergillus infections [6]. The deoxycholate preparation was used without requirements for pre-medication, which is generally advised to reduce infusion-related toxicity, or substitution with electrolytes and fluid as advised to reduce nephrotoxicity, which can be practically abolished by this precaution [10, 11]. These omissions are particularly disturbing considering that the planned duration of treatment was 84 days. Voriconazole was given for 77 days on average, but the comparator for only 10 days, which precludes a meaningful comparison of the two drugs.

Both trial reports have been extensively quoted. As of 23 November 2005, the paper by Walsh et al. had been quoted 192 times and the paper by Herbrecht et al. 344 times [12]. We selected randomly 50 articles of any type, e.g. research reports, reviews, editorials and letters, that quoted a trial report (25 for each report) to elucidate whether the quotations reflected uncritically the main conclusions of the trial reports. Two articles that were coincidentally selected for both groups were replaced by the next on the randomisation list (see final list in the Additional file 1). We also looked at conflicts of interest, defined as being author of one of the quoted trial reports; being a Pfizer employee; being a consultant, speaker or on the advisory board for Pfizer; or being author of an article with no declaration of conflicts of interest, but having declared a conflict of interest related to Pfizer in another of the included articles.

Two authors independently evaluated each article and differences were settled by discussion. We excluded 10 articles that quoted Walsh et al. and 7 articles quoting Herbrecht et al. as they did not refer to any of the main conclusions, but to minor issues, e.g. hepatotoxicity.

Five of the 15 included articles that quoted Walsh et al. accepted their main conclusion whereas 7 noted that non-inferiority had not been demonstrated. Seven articles mentioned the confidence interval by Walsh et al., whereas only one article referred to the confidence interval published by the FDA. Six articles noted that voriconazole led to fewer break-through infections whereas one noted that this was not a protocol-defined successful outcome and another mentioned problems with the multiplicity of outcomes without allowance for multiple testing as also noted by the FDA. Two articles claimed that voriconazole had less nephrotoxicity whereas one noted that the proportions that had a doubling in serum creatinine was similar. Only 3 articles had authors with a conflict of interest, and only one criticism was raised in these.

None of the 18 included articles that quoted Herbrecht et al. raised any criticism of the design of this trial and only one mentioned that concerns had been raised about "differing durations of therapy, as well as the possibility of excessive interruptions in the amphotericin arm", but the authors went on to say, in the next sentence, that: "the conclusion that voriconazole is superior to amphotericin B for treatment of invasive aspergillosis has been welcomed by most clinicians" [13]. Eleven articles had authors with a conflict of interest.