Background
In endemic settings malaria usually presents with rather non-specific symptoms, such as fever, and not all sick individuals with malaria parasites are really suffering from clinical malaria. This is because most of the population may be infected with Plasmodium falciparum parasites, without these causing any acute illness. It follows that the presence of parasites in a sick person does not necessarily mean that malaria is the cause of the illness. In field trials of novel interventions, estimates of efficacy need to be made using case definitions with high specificity. Otherwise efficacy will be underestimated.
The greater the parasite density in the blood the more reasonable it is to assume that an illness is caused by malaria, so a case definition for clinical malaria for use in a trial can be obtained by defining a parasite density cut-off specific for the surveillance mechanism of choice (local health centre, hospital, active case detection). The sensitivity and specificity of different parasite density cut-offs can be obtained by modeling the excess risk of fever as a function of parasite density [
1,
2], where the comparator is the risk in aparasitaemic individuals. This has been used in a number of trials to decide upon an appropriate cut-off [
3‐
5]. Ideally this analysis is carried out in the same population (and age groups) as the vaccine trial and using the same morbidity surveillance system, since relationships between morbidity and infection depend on age and immune status [
6‐
8].
This approach has been endorsed by the WHO Study Group on Measures of Malaria Vaccine Efficacy for obtaining case definitions for use in pivotal trials of malaria vaccine [
9]. Efficacy estimates based on this algorithm can be easily obtained using standard software and are appropriate for defining the primary outcome for trials aiming to achieve registration.
Applying such an algorithm though does not guarantee that efficacy estimates will be unbiased, and does not provide an interpretation of how a vaccine is acting. As secondary objectives of malaria vaccine trials, investigators should be interested also in drawing inferences about whether the vaccine acts in accordance with its design and how it interacts with natural immunity. This paper uses simulations of trials to consider the theoretical performance of this method for different kinds of vaccines, and suggests a range of additional exploratory analyses that can be carried out in order to better understand vaccine action. The simulations consider the likely effects of different kinds of vaccines but the same approach is applicable to the analysis of the clinical impact of any effective intervention against malaria.
Results
The parasite density distributions in the community (Figure
1a) vary as straightforward consequences of the primary effects of vaccination. The pre-erythrocytic vaccine (A) halves the number of slide positive individuals in each category (a simplification of what we expect in a field study, where superinfection may occur); the effect of the simulated asexual blood stage vaccine (B) is more complicated, as it disproportionately reduces the frequency of high parasite densities, and slightly increases the frequency of very low parasite densities by shifting each individual to a lower density (Figure
1a). One consequence of this is that the highest density class is not represented among individuals who receive vaccine B, since any individual who would have been in this density class is now in the second highest class. The simulated anti-disease vaccine (C) has no effect on the parasite density distribution in the community.
The distributions of parasite densities in the clinical cases differ among the three vaccines (Figure
1b). For all the vaccines there is a background incidence of non-malaria disease, which is assumed to occur independently of the parasite density (left hand side of figure
1b), corresponding to non-malaria illness and is the same in all groups. For all three vaccines fewer cases are expected at each positive value of the density distribution than occur in the placebo group.
These differences in parasite density distributions lead in turn to different relationships between incidence of disease and the community parasite density distribution, depending on the action of the vaccine (Figure
1c), and hence to different curves for the relationship between the attributable fraction, the frequencies of clinical cases with different densities, and the operating characteristics of case definitions (Figure
1defg, Table
2). The relative risk of a given parasite density among cases, relative to the risk in controls, is the same for the anti-blood stage vaccine B as for the placebo arm (because the risk of disease, conditional on the parasite density, is the same in both arms, and the number of disease cases with no parasites is unchanged by the vaccine). For vaccine A, the proportion of cases at any given positive density is lower than in the corresponding proportion of cases in placebo recipients, because more of the cases with non-malaria etiology are now aparasitaemic, so the relative risk of a given parasite density among cases relative to controls is lower than in placebo (Figure
1c). For vaccine C the curve in Figure
1c is also lower than that for placebo, but this is because there are fewer parasitaemic cases- there is no change in the number of aparasitaemic ones.
Table 2
Results of simulated vaccine trials
Figure 1a: Distribution of parasite densities in the community (from which simulated datasets are sampled) | Frequency is halved at each density above zero. Frequency of zero parasite density increases to compensate. | Frequency of low parasite densities increases; frequency of high parasite densities decreases. Frequency of zero parasite density unchanged. | Same as placebo |
Figure 1b: Distribution of parasite densities in all disease cases 1b) (from which simulated datasets are sampled) | Frequency relative to that in placebo decreases with increasing parasite density. | Frequency relative to that in placebo decreases with increasing parasite density | Frequency relative to that in placebo decreases with increasing parasite density |
Figure 1c: Relative risk of given parasite density in disease cases relative to controls | At any given density, reduced relative to placebo | Same as placebo | At any given density, reduced relative to placebo |
Figure 1d: Attributable fraction of cases by parasite density (Figure 1d) | At any given density, reduced relative to placebo | Same as placebo | At any given density, reduced relative to placebo |
Figure 1e: Distribution of parasite densities in clinical malaria cases | Frequency of high parasite densities lower than in placebo | Frequency of high parasite densities lower than in placebo | Frequency of high parasite densities lower than in placebo |
Figure 1f: Sensitivity of case definition, by parasite density | Same as placebo | At any given density, reduced relative to placebo | At any given density, reduced relative to placebo |
Figure 1g: Specificity of case definition by parasite density | Same as placebo | At any given density, increased relative to placebo | Same as placebo |
Figure 1h: Efficacy estimate by parasite density cut-off (x) | Estimated efficacy increases with cut-off approximates the true efficacy at high cut-off values | Estimated efficacy increase with cut-off and exceeds the true efficacy at high cut-off values | Estimated efficacy increase with cut-off and approximates the true efficacy at high cut-off values |
Figure 1i: Power of study, by parasite density cut-off | Reaches a maximum of about 67% at a cut-off of about 10,000/μl | Reaches a maximum of about 87% at a cut-off of about 40,000/μl | Increases to 100% at a parasite density of about 60,000/μl |
Estimated efficacy using latent class model | 46.1% (18.7%) | 55.6% (23.1%) | 55.2% (16.5%) |
Power using latent class model (1-β) | 59.2% | 82.4% | 71.2% |
Similarly, and as a direct consequence of the curves shown in Figure
1c, for these vaccines A and C, but not for vaccine B, the proportion of disease cases attributable to malaria at any given density is less in the active than in the placebo arm (Figure
1d).
The relative frequencies of malaria cases at different parasite densities (Figure
1e) show similar patterns to those of the relative frequencies of all disease cases (Figure
1d), but instead of intersecting at a non-zero point on the vertical axis, the plots pass through the origin, since clinical malaria cannot occur in the absence of parasites.
Integration of the curves in Figure
1e (Equation 3) then gives the curves for the sensitivity of parasite density cut-offs. There are clear differences between the vaccines. At any parasite density the sensitivity for vaccine A is equivalent to that for placebo, but for vaccines B and C it falls below that of placebo. The specificity, in contrast, is the same as placebo for vaccines A and C, but is higher than placebo in vaccine B (Figure
1g).
These differences in sensitivity and specificity have effects on the estimation of efficacy. At high values of
x, corresponding to high specificity, the mean efficacy estimate (of the 500 simulated trials) for vaccines A and C approaches the true efficacy, while for vaccine B (where specificity in the vaccine arm is higher than in the placebo arm) the efficacy is overestimated. The proportion of trials giving statistically significant results (Figure
1i) (assuming them to have been analysed using a fixed cut-off) gives the power of the study. The power of the trials of vaccines A and C showed maxima at relatively low cut-offs, indicating that different cut-offs must be used if the aim is to avoid bias in the estimate of efficacy, from those used to optimise power.
Discussion
Field trials of interventions against malaria need to have easily interpretable primary outcome measures in order to make an impact on regulatory and policy decisions. At the same time, field trials represent the main opportunity for experimental study of immuno-epidemiology of malaria and need to be fully exploited to further understanding of the mechanisms of action of the interventions. The analyses demonstrated in this paper are intended to contribute to plans for such secondary analyses.
The three hypothetical vaccines simulated in this study represent limiting cases of the effects of different interventions on clinical malaria. They do not correspond on a one-to-one basis to real vaccines, but rather to possible intervention effects. Any real intervention might have secondary effects on the other measures in addition to a primary effect on force of infection, asexual parasite growth, or on pyrogenic thresholds. Analyses of trial datasets should aim to identify contributions of an intervention to each of these dimensions of protective efficacy.
The primary outcome of most trials is likely to use a single parasite density cut-off that is chosen to give a high specificity in order to reduce underestimation of efficacy since decisions to develop a vaccine depend on the magnitude of protection. However a highly specific case definition does not necessarily result in optimization of study power (Figure
1i) and in early stages of vaccine development it might be most important to to test whether there is any effect at all so a lower cut-off would be more appropriate. There is no reason why a threshold chosen to reduce bias in efficacy should be particularly appropriate for any other purpose and in particular a diagnostic threshold optimized for use in a trial is not necessarily appropriate as a tool in clinical management [
12].
Analyses of trial data using such parasitaemia cut-offs have generally not quantified the bias that remains. The true efficacy is defined as
E = 1 -
I
V
/
I
P
where
I
V
is the case incidence in vaccinees and
I
P
is the case incidence in the placebo arm, and the usual estimate of efficacy is
= 1 -
I
V
(
x)/
I
P
(
x) where
I
V
(
x) is the incidence of cases at or above cut-off (x) in the vaccine arm, and
I
P
(
x) the corresponding incidence in the placebo arm. Assuming the specificity (ψ) of the diagnostic cut-off to be the same in both arms then an estimate of E adjusted for the effects of the imperfect case definition is:
where
λ is the attributable fraction in the placebo arm. A potential improvement in efficacy estimates is to thus to estimate
from
,
λ, and ψ and to use
as an estimate of
E. If ψ is sufficiently close to unity, then the difference between these two estimates is small.
Exploratory analyses of the behaviour of
suggest that it can be sensitive to
x (Aponte, pers. comm), though it should not be so if the assumptions underlying its estimation are correct. The non-linear logistic regression model most widely used for defining the parasitaemia cut-off[
1] assumes a specific parametric form for the relationship between relative risk and parasite density. This can lead to severely biased estimates of the specificity of the cut-off if the relationship happens not to conform to this pattern [
10]. This assumption is avoided in the latent class models that we have used in this paper which fit non-parametric curves for this relationship.
It is evident from Figure
1h though that the bias in efficacy does not only arise from lack of specificity in cut-off, and need not always be in the direction of underestimating efficacy. Bias also arises because of the specificity of cut-offs can differ between vaccine and placebo. Our model indicates that this is particularly a problem for asexual blood stage vaccines (vaccine B) (Figure
1g). This leads to the idea that perhaps different cut-offs should be used for vaccine and for placebo groups[
7]. To justify this in practice though, it would be necessary to demonstrate a statistically significant difference between trial arms in the specificity vs cut-off relationship. This would be a difficult statistical exercise, (because the specificity is estimated only indirectly), and would lead to considerable difficulties in describing the results convincingly especially if the efficacy proved highly sensitive to the choices of cut-offs in the different groups. Because sample size is determined in order to give adequate power to measure the primary outcome (effect on case incidence), most trials are too small to conclusively demonstrate whether the specificity vs cut-off relationship varies between arms. The decision of the WHO Study Group on Measures of Malaria Vaccine Efficacy not to recommend trial-arm specific cut-offs [
9] is therefore probably well-founded.
The most satisfactory alternative to using a single cut-off would probably be to estimate the total number of clinical malaria cases in each arm of the trial by assigning a probability to each fever case, rather than classifying each case dichotomously as above, or below, cut-off. This approach has not so far been used in analyses of clinical trials though it has been proposed as an alternative to the arbitrary choice of a cut-off [
12]. It has been used in observational epidemiological studies [
13,
14]. The preferred estimation method is to use a Bayesian latent class model to estimate the probabilities [
11] carrying out this analysis separately for both placebo and vaccine arms. The simulations of this approach presented in Table
2 suggest that it has comparable power to that of the cut-off method. Moreover, interval estimates for all the quantities involved are readily available using software written in Winbugs [
15] available from the author.
Such secondary analyses using latent class models, or considering the whole range of possible parasite density cut-offs will also help to identify possible biases in efficacy estimates made using single case definitions, at the same time as analysing the kind of protection. Where multi-centre trials give heterogeneous efficacy estimates, it will be important to examine whether this can be accounted for by differential bias in the primary outcome measurements.
In a real trial the reduction in proportion of individuals infected varies over the trial period, depending on the time course of incidence, patterns of treatment with anti-malarial drugs, and on the variation between individuals in exposure to vectors and responses to vaccination. These factors significantly complicate the analysis of relationships between infection and morbidity because, strictly speaking, the comparison should always be between contemporaneous data. This problem is particularly acute if parasites are cleared at the start of the trial, leading to complicated dynamics of infection and disease during the trial follow-up period. The present simulations do not address the implications of sub-patent parasitaemia. This especially complicates analysis of effects of asexual blood stage vaccines because reduction of parasite densities differentially inflates the proportion of false-negative blood slides in vaccinated individuals.
It follows that the analyses illustrated in this paper represent considerable simplifications of those that might be carried out in a real trial, where these complicating factors need to be taken into account. Nevertheless, when feasible, it would be logical to carry out secondary analyses corresponding to the different panels in Figure
1. Such analyses would help to clarify whether the effect of an intervention corresponds to that anticipated on the basis of the parasite stage that is targeted, and would highlight whether the primary measure of efficacy results from any unexpected behaviour in the parasitological and clinical data used to estimate it.