Weather-based prediction of Plasmodium falciparum malaria in epidemic-prone regions of Ethiopia II. Weather-based prediction systems perform comparably to early detection systems in identifying times for interventions

Malaria epidemics are reported frequently and have caused high morbidity and mortality among all age groups in the African highlands [1‐4]. Early detection and accurate forecasting of the time, place and intensity of these epidemics is important for emergency preparedness, planning and response [5, 6]. Considerable efforts are being made to promote, develop and implement early warning systems for malaria epidemics in Africa [5, 7]. Ideally, public health and vector control workers would have access to a system that alerts them when substantial numbers of excess cases are expected, and such alerts should be sensitive (so that alerts are reliably generated when excess cases are imminent), specific (so that there are few "false alarms") and timely (so that there is adequate lead time to act). Generally, each of these performance characteristics is enhanced at the expense of another. The value of interventions – such as larviciding, residual house spraying and mass drug administration – to control malaria epidemics has been documented [8]. However, due to the explosive nature of malaria epidemics, the usefulness of such interventions in epidemic settings depends on timely information about the onset of a severe epidemic.

Early detection systems, which are used to detect epidemics once they have begun, can correctly identify periods that are defined by expert observers as "epidemic," albeit with varying specificity. A number of such systems have been proposed or implemented. For example, WHO has advocated the use of alerts when weekly cases exceed the 75^th percentile of cases from the same week in previous years [9] and other methods, based on smoothing or parametric assumptions, have also been considered [10‐12]. However, an early detection system, which generates alerts once unusually high case numbers are already observable, will be useful for targeting interventions only if it identifies epidemics at an early phase, when there is still time to take effective action [13] and if the epidemics persist (and indeed grow) over time so that action taken after the warning can still have an effect. It was previously shown, using weekly case numbers from 10 districts in the Ethiopian highlands over approximately 10 years, that simple weekly percentile cutoffs used for early detection are capable of identifying periods with unusually high malaria incidence, and that interventions that take effect within two weeks of such alerts could have a substantial impact in reducing excess cases [14].

While early detection systems appear to provide timely information about the onset of severe epidemics, they intrinsically trigger alerts only when unusual transmission is already underway. Another approach, known as "early warning," attempts to predict epidemics before unusual transmission activity begins, usually by using weather variables that predict vector abundance and efficiency, and therefore, transmission potential [6, 15‐18]. The advance notice provided by an early warning system could allow action to be taken earlier in the course of the epidemic, or could increase the span of time available to undertake control measures before the predicted excess cases occur. A number of authors have used weather factors to attempt to predict malaria epidemics [19‐24], and Teklehaimanot et al. [25] showed that polynomial distributed lag (PDL) models incorporating lagged effects of minimum temperature, maximum temperature and rainfall could mimic seasonal patterns of malaria incidence in the same ten sites for which early detection algorithms were evaluated. Because the significant weather predictors of malaria cases are lagged by four or more weeks, such prediction systems may, in principle, provide a means of anticipating unusual malaria incidence with more lead time than early detection methods.

Here, an attempt to combine these avenues of previous work is described, using modified versions of previously described models based on weather factors to provide predictions of Plasmodium falciparum cases in these 10 districts of Ethiopia, and evaluating thresholds that trigger warnings. The hypothesis tested here is that the use of predicted cases (rather than actual cases, as in our previous work on early detection) would reduce the precision of the alert thresholds (resulting in alerts whose timing was less well matched to periods of excess cases than those generated by early detection), as the price of obtaining the alerts with greater advance notice. In fact, the early warning system based on predicted cases performed slightly worse in most cases than the early detection system, but the performance was rarely much worse and occasionally slightly better. These comparisons are described and their implications for the choice of malaria prediction/detection systems in epidemic-prone areas of Africa are discussed.

The prediction algorithm indicates the overall pattern of cases well, yet underestimates the height of the largest peaks. Comparisons of the predicted and observed malaria cases, for each week in six of the ten districts, are shown in Figure 1. The model predicted the actual cases well, although the agreement between the observed and predicted cases varied from district to district. However, the models were not able to differentiate clearly between years with very high and moderately high peaks. To explore whether the predicted number of malaria cases using weather factors can accurately identify time periods with increased number of malaria cases, the timing of alerts triggered, for example, by a mean plus 1.5 standard deviation threshold algorithm, is presented in the same figure. Despite the fact that the actual height of peaks in the highest-incidence periods is poorly predicted by the model, the model nonetheless often triggered alerts prior to these high-incidence periods.

The prediction system generates alerts that could prevent nearly as many cases as alerts generated by a detection system. To obtain a quantitative estimate of the usefulness of the prediction algorithm as an early warning system, the %PPC obtained from alerts triggered by predicted cases were compared with %PPC obtained from alerts based on observed cases (Figure 2) under an early detection scheme similar to that previously analyzed [14]. Percentile and mean + standard deviation thresholds are shown, with each point representing a particular value of the threshold (e.g., 85^th percentile or mean + 1.50 standard deviations). The horizontal axis gives the number of alerts per year triggered by the particular threshold value, while the vertical axis shows the %PPC associated with that threshold value. Each point represents the mean across all 10 districts. Two different choices of the function for determining PPC (reducing cases to weekly mean: low-effectiveness, a and c, or weekly mean minus one s.d.: high-effectiveness, b and d) and the choice of window of effectiveness (eight weeks, a and b; 24 weeks, c and d) were considered. The performance of the predicted number of malaria cases using the mean plus (0.5, 1, and 1.5) standard deviation algorithm (for an eight-week window of low-effectiveness) reveals that it prevented 29%/0.9 alerts, 27.3%/0.6 alerts and 24.2%/0.43 alerts per year, which compares with 31.4%/0.85 alerts, 29.8%/0.65 alerts and 27.5%/0.52 alerts per year respectively when the observed cases are used to trigger alerts (Figure 2a). In general, relative to alerts triggered by observed cases, the alerts triggered by the predicted number of malaria cases performed slightly worse, within 5% of the detection system. All alerts triggered by predicted and observed cases potentially prevented larger numbers of cases than random alerts. Relative to the optimally timed alerts, both systems performed well, within 10%–20% of the best achievable performance. On average, the number of alerts per year triggered by the prediction system is less than the number of alerts triggered by the observed cases for the corresponding level of alert threshold. Comparative performance of the detection and prediction methods was insensitive to the length of the window of effectiveness and the choice of function to define potentially prevented cases (Figure 2).

Prediction-based systems perform much better in cold than in hot districts. To compare the relative importance of weather factors in cold and hot districts, the %PPC obtained from predicted cases in the cold and hot districts were evaluated separately. Figure 3 shows that alerts triggered by the predicted number of malaria cases in the cold districts perform much better than in the hot districts. Comparative performance in the cold and hot districts was insensitive to the length of the window of effectiveness and the choice of function to define potentially prevented cases. In all cases, the prediction-based alerts were able to prevent 10–25% more cases of malaria at a given sensitivity in cold districts than in hot ones. On the other hand, although, the performance of the detection algorithms in the cold and hot districts was similar with eight-week window of effectiveness, it performed better in the cold than in hot districts with 24-week effectiveness (not shown).

Timely and accurate information about the onset of P. falciparum epidemics is essential for effective control activities in epidemic-prone regions, especially those in which limited resources must be deployed to the areas of greatest need. In the Ethiopian highland fringe region, one such epidemic-prone area, early detection of epidemics based on simple algorithms for detecting excess cases had been shown to generate alerts that are well timed to precede periods of high incidence [14]. Early warning methods that provide earlier alerts may allow the interruption of transmission earlier in the epidemic, but perhaps at the expense of some level of accuracy. In this study, we have shown that predictions four weeks ahead, based on weather factors and past case numbers, can provide alerts that are of comparable value to those provided by an equivalent early detection system, based simply on observed cases.

An interesting feature of the results was that the prediction system performed well in generating alerts for control measures, despite the fact that the model under-predicts high peaks. Correlation analyses (data not shown) indicate that for most (but not all) districts, the model performed well qualitatively, in the sense of predicting more cases than expected from the weekly mean when such excess cases occurred, and predicting fewer when in fact fewer cases than the weekly mean occurred. This finding focuses attention on the fact that a system can give timely and accurate alerts for epidemic control, even if it is unable to provide accurate predictions of case numbers (Figure 1). The initial hypothesis was that the improved timeliness of an early detection system comes at the expense of some accuracy. The overall results show that these two effects nearly balance each other, so that early warning systems based on our predictive model provide alerts whose value in terms of epidemic control is comparable to those provided by equivalent early detection systems. In a separate analysis (not shown), these two effects were separated out. If the alert system is based on prediction, but the alerts are timed such that their effects start eight weeks after the alert (i.e., four weeks after the week in which the predicted cases cross the alert threshold, equivalent to the timing for early detection), they identify 5% to 10% fewer PPC than the equivalent detection algorithm. The main analysis (Figure 2) showed that the additional four weeks of notice available by implementing control measures so that their effects begin by the week on which excess cases are predicted (four weeks earlier than if the detection algorithm were used) nearly makes up for this deficit.

Studies have shown that temperature affects transmission in cold environments more than it does in hot environments [31, 32]. Thus the addition of minimum and maximum temperature into the prediction model contributes less to predictions in the hot districts than it does in the cold districts. The study revealed this differential effect of weather on malaria transmission. The weather-based prediction system performed much better in the cold than the hot districts. Two mechanisms could have been responsible for this difference: epidemic alert algorithms in general could be less useful in hot districts, or weather-based algorithms, specifically, may be less useful in hot districts. Since simple detection-based alerts performed similarly in hot and cold districts (at least with an eight-week window of effectiveness), it appears that the problem in hot districts is with prediction-based methods. However, when 24 weeks were used as the window of effectiveness, the early detection system, like the prediction system, performed better in the cold than the hot districts. This may be because of the shorter transmission season in the hot than cold districts, due to evaporation and drying up of breeding sites in hot districts, such that rainfall's effects on transmission last for fewer weeks in hot areas [25]. In conclusion, an early warning system using weather and other predictor variables is more reliable in relatively cold than hot districts.

Non-climatic factors such as population immunity, migration and drug resistance are believed to influence malaria transmission and have been cited as causes of malaria epidemics [33‐36]. The variability in accuracy of prediction seen in the ten districts may have been due to such factors and others [37‐41]. These findings are consistent with the findings of Zhou et al. which indicated that there was high spatial variation in the sensitivity of malaria outpatient number to climate fluctuations in East African highlands [30]. Determining the relative contribution of the non-climatic factors would be an important step in the development of an early warning system for malaria and a predictive model which incorporates such indicators would give more accurate predictions, but this is not feasible in practice at this moment due to the absence of quantitative data on these factors.

The model chosen for the prediction of malaria cases was based loosely on a model previously evaluated for its ability to explain seasonal variation in malaria incidence in the same data set [25]. The former model, in turn, used polynomial distributed lags of weather factors based on biological considerations about the effects of these weather factors on malaria cases. To that model, additional terms – an autoregressive term and an indicator variable for the week of the year (on the Ethiopian calendar) were added – to improve predictive power. The usefulness of this predictive model has been shown, but modifications of the model have not been systematically explored which might improve its predictive ability still further. Further work should consider a range of prediction models for their ability to generate accurate and timely alerts.