Comparing the new Ifakara Ambient Chamber Test with WHO cone and tunnel tests for bioefficacy and non-inferiority testing of insecticide-treated nets

Bioefficacy tests were conducted as part of a 3-year prospective project (the ABCDR-Attrition, Bioefficacy, Chemical residual, Damage and Resistance project) to assess of the useful life of three brands of ITNs in Tanzania [24]. The main design characteristics of each bioassay performed in this study are presented in Table 1. The study involved two experiments. In the first experiment, ITNs efficacy, measured by cone bioassay and I-ACT were compared. In the second experiment, ITNs bioefficacy measured by WHO tunnel test and the I-ACT was compared. The overall pass/fail rate for each net brand by year were examined following the criteria outlined in both 2005 and the 2013 World Health Organization guidelines for evaluation of long lasting nets [14, 25]. The pass rate of each product by standard WHO methods and I-ACT was compared.

Mosquitoes used during testing were laboratory-reared fully pyrethroid susceptible 3–8 days old female adult An. gambiae s.s. (Ifakara strain, Njage 1996) reared following standard methods [26].

All mosquito nets used in this study came from a 3-years prospective longitudinal follow-up study between 2013 and 2016 (ABCDR Project) conducted in eight districts of Tanzania. Net samples were randomly selected using net codes from master list from the three surveys conducted between October and December 2014 (year 1), October–December 2015 (year 2) and October–December 2016 (year 3). The detailed description of the ABCDR Project has been reported previously [24]. Three net brands were used for this study: (1) Olyset^® net (permethrin incorporated into polyethylene fibres @ 1000 mg/m²), (2) PermaNet^® 2.0 net (deltamethrin coated on polyester fibres @ 55 mg/m²) and (3) Netprotect^® net (deltamethrin incorporated into polyethylene fibres @ 63 mg/m²). All nets were rectangular, white, double sized (190 cm length × 180 cm width × 150 cm height) and recommended by WHO [27]. All nets were used in the I-ACT as found i.e. with damage due to wear and tear. In the first experiment, to compare between cone test and I-ACT, a total of 432 nets (144 per net brand) were evaluated and results compared. In the second experiment to compare between tunnel test and I-ACT, nets those failed to meet cone test threshold criteria in the first experiment were assessed using WHO tunnel test (132 nets) and their results were compared with those from I-ACT.

This is a 50 m long, 3 m wide and 2.1 m high steel tube frame construction (Fig. 1a) covered by durable UV resistant polyurethane coated netting with an overlaid polyurethane sheet to minimize wind so that bioassays are conducted in still air (as would occur in a house). The structure is constructed upon a concrete base surrounded by a water channel to prevent entry by ants and spiders that eat mosquitoes during the conduct of experiments. The tunnel sits beneath a simple beamed wooden frame supporting a corrugated steel roof to allow work to continue in all weather conditions. The netted tunnel is divided into ten individual test chambers with interconnecting doors that are sealed by means of zips and Velcro to prevent mosquitoes moving from one test chamber to another. Each compartment contains a white netted chamber 5 m long, 2 m wide, and 2 m high that seals with a zip, in which the ITN is hung from a frame with a human volunteer sleeping underneath (Fig. 1b). At each end of the tunnel is an additional double door module to ensure no loss of laboratory-reared mosquitoes into the wild. Mosquitoes are released from the holding cages within each netted chamber by means of raising a netted cage from its removable wooden base. This is achieved by the technician in situ underneath his net pulling a nylon line attached to the mosquito release cage to elevate it (Fig. 1c). After the allotted experimental time period all mosquitoes within each of the compartments are recovered by mouth aspiration (for mosquitoes inside the net) and by a battery powered Prokopack aspirator (for mosquitoes outside the net but inside the compartment). This allows whole ITNs to be tested in a controlled ambient chamber test with a human host sleeping beneath (Fig. 1d) to measure the protective efficacy (both personal protection measured by feeding inhibition and community protection measured by mosquito mortality) under user conditions. The design of the chambers allows 100% recovery of released mosquitoes that improves precision of the data, and experiments can be conducted year-round.

Each of the ten testing chambers was randomly assigned one whole net (with wear and tear as found after use for 1, 2 or 3 years) from one of the three net brands using a random number generator. Two chambers were used each night as negative control with untreated SAFI Net (A to Z, Tanzania) holed with six holes 4 × 4 cm (Additional file 1) i.e. two holes on each large side and one hole on each small side (hole surface area of 96 cm²) according to WHO guidance [9]. One adult volunteer per chamber slept underneath the nets from 21:00 to 06:30 h and collected mosquitoes in the mornings. Each volunteer was fixed to the same chamber for the duration of the experiment. On each night of experiment, each volunteer hung the tested net on the bednet frame and tucked it underneath the mattress (between 28 and 35 cm of each net was tucked). At 21:00 h, each volunteer released 30 mosquitoes within the chamber but outside of the ITN by opening the mosquito release cage while remaining beneath their test net. The following morning, at 06:30 h, mosquitoes inside the net were collected first using a mouth aspirator and mosquitoes outside the net but within the chamber (floor and walls) were collected using a 6 V battery driven mechanical aspirator (Prokopack). A study supervisor checked the start and finish of the experiment and intermittently spot checked that the volunteers were in position overnight to ensure good conduct of the experiment. All collected mosquitoes were placed in paper cups and scored as dead-fed, alive-fed, dead-unfed, alive-unfed after which mosquitoes were held for 24 h in the laboratory with access to 10% sugar solution at 27 °C ± 2 and 80% ± 10 relative humidity and scored again. After every experimental night, all tested nets were taken out and chambers were aired and bed sheets were washed daily to prevent any carry-over insecticide residue. Each net sample was tested on two consecutive nights (fixed to a chamber and volunteer) to improve the precision of the estimation of performance of each net. Outcome measures were 24 h mortality and blood feeding inhibition. Nets that induced ≥ 90% blood-feeding inhibition and/or ≥ 80% mortality were regarded as meeting WHO efficacy criteria. Data were discarded and test repeated if control mortality exceeded 10% or control blood-feeding success was less than 50%. A Standard Operating Procedure for conducting Ifakara Ambient Chamber Test is provided as an Additional file 2.

Cone tests were conducted following WHO guidelines [8] to determine the bioefficacy of insecticides on sampled netting pieces. For each of the sampled whole nets, after completion of the I-ACT, four 30 cm × 30 cm sub-samples were cut from positions 2, 3, 4 and 5 from each net sample (Additional file 3). Cone bioassays were held at a 60° vertical angle on the netting sub-samples [28]. Anopheles gambiae s.s., aged 3–5 days old, were exposed for 3 min after which they were held for 24 h with access to 10% sugar solution at 27 °C ± 2 and 80% ± 10 relative humidity. The numbers of mosquitoes knocked down after 60 min (KD60) and dead at 24 h post-exposure were recorded. A sub-sampled net that caused ≥ 95% KD60 and/or ≥ 80% 24 h mortality in the cone test was regarded as meeting WHO efficacy criteria. Tests, where control mortality at 24 h exceeded 10%, were discarded and repeated.

WHO tunnel tests were conducted following WHO guidelines [8] to assess the efficacy of netting sub-samples that failed to meet the WHO threshold criteria for cone test (95% KD60 and/or 80% 24 h mortality). The surface area of the sample netting accessible to mosquitoes was approximately 625 cm² (25 × 25 cm) with nine holes cut in the net, each 1 cm in diameter: one at the centre of the square and the other eight holes were equidistant and located 5 cm from the border. The sampled net piece was inserted on a cardboard frame and positioned across the tunnel, one-third of the length of the tunnel. A total of 100 sugar-starved An. gambiae s.s. aged 5–8 days were released in the long section of the glass tunnel at 18:00 h. A rabbit was used as a bait and positioned on the other side of the net so that mosquitoes must pass through the holed net to access the bait and feed. The following morning, between 06:00 and 09:00 h, mosquitoes were removed (separately from each section of the tunnel) using a mouth aspirator, counted, scored as alive or dead, blood fed or unfed after which they were held for 24 h with access to 10% sugar solution at 27 °C ± 2 and 80% ± 10 relative humidity. The main outcome measures were 12 h mortality measured in the morning after the experiment and blood feeding inhibition [14]. The 24 h mortality was also recorded as a secondary outcome [25]. Nets that caused ≥ 90% blood-feeding inhibition and/or ≥ 80% mortality was regarded as meeting WHO efficacy criteria [25]. Tests were discarded if control mortality at 24 h exceeded 10% or control blood-feeding success was less than 50%.

A sample size calculation for generalized linear mixed effects models (GLMMs) through simulation [12] in R statistical software 3.02 http://​www.​r-project.​org [13] was performed for the semi-field experiments to detect a difference between the nets of 5% mortality (half the smallest anticipated effect size). Simulations were performed using an estimated mosquito mortality of 70% with the Olyset^® and 80% for the PermaNet^® 2.0 and NetProtect^®. With 44 replicates tested on two occasions with an inter-observational variance of 0 for the chamber (controlled environment) and 0.1 for individual and 0.1 for the night of observation based on the variance of the random effects observed in a pilot study. Power was estimated at > 90% with a density of 30 mosquitoes per chamber per night using 1000 simulations.

Data were collected on standardized data collection forms and double entered into Microsoft Excel. Data were cleaned and analysed following a predefined analysis plan using STATA 14.1 (Stata Corp., College Station TX, USA) with significance level of ≤ 0.05 for rejecting the null hypothesis. Descriptive statistics were used to present the comparison of proportion of nets passing WHO threshold criteria as measured by each method. Generalized linear mixed models (GLMM) with a binomial error distribution and logit link function were used to analyse the main outcome measures from cone, tunnel and I-ACT (mortality and blood feeding) as well as the proportion of nets passing WHO criteria in order to detect differences between the two evaluation methods. Net brand, age of net and bioassay method were fitted as fixed effects while date was fitted as a random effect to account for repeated testing of individual nets. Several GLMMs were run for each comparison (with interactions) and the final model selected was that with the lowest Akaike’s Information Criterion (AIC). Residuals were plotted using histogram, qnorm plots and comparison with fitted values to ensure appropriateness of the model selection and testing if the residuals are normally distributed. Odds ratio (OR) and 95% confidence interval were calculated for the differences between methods in each comparison.

In addition, non-inferiority between net products (PermaNet^® 2.0, Netprotect^® with Olyset^® as reference/comparator) measured by I-ACT was analysed using a paired t-test with a 90% confidence interval of the observed effect difference in the mortality and bloodfeeding inhibition rates to measure non-inferiority at a margin of 10% and data were presented for comparison using a Forest Plot [29].

The data presented in Table 2 show that a smaller percentage of nets passed WHO threshold criteria using cone test 62% (268/432) than passed using by I-ACT 97% (417/432) irrespective of brand and net age (cone test criteria ≥ 80% 24-h mortality and/or ≥ 95% KD60; I-ACT criteria ≥ 80% 24-h mortality).

Using cone bioassays, 8% (12/145) of Olyset^® nets, 88% (126/144) of PermaNet^® 2.0 nets and 91% (130/143) of NetProtect^® nets passed the cone test. When tested by the I-ACT, 94% (137/145) Olyset^®, 98% (141/144) PermaNet^® 2.0 and 97% (139/143) NetProtect^® passed (Table 2). Table 3 shows that, overall, I-ACT measured higher 24 h mosquito mortality than cone test regardless of net brand (OR: 7.9, 95% CI 7.4–8.4, p < 0.0001). Disaggregated by brand the same trend was evident and I-ACT measured higher mortality than cone test: Olyset^® nets (OR: 17.8, 95% CI 16.3–19.5%; p < 0.0001), PermaNet^® 2.0 (OR: 2.1, 95% CI 1.8–2.3%; p < 0.0001) and Netprotect^® (OR: 3.6, 95% CI 3.2–4.1, p < 0.0001).

Figure 4 shows that even after 3 years of field use PermaNet^® 2.0 and NetProtect^® killed a greater proportion of mosquitoes than Olyset^® resulting in a higher pass rate by WHO (combined cone/tunnel) methods, while there was less contract in the performance of the three products overall when tested using I-ACT. The data (Table 6) show that overall more nets passed WHO threshold criteria using I-ACT than using standard WHO (combined cone/tunnel) methods irrespective of brand and age (OR: 3.5, 95% CI 1.9–6.5, p < 0.0001). The proportion of nets passing using combined WHO methods agreed with I-ACT for NetProtect^® and PermaNet^® 2.0 but differed different for Olyset^® with 94% passing in I-ACT vs 77% by standard bioassays (OR 5.2, 95% CI 2.3–11.8, p < 0.0001). A second notable difference was that holding time was important in determining the pass rate of Olyset^® net with a significant increase in the proportion of Olyset^® that passed when 24 h mortality vs immediate mortality scoring was used: 77% (95% CI 69–83%) vs 71% (95% CI 62–78%) pass. A small and not significant difference was observed in the proportion of PermaNet^® 2.0 (94% vs 98%) and NetProtect^® (98% vs 97%) passing by either WHO methods or I-ACT methods, respectively.

In order to measure non-inferiority of field aged nets Olyset^® was used as the reference net (active comparator for non-inferiority testing) and the other two nets were compared to it. Using I-ACT data it can be seen that overall, PermaNet^® 2.0 and Netprotect^® killed greater proportion of mosquitoes than Olyset^®. Using a t-test of the effect difference between the products using the 24 h mortality endpoint with a margin of 10% of non-inferiority it can be seen that both PermaNet^® 2.0 and NetProtect^® were superior to Olyset^® (Fig. 5). It should be noted that the figure shows a negative value for superior products because the effect difference is calculated by subtracting the induced mortality of the test net from the reference net. However, using the feeding inhibition endpoint, both PermaNet^® 2.0 and NetProtect^® were non-inferior to Olyset using a 10% margin of non-inferiority. It can, therefore, be concluded that the three products are equivalent based on a combined mortality and feeding inhibition endpoints, as is current WHO practice.