Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

External Quality Assurance of Malaria Nucleic Acid Testing for Clinical Trials and Eradication Surveillance

  • Sean C. Murphy ,

    murphysc@uw.edu

    Affiliation Department of Laboratory Medicine and Center for Emerging and Re-Emerging Infectious Diseases, University of Washington (UW), Seattle, Washington, United States of America

  • Cornelus C. Hermsen,

    Affiliation Department of Medical Microbiology, Radboud University Medical Center (RUMC), Nijmegen, The Netherlands

  • Alexander D. Douglas,

    Affiliation The Jenner Institute, University of Oxford, Oxford, United Kingdom

  • Nick J. Edwards,

    Affiliation The Jenner Institute, University of Oxford, Oxford, United Kingdom

  • Ines Petersen,

    Affiliation European Vaccine Initiative (EVI), Heidelberg, Germany

  • Gary A. Fahle,

    Affiliation Department of Laboratory Medicine, Clinical Center, National Institutes of Health (NIH), Bethesda, Maryland, United States of America

  • Matthew Adams,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Andrea A. Berry,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Zachary P. Billman,

    Affiliation Department of Laboratory Medicine and Center for Emerging and Re-Emerging Infectious Diseases, University of Washington (UW), Seattle, Washington, United States of America

  • Sarah C. Gilbert,

    Affiliation The Jenner Institute, University of Oxford, Oxford, United Kingdom

  • Matthew B. Laurens,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Odile Leroy,

    Affiliation European Vaccine Initiative (EVI), Heidelberg, Germany

  • Kristen E. Lyke,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Christopher V. Plowe,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Annette M. Seilie,

    Affiliation Department of Laboratory Medicine and Center for Emerging and Re-Emerging Infectious Diseases, University of Washington (UW), Seattle, Washington, United States of America

  • Kathleen A. Strauss,

    Affiliation Center for Vaccine Development and Howard Hughes Medical Institute, University of Maryland School of Medicine, Baltimore, Maryland, United States of America

  • Karina Teelen,

    Affiliation Department of Medical Microbiology, Radboud University Medical Center (RUMC), Nijmegen, The Netherlands

  • Adrian V. S. Hill,

    Affiliation The Jenner Institute, University of Oxford, Oxford, United Kingdom

  •  [ ... ],
  • Robert W. Sauerwein

    Affiliation Department of Medical Microbiology, Radboud University Medical Center (RUMC), Nijmegen, The Netherlands

  • [ view all ]
  • [ view less ]

Abstract

Nucleic acid testing (NAT) for malaria parasites is an increasingly recommended diagnostic endpoint in clinical trials of vaccine and drug candidates and is also important in surveillance of malaria control and elimination efforts. A variety of reported NAT assays have been described, yet no formal external quality assurance (EQA) program provides validation for the assays in use. Here, we report results of an EQA exercise for malaria NAT assays. Among five centers conducting controlled human malaria infection trials, all centers achieved 100% specificity and demonstrated limits of detection consistent with each laboratory's pre-stated expectations. Quantitative bias of reported results compared to expected results was generally <0.5 log10 parasites/mL except for one laboratory where the EQA effort identified likely reasons for a general quantitative shift. The within-laboratory variation for all assays was low at <10% coefficient of variation across a range of parasite densities. Based on this study, we propose to create a Molecular Malaria Quality Assessment program that fulfills the need for EQA of malaria NAT assays worldwide.

Introduction

During the past two decades, numerous nucleic acid testing (NAT) approaches for the diagnosis of human malaria infection have been developed [1][21]. NAT can detect and quantify parasites more sensitively and precisely than by microscopy or rapid diagnostic tests (RDTs). NAT approaches are valuable for controlled human malaria infection (CHMI) studies of investigational drug and vaccine candidates, for drug efficacy studies and for epidemiological surveillance [22]. In CHMI studies for example, healthy human volunteers are infected via the bites of Plasmodium falciparum-infected mosquitoes [23][25] or by needle-based delivery of purified sporozoites [26], [27]. CHMI is used in initial efficacy studies of investigational drugs and vaccines because of its reproducibility and convenience as compared with efficacy studies in malaria-endemic populations that require larger studies and rely on natural exposure [26], [28][40]. NAT assays allow for quantitative measurement of peripheral parasitemia up to 2–6 days earlier than microscopy [26], [27], [30], [41], further improving the safety of this already very safe model. Depending on the trial design and the laboratory capabilities, samples can be tested either in real time using fresh samples or retrospectively using archived samples. NAT is less operator-dependent and more amenable to high throughput testing than microscopy but is more expensive. The increased cost of NA testing in clinical trials affords improved discrimination between infected and uninfected subjects (e.g., fewer false positives and false negatives) and production of quantitative datasets that can be used for modeling parasite growth. In addition, because earlier detection and therefore earlier treatment of asymptomatic parasitemia decreases both volunteer risk and discomfort, use of NAT assays can facilitate the elimination of the costly but traditional ‘hotel’ phase of many studies where volunteers are housed near study staff for close monitoring. The many advantages and disadvantages of microscopy, RDTs and NAT vary depending on whether the result is used for monitoring of clinical trials, for clinical care in endemic or non-endemic settings or for epidemiological surveillance, as recently reviewed [22].

When testing is performed on the day of collection, NAT results are used in some centers alongside clinical assessments and microscopic findings to inform treatment decisions. This is useful since the symptoms of malaria are non-specific and overlap with common viral illnesses. NAT results can be particularly useful when these traditional measures like microscopy are inconclusive. Despite use of highly-trained slide readers, detection of 1–2 parasites on a thin blood smear is neither 100% sensitive nor specific for the low parasite density capable of causing symptomatic malaria. Some CHMI centers have even replaced primary microscopy-based endpoints with primary NAT endpoints. When NAT was performed in real time and used to make treatment decisions, this approach reduced reported clinical symptoms in CHMI subjects relative to microscopy-based treatment decision making (G. Bastiaens, unpublished data). Thus, the NAT result can help eliminate diagnostic uncertainly that may otherwise occur in subjects who are either ostensibly slide positive or symptomatic but not both. Overall, real time availability of NAT apparently helps avoid false positive and false negative diagnoses, thereby increasing both the safety of the volunteers and the accuracy of the data. Aside from lower limits of detection (LoD) that afford earlier diagnoses, quantitative NAT data provide day-by-day measures of the rise and fall of parasitemia and allow for model-based assessments of liver-to-blood inocula levels and parasite multiplication rates, which can be used to calculate efficacy estimates for partially-effective liver vaccines and for blood-stage vaccines [42], [43].

Quality systems are critical to clinical trial and surveillance networks because the validity of diagnostic and monitoring tests and the ability to compare trial results amongst network laboratories is entirely dependent on the procedures used before, during and after each assay at each site [22]. Consistently dependable results are provided when the overall program includes quality control (QC), quality assurance (QA) and proficiency testing (PT) [44]. Malaria QA systems have been developed to address the wide performance variations observed in first generation RDTs [45]. These efforts resulted in consistently improved RDT performance with each successive round of evaluations and led to elimination of poor RDT products. External QA (EQA) programs and control reagents for RDTs are also emerging [46][48]. Terminology related to quality management and assay performance is defined in the Glossary section.

Because of the availability of cryopreserved metabolically-active, non-replicating sporozoites [49] and the renewed vigor of vaccine and drug pipelines, CHMI studies are now being conducted at an increasing number of institutions worldwide [50]. However, while consensus procedures for CHMI studies and microscopy are available [51], no such effort has been made to standardize malaria NAT assays or provide widespread ongoing EQA oversight. Some laboratories have exchanged small panels of malaria-infected whole blood with collaborators as part of NAT assay validation (S. Murphy, C. Hermsen, N. Edwards, A. Stewart, unpublished data). In addition, the World Health Organization (WHO) previously generated freeze-dried P. falciparum-infected blood samples at a single high parasitemia level for use as an international DNA standard [52]. Some laboratories are using the WHO material (for example see reference [53]), but the WHO material is not provided through a formal EQA program. Because the material was freeze-dried from whole blood without buffers or steps to protect RNA, it also cannot be used to support RNA-based assays, and the freeze-dried material does not precisely mimic material obtained from clinical trial participants. Therefore, a formalized, funded program is needed to ensure assay validation and provide malaria NAT EQA in line with that commonly used for assays detecting HIV and other infectious pathogens [54][56]. As a trial EQA platform, we sent blinded specimens to five malaria centers conducting CHMI studies. Here, we report the results of this exercise and propose a framework for ongoing EQA and eventually for assay harmonization.

Materials and Methods

Malaria culture and production of samples

P. falciparum strain 3D7 was cultured, synchronized and diluted as previously reported [41]. For multiply-infected cells, each parasite was counted in microscopic parasite density measurements. A ring-stage synchronous high parasitemia culture was diluted into type A+ whole human blood obtained from the Puget Sound Blood Center (www.psbc.org). After preparing the ‘master’ tube at each density, samples were aliquoted into bar code-labeled tubes [each bearing a unique specimen identifier generated in the Laboratory Data Management System (Frontier Science)] and prepared for frozen storage according to the individual standard operating procedures for each final testing laboratory. Aliquot sizes were as follows: RUMC and University of Maryland 0.5 mL; NIH 0.2 mL into 2 mL NucliSENS lysis buffer (bioMérieux); University of Washington (UW) 0.05 mL into 2 mL NucliSENS lysis buffer; Oxford filtered to remove leukocytes as described [57] and aliquoted as 0.5 mL volumes. Once aliquoted, all samples were frozen at −80°C before courier shipments to partner laboratories on dry ice.

Sample testing

Laboratories received and stored samples at ≤−70°C before testing. Each lab except for the NIH received 10 de-identified parasite-containing samples at each of five different concentrations plus 10 parasite-negative samples; the NIH received 20 samples at each level. Labs were blinded to the parasite concentration in each sample. Each laboratory tested their designated samples according to the laboratory-specific standard operating procedure (SOP) and reported data to the UW coordinating center. Testing laboratories (including the technologists at the UW coordinating laboratory) were blinded to the nominal parasite density of each sample. Assays used included P. falciparum quantitative reverse transcription polymerase chain reaction (qRT-PCR; UW, [58]); quantitative PCR (qPCR; RUMC, [21], [59]; Oxford [26]; University of Maryland [35], [37]) and standard PCR (NIH [34]). If known, laboratories indicated their in-house determined limits of detection (LoD) and nucleic acid target characteristics (Table 1). All laboratories reported quantitative data, except the NIH which reported qualitative results and cycle thresholds (CT) from standard PCR. Each laboratory was asked to run the assay and provide data strictly in accordance with the SOP used in their clinical trials.

thumbnail
Table 1. Reported characteristics and use of network assays.

https://doi.org/10.1371/journal.pone.0097398.t001

Filtration studies (Oxford only)

Ring-stage cultured 3D7 strain P. falciparum parasites were added to leukocyte-depleted [26] whole blood. Each sample was subsequently divided and half of the material at each density was subjected to Whatman VFE filtration while half remained unfiltered. The material was then subjected to DNA extraction and qPCR by the standard Oxford protocol [26].

Calibrator matrix studies (Oxford only)

DNA was extracted from blood from malaria-negative volunteers using the standard Oxford protocol [26] to generate ‘negative blood matrix' samples. qPCR was performed on 150 copies of the Oxford plasmid DNA calibrator in the presence of a blood matrix sample or a water control (20% v/v) and differences in the CT and quantity were evaluated.

Data analysis

Data were transformed to log10 parasites/mL of whole blood and analyzed using Excel 2010 (Microsoft) and Prism 6 (GraphPad). Intra-laboratory performance was evaluated using sensitivity/specificity analyses, precision analyses and Bland-Altman (difference) plots. The lowest parasite density samples (6 parasites/mL) were not included in statistical analyses. Data were plotted on a log10 parasites/mL scale; data from the NIH was regressed to the nominal values of the provided samples and plotted for illustration purposes. For precision studies, the percent coefficient of variation was calculated as %CV  =  standard deviation/mean.

Results

Sample production

A high parasitemia starting culture was prepared to 2.4×108 (8.38 log10) parasites/mL based on repeated counts of Giemsa-stained thin blood smears by multiple readers combined with determination of RBC density (RBC/mL) by hemocytometer counting. This material was devoid of trophozoite- and schizont-stage parasites and contained singly (70%) and multiply-infected (30%) ring-stage parasites as judged by microscopy. Dilutions were made into whole blood to nominal parasite densities as follows: High (300,000 parasites/mL or 5.48 log10 parasites/mL); Mid (6,000 parasites/mL or 3.78 log10 parasites/mL); Low (600 parasites/mL or 2.78 log10 parasites/mL); Very Low (60 parasites/mL or 1.78 log10 parasites/mL), Trace (6 parasites/mL or 0.78 log10 parasites/mL) and Negative (no parasites); these designations are used throughout the paper to refer to these specific parasite densities.

Network laboratory results

Laboratories reported the results shown in the Figure 1. The sensitivity and specificity of each laboratory's assay was calculated based on the negative control samples plus either all samples ≥60 parasites/mL or those ≥600 parasites/mL; samples at 6 parasites/mL were not included in these analyses since this ultra-low level of parasite density was below the stated LoD for all assays (Table 2). All laboratories demonstrated 100% specificity. Most laboratories detected all samples at ≥600 parasites/mL, although the University of Maryland reported one false negative at this level.

thumbnail
Figure 1. Study data.

Results were plotted on a log10 parasites/mL scale for the five participating laboratories; bars show the mean and 95% confidence interval. Nominal (expected) values for all samples are plotted as follows: high (300,000 parasites/mL or 5.48 log10 parasites/mL); mid (6,000 parasites/mL or 3.78 log10 parasites/mL); low (600 parasites/mL or 2.78 log10 parasites/mL); very low (60 parasites/mL or 1.78 log10 parasites/mL), trace (6 parasites/mL or 0.78 log10 parasites/mL) and negative (no parasites). Samples with no parasites detected were plotted as 0.1 log10 parasites/mL. Two-way ANOVA comparisons across all high, mid and low parasite density samples with quantitatively positive results showed non-statistically significant differences amongst all groups (p>0.05) except for RUMC vs. Oxford at high (p<0.0001), mid (p<0.01) and low (p≤0.05) parasite densities. *NIH quantities were generated by regression of CT values to expected EQA values and are provided to visualize variation and qualitative agreement; quantitative statistical comparisons were not included. 60 of 120 representative NIH samples are displayed for consistency.

https://doi.org/10.1371/journal.pone.0097398.g001

Analytical sensitivity was below 100% at the 60 parasites/mL level: UW detected 6/10 samples, Oxford 4/10, NIH 6/20, Maryland 4/10 and RUMC 2/10. One positive sample was detected by the UW laboratory amongst the 6 parasites/mL samples at a concentration consistent with the presence of a single parasite in the sample and consistent with the overall frequency of positives at the 60 parasites/mL level as well. The ‘trace’ (6 parasites/mL) specimens were not included in estimates of sensitivity and specificity because they cannot be categorically considered positive for the following assay-specific reasons. For the low volume RT-PCR assay at UW, most of these samples are negative because a density of 6 parasites/mL equates to less than one parasite per sample on average. Nonetheless, when samples at this level are positive by RT-PCR due to an intact parasite, the parasite contains ∼3500 copies of the targeted 18S rRNA. For higher volume DNA assays, the lack of positive results at the 6 parasites/mL level reflects a combination of similar Poisson distribution limitations (e.g., 6 parasites/mL is an average of 1.2 parasites/sample for 200 µL samples or 3 parasites/sample for 500 µL samples) as well as limitations on PCR performance at low copy numbers (e.g., 2–10 DNA copies/sample). Thus, the virtual absence of positive results in samples at 6 parasites/mL was expected, indicates that positive results are due to intact parasites and not free template and supports the high specificity shown by all laboratories. False negative results at the 60 parasites/mL level can be attributed to the presence of multiply-infected RBCs that result in a more heterogeneous distribution of parasites upon dilution to low parasitemia than would be expected if virtually no multiply-infected cells were present. A related type of heterogeneity has also reported for malaria microscopy – in thick blood smears, malaria parasites are sometimes unevenly distributed (known as ‘grouping’) [60]. At low parasite densities, grouping results in a few microscopic fields containing more parasites than expected while most fields contain fewer or zero parasites, leading to overdispersion of the Poisson distribution [61]. Much like ‘grouping’ in thick blood smears, multiply-infected cells contain a larger-than-expected proportion of the NAT target(s), thereby altering the frequency of parasite-containing samples at low parasitemias. Nonetheless, the overall data indicate agreement between laboratories across a wide range of parasite densities, with gradual loss of positivity as predicted by assay LoDs.

Differences from the expected values (quantitative bias) were assessed using correlation and Bland-Altman (difference) plots, which are summarized in Table 3. In general, all assays behaved linearly (slope ∼1.0 and r2>0.9), indicating comparable template amplification efficiencies. The average quantitative bias across all laboratories was <±0.18 log10 parasites/mL compared to expected values, with the exception of the Oxford qPCR. In addition, most laboratories maintained a 95% confidence range within ±0.5 log10 parasites/mL compared to the expected values with the exception of Maryland whose assay had lower limit of the 95% confidence interval extending to −0.66 log10 parasites/mL. The Oxford qPCR showed an average difference of −0.54 log10 parasites/mL (95%CI −0.90 to −0.19 log10 parasites/mL), thereby underestimating relative to expected values and other laboratories. This difference was consistent across the range of parasite densities for Oxford and the confidence interval for the Oxford qPCR was within ±0.5 log10 parasites/mL of the average Oxford values.

thumbnail
Table 3. Correlation and agreement between assay-derived data and expected values.

https://doi.org/10.1371/journal.pone.0097398.t003

Precision (%CV) was evaluated for each laboratory's assay for the high, mid and low parasite density samples (Table 4). As expected, the highest parasite density samples showed the lowest degree of variation and, with the exception of the Maryland qPCR, the lowest parasite density samples had the highest variation. Variation was generally <10%CV at all levels. This level of precision is considered acceptable by most validation criteria and is likely to be more than adequate for purposes of modeling parasite growth dynamics [42], [43]. Very low parasite density samples were not included in this analysis because of the smaller number of positive samples.

Root cause analyses of differences

Two of the network groups further investigated apparently aberrant results to determine if the root cause could be identified. The single false negative 600 parasites/mL sample at the University of Maryland was found to be qualitatively positive, but at a concentration below the quantitative LoD established for that assay. Root cause analysis showed that the DNA extraction control used to monitor extraction efficiency was lower than the expected value, indicating that poor sample extraction was at fault - re-extraction of the specific sample was not possible since the material was completely consumed in the initial testing.

Additional studies were conducted by Oxford University to estimate the contribution of blood filtering and calibrator matrix differences to the observed quantitative shift. To test the hypothesis that parasites may be lost during blood filtration, parasites diluted in pre-leukocyte-depleted blood were tested by Oxford qPCR either with or without filtration. Filtration reduced the measured parasite density by 13–57% (−0.06 to −0.37 log10-fold) (Figure S1). To determine if the matrix used to dilute the plasmid DNA calibrators at Oxford also contributed to the quantitative shift, DNA extracted from malaria-negative volunteers was used in lieu of water to dilute the plasmid DNA calibrator in the qPCR reaction. One-hundred fifty copies of plasmid DNA were added since this corresponds to the amount of plasmid DNA equal to 1000 parasites/mL by the Oxford qPCR assay. In the presence of the negative blood matrix, the Oxford qPCR CT was delayed by a median of 1.1 cycles as compared to the previously used water matrix – this difference was consistently observed using blood from three different donors (range 1.1–1.4 cycles) (Figure S1). This CT shift corresponds to a 58–65% reduction in the apparent parasite density (median −0.39 log10-fold, range −0.37 to −0.46 log10-fold) relative to amplification using a water-only diluent reduction (Figure S1).

Discussion

This study represents the first major malaria NAT EQA exercise to our knowledge amongst CHMI centers providing assay-to-assay comparisons that attempt to fully account for all variables contributing to assay performance. The data indicate that CHMI centers listed here are reporting comparable results. CHMI studies seek to ensure the safety of the trial participants and the integrity of the trial data. In this regard, all centers achieved expected analytical sensitivities based on known LoDs. Based on clinical trial data and supported by modeling studies [41][43], the assays compared here are likely to become positive before patent parasitemia develops following CHMI with five mosquito bites. No false positive results were reported for any of the 10–20 blinded malaria-negative samples sent to each laboratory. Quantitative variation was <10%CV amongst the high, mid and low parasite density samples. The average quantitative bias across all laboratories was <±0.18 log10 parasites/mL compared to expected values, and all laboratories except for Oxford showed a 95% confidence range within ±0.5 log10 parasites/mL compared to the expected values.

The Oxford assay was highly sensitive, specific and precise, but was quantitatively shifted compared to other assays and to expected results. As described in the Results, additional studies conducted by Oxford University determined that both parasite losses due to Whatman VFE filtering of whole blood and differences in the matrix used for plasmid DNA calibrators at Oxford contributed to this shift. This is the first report to our knowledge that demonstrates parasite losses due to the filtering step. This step was added at Oxford to remove co-purified leukocyte genomic DNA (gDNA), which may inhibit some PCR assays [62]. Thus, while filtration removes inhibitors like gDNA, it also removes parasites. In addition, the Oxford plasmid calibrators are normally diluted in a water matrix, which lacks additional PCR inhibitors common to whole blood extractions (e.g., immunoglobulin G, hemoglobin and lactoferrin [63], [64]). Since the water-diluted plasmid calibrator was detected earlier than when diluted in a whole blood matrix, it appears that PCR inhibitors present in the eluates obtained from blood delay PCR target amplification and also contribute to the quantitative reduction in parasites when a water-diluted standard curve was used. This type of matrix effect has been reported in biological samples [65]. Despite the shift in absolute quantitation, relative quantitation between samples using the Oxford assay was comparable to that of other centers. The Oxford qPCR is reporting positive and negative results in complete agreement with other laboratories, even at the low 60 parasites/mL level suggesting that the combined effects of blood filtration and the different calibrator matrix did not markedly change the overall qualitative results. Nonetheless, re-calibration of the Oxford qPCR method that will account for these differences is underway. This EQA effort helped to identify an easily fixed methodological difference in calibration that may account for differences in absolute quantification, and work is underway at Oxford to rectify this shift. Overall, the results of most laboratories were within the expected quantitative range, and these findings should be reassuring to vaccine and drug makers, sponsors and regulators as NAT gradually replaces microscopy for safety and efficacy endpoints.

With the plethora of malaria NAT methods reported in the literature, quality indicators are needed to help select and maintain methods suitable for use in CHMI studies. However, malaria NAT EQA and consensus malaria NAT guidelines do not yet exist in part because over 65 different assays have been reported for NAT-based malaria diagnosis. The literature includes many variations on extraction, amplification and detection, including single-step and nested electrophoresis-based PCR, qPCR, qRT-PCR and nucleic acid-based sequence amplification (NASBA) with an even wider variety of reported primer and probe combinations (briefly reviewed in [22]). Additional novel assays continue to be developed, including potentially field-friendly approaches such as loop-mediated isothermal amplification [66]. While numerous other gene targets are reported, the most widely used Plasmodium NAT targets are sequences within the developmentally-regulated 18S rRNAs (by RT-PCR or NASBA) or their coding genes (by PCR) [67]. However, 18S rRNAs and their corresponding coding genes are not uniformly captured by a single set of reagents (primers and probe), and assay design can greatly alter sensitivity and specificity as recently reviewed [22]. RT-PCR tests target A-type (asexual stage) 18S rRNAs to take advantage of the fact that A-type (but not sexual-stage S-type) 18S rRNAs are biologically amplified to ∼3500 copies per ring-stage parasite during the red blood cell stage of infection [41]. PCR tests can target any of the 18S rRNA-coding genes, and some primers and/or probes are designed to capture sequences shared by more than one of the genes (up to five including one pseudogene) thereby incrementally increasing assay sensitivity. Thus, RT-PCR assays become positive earlier in the amplification process (lower cycle threshold) compared to PCR, and this can improve RT-PCR sensitivity particularly with small sample volumes and/or extremely low parasite densities (S. Murphy, unpublished data). While rRNA copies are more abundant than the parent genes, samples for RT-PCR must be preserved by adding a stabilizing buffer or by making dried blood spots at the time of collection [58], whereas samples for PCR can be simply frozen. These aspects of malaria NAT have mostly been studied using laboratory strains of P. falciparum, and future work will need to address assays designed to test for other human Plasmodium spp. since species and sub-species diversity in field settings affects assay performance [68], [69].

Thus, because of the diversity of pre-analytical processing, extraction and amplification techniques, a single EQA sample type cannot currently be used in all assays. EQA efforts must therefore account for these methodological variations in pre-analytical (collection, stabilization) and analytical (extraction, amplification) steps.

To support clinical trials using malaria NAT, a formal EQA program is needed. Some previous efforts have been made to improve the quality of malaria NAT, although none provide formal EQA. As mentioned above, the WHO previously developed and distributed its single concentration standard to many laboratories for characterization purposes [52], however, this standard is unsuitable for RNA-based assays and does not precisely mimic clinical samples. Another group compared PCR primers and probes used by several centers but it was not possible for the investigators to perform all the pre-analytical, analytical (extraction, amplification and detection) and post-analytical steps for each assay as originally described and therefore the study did not assess overall assay performance [53]. Most recently, a study of field samples in Brazil compared two conventional PCR methods against microscopy and concluded that the PCR protocols showed low reproducibility at sub-microscopic densities [70]. In contrast, the groups tested in our EQA study generated reproducible data at sub-microscopic densities (e.g., 600 parasites/mL), but our study was performed using laboratory-generated samples so this may change when field samples are used. True EQA comparisons, such as the work described herein, are needed to fully account for all factors that lead to assay variability and ensure that high quality assays are in place to support clinical and field studies.

To continue our work and fulfill the need for an ongoing EQA program, we propose creation of the Molecular Malaria Quality Assessment (MolMalQA) Program as a multi-lateral effort involving one or more core reference laboratories and a larger network of partner laboratories at centers performing malaria clinical trials, drug efficacy testing and surveillance activities. The core laboratories would provide malaria EQA samples, produce an international calibrator suitable for use in DNA and RNA assays and pursue harmonization activities in consultation with network partners. The malaria effort will be somewhat more complicated than EQA programs for many viral pathogens because of the complexity of the parasite lifecycle and the diversity of available tests. At a minimum, it will be necessary to provide EQA samples for DNA- and RNA-based assays alike and to support both liquid and dried blood spot formats. For global EQA, we propose to use synchronized, cultured ring-stage malaria parasites diluted into whole blood since this mimics clinical samples. The EQA program will need to develop whole blood panels derived from a spectrum of healthy human subjects with diverse blood types to ensure that results are not skewed by a single blood source. In future, gametocytes, the sexual malaria stage parasite found in humans, can be provided as EQA samples if needed. The EQA program can also undertake stability studies to assess the collection, transport and long-term storage stability of samples intended for malaria DNA and RNA testing to determine if the relatively greater stability of DNA outweighs the increased abundance of RNA for use in large field studies.

Eventually, assays in use should also be harmonized around a single or very few high-quality diagnostic protocols. Selection of such assays could be made by reviewing EQA data and studying protocols and instrument requirements in consultation with the network of CHMI centers.

In addition to ensuring valid diagnostic data in clinical trials, an EQA program would also aid in NAT surveillance for the malaria elimination/eradication agenda. The relevance of malaria NAT EQA for surveillance efforts should not be overlooked because global success in reducing malaria incidence will reach a point at which control and eradication decisions will require monitoring of infection among individuals harboring parasites at densities beneath the LoD of both microscopy and RDTs [71].

To more closely align quantitative results across centers and eliminate lot-to-lot variation in cultured parasites, we further advocate for development of synthetic nucleic acid sequences diluted in whole blood for use as absolute calibrators. Such reagents allow determination of exact target copy numbers, which can be translated to parasite densities by testing parasite-containing controls against a calibrator standard curve. However, since the 18S rRNA targets vary between testing centers, no single naturally-occurring target sequence is shared by all laboratories. Some centers target the asexual-type 18S rRNAs or the coding genes while others target the genes encoding the sexual-type 18S rRNAs. Thus, cloning just one of these genes fails to capture other laboratories' targets. Novel synthetic sequences are therefore needed to provide cross-network calibrators suitable for more than one assay - such materials are in development (S. Murphy, unpublished data).

In summary, malaria NAT EQA will help safeguard the reliability and comparability of data produced in clinical trials by CHMI centers and will support future extended use of malaria NATs in other contexts. Through collaboration and with multi-lateral funding, a formal EQA program can be developed and implemented worldwide for the benefit of the malaria field and those it serves.

Supporting Information

Figure S1.

Parasite loss due to filtering and calibrator matrix differences contribute to the quantitative shift in Oxford qPCR. A: Filtration using the Whatman VFE plate results in loss of parasites. Cultured parasites were combined with leukocyte-depleted blood and then filtered or not as indicated in the figure. Results of Oxford qPCR are shown; each point represents the mean of triplicate PCR wells. B-C: Oxford qPCR in a whole blood matrix results in delayed CT and lower apparent parasite density than when a water matrix is used. Eluates from leukocyte-depleted blood or a water control were added to qPCR reactions containing 150 copies of a plasmid DNA calibrator. Panel B depicts CT values. Panel C shows the apparent parasite density; the horizontal dashed line represents the result in the presence of water-only diluent (150 plasmid copies/reaction  = 1000 parasites/mL in the Oxford assay by definition). Each point in B-C represents the result in an individual PCR well.

https://doi.org/10.1371/journal.pone.0097398.s001

(TIF)

Acknowledgments

The authors thank the UW Center for AIDS Research (CFAR, AI27757) Clinical Research and Retrovirology Core (Ming Chang, Glenda Daza, Chang Liu, Kenneth Dixon and Robert Coombs) and the UW CFAR Biometrics Core (Ken Tapia) for assistance. These data were presented in part at a meeting of the European Vaccine Initiative in Heidelberg, Germany on December 4, 2013.

Author Contributions

Conceived and designed the experiments: SCM CCH ADD NJE IP OL. Performed the experiments: SCM CCH NJE GAF MA AAB ZPB AMS KAS KT. Analyzed the data: SCM CCH ADD NJE GAF MA AAB SCG MBL CVP AVSH RWS. Contributed reagents/materials/analysis tools: SCM ZPB AMS. Wrote the paper: SCM CCH ADD IP GAF MA AAB ZPB SCG MBL OL KEL CVP AVSH RWS. Arranged international meeting to discuss findings: IP OL.

References

  1. 1. Ciceron L, Jaureguiberry G, Gay F, Danis M (1999) Development of a Plasmodium PCR for monitoring efficacy of antimalarial treatment. J Clin Microbiol 37: 35–38.
  2. 2. Coleman RE, Sattabongkot J, Promstaporm S, Maneechai N, Tippayachai B, et al. (2006) Comparison of PCR and microscopy for the detection of asymptomatic malaria in a Plasmodium falciparum/vivax endemic area in Thailand. Malar J 5: 121.
  3. 3. Das A, Holloway B, Collins WE, Shama VP, Ghosh SK, et al. (1995) Species-specific 18S rRNA gene amplification for the detection of P. falciparum and P. vivax malaria parasites. Mol Cell Probes 9: 161–165.
  4. 4. Han ET, Watanabe R, Sattabongkot J, Khuntirat B, Sirichaisinthop J, et al. (2007) Detection of four Plasmodium species by genus- and species-specific loop-mediated isothermal amplification for clinical diagnosis. J Clin Microbiol 45: 2521–2528.
  5. 5. Kho WG, Chung JY, Sim EJ, Kim MY, Kim DW, et al. (2003) A multiplex polymerase chain reaction for a differential diagnosis of Plasmodium falciparum and Plasmodium vivax. Parasitol Int 52: 229–236.
  6. 6. Mangold KA, Manson RU, Koay ES, Stephens L, Regner M, et al. (2005) Real-time PCR for detection and identification of Plasmodium spp. J Clin Microbiol 43: 2435–2440.
  7. 7. Mens PF, Schoone GJ, Kager PA, Schallig HD (2006) Detection and identification of human Plasmodium species with real-time quantitative nucleic acid sequence-based amplification. Malar J 5: 80.
  8. 8. Mixson-Hayden T, Lucchi NW, Udhayakumar V (2010) Evaluation of three PCR-based diagnostic assays for detecting mixed Plasmodium infection. BMC Res Notes 3: 88.
  9. 9. Perandin F, Manca N, Calderaro A, Piccolo G, Galati L, et al. (2004) Development of a real-time PCR assay for detection of Plasmodium falciparum, Plasmodium vivax, and Plasmodium ovale for routine clinical diagnosis. J Clin Microbiol 42: 1214–1219.
  10. 10. Rougemont M, Van Saanen M, Sahli R, Hinrikson HP, Bille J, et al. (2004) Detection of four Plasmodium species in blood from humans by 18S rRNA gene subunit-based and species-specific real-time PCR assays. J Clin Microbiol 42: 5636–5643.
  11. 11. Safeukui I, Millet P, Boucher S, Melinard L, Fregeville F, et al. (2008) Evaluation of FRET real-time PCR assay for rapid detection and differentiation of Plasmodium species in returning travellers and migrants. Malar J 7: 70.
  12. 12. Schneider P, Wolters L, Schoone G, Schallig H, Sillekens P, et al. (2005) Real-time nucleic acid sequence-based amplification is more convenient than real-time PCR for quantification of Plasmodium falciparum. J Clin Microbiol 43: 402–405.
  13. 13. Schoone GJ, Oskam L, Kroon NC, Schallig HD, Omar SA (2000) Detection and quantification of Plasmodium falciparum in blood samples using quantitative nucleic acid sequence-based amplification. J Clin Microbiol 38: 4072–4075.
  14. 14. Singh B, Bobogare A, Cox-Singh J, Snounou G, Abdullah MS, et al. (1999) A genus- and species-specific nested polymerase chain reaction malaria detection assay for epidemiologic studies. Am J Trop Med Hyg 60: 687–692.
  15. 15. Singh B, Kim Sung L, Matusop A, Radhakrishnan A, Shamsul SS, et al. (2004) A large focus of naturally acquired Plasmodium knowlesi infections in human beings. Lancet 363: 1017–1024.
  16. 16. Snounou G, Singh B (2002) Nested PCR analysis of Plasmodium parasites. Methods Mol Med 72: 189–203.
  17. 17. Snounou G (2002) Genotyping of Plasmodium spp. Nested PCR. Methods Mol Med 72: 103–116.
  18. 18. Swan H, Sloan L, Muyombwe A, Chavalitshewinkoon-Petmitr P, Krudsood S, et al. (2005) Evaluation of a real-time polymerase chain reaction assay for the diagnosis of malaria in patients from Thailand. Am J Trop Med Hyg 73: 850–854.
  19. 19. Tangin A, Komichi Y, Wagatsuma Y, Rashidul H, Wataya Y, et al. (2008) Detection of malaria parasites in mosquitoes from the malaria-endemic area of Chakaria, Bangladesh. Biol Pharm Bull 31: 703–708.
  20. 20. Vo TK, Bigot P, Gazin P, Sinou V, De Pina JJ, et al. (2007) Evaluation of a real-time PCR assay for malaria diagnosis in patients from Vietnam and in returned travellers. Trans R Soc Trop Med Hyg 101: 422–428.
  21. 21. Hermsen CC, Telgt DS, Linders EH, van de Locht LA, Eling WM, et al. (2001) Detection of Plasmodium falciparum malaria parasites in vivo by real-time quantitative PCR. Mol Biochem Parasitol 118: 247–251.
  22. 22. Murphy SC, Shott JP, Parikh S, Etter P, Prescott WR, et al. (2013) Malaria diagnostics in clinical trials. Am J Trop Med Hyg 89: 824–839.
  23. 23. Epstein JE, Rao S, Williams F, Freilich D, Luke T, et al. (2007) Safety and clinical outcome of experimental challenge of human volunteers with Plasmodium falciparum-infected mosquitoes: an update. J Infect Dis 196: 145–154.
  24. 24. Roestenberg M, McCall M, Hopman J, Wiersma J, Luty AJ, et al. (2009) Protection against a malaria challenge by sporozoite inoculation. N Engl J Med 361: 468–477.
  25. 25. Bijker EM, Bastiaens GJ, Teirlinck AC, van Gemert GJ, Graumans W, et al. (2013) Protection against malaria after immunization by chloroquine prophylaxis and sporozoites is mediated by preerythrocytic immunity. Proc Natl Acad Sci U S A 110: 7862–7867.
  26. 26. Sheehy SH, Spencer AJ, Douglas AD, Sim BK, Longley RJ, et al. (2013) Optimising Controlled Human Malaria Infection Studies Using Cryopreserved Parasites Administered by Needle and Syringe. PLoS One 8: e65960.
  27. 27. Roestenberg M, Bijker EM, Sim BK, Billingsley PF, James ER, et al. (2013) Controlled human malaria infections by intradermal injection of cryopreserved Plasmodium falciparum sporozoites. Am J Trop Med Hyg 88: 5–13.
  28. 28. McCarthy JS, Sekuloski S, Griffin PM, Elliott S, Douglas N, et al. (2011) A pilot randomised trial of induced blood-stage Plasmodium falciparum infections in healthy volunteers for testing efficacy of new antimalarial drugs. PLoS One 6: e21914.
  29. 29. Andrews L, Andersen RF, Webster D, Dunachie S, Walther RM, et al. (2005) Quantitative real-time polymerase chain reaction for malaria diagnosis and its use in malaria vaccine clinical trials. The American journal of tropical medicine and hygiene 73: 191–198.
  30. 30. Roestenberg M, O'Hara GA, Duncan CJ, Epstein JE, Edwards NJ, et al. (2012) Comparison of clinical and parasitological data from controlled human malaria infection trials. PLoS One 7: e38434.
  31. 31. Nyunt MM, Hendrix CW, Bakshi RP, Kumar N, Shapiro TA (2009) Phase I/II evaluation of the prophylactic antimalarial activity of pafuramidine in healthy volunteers challenged with Plasmodium falciparum sporozoites. Am J Trop Med Hyg 80: 528–535.
  32. 32. Teirlinck AC, Roestenberg M, van de Vegte-Bolmer M, Scholzen A, Heinrichs MJ, et al. (2013) NF135.C10: a new Plasmodium falciparum clone for controlled human malaria infections. J Infect Dis 207: 656–660.
  33. 33. Genton B, D'Acremont V, Lurati-Ruiz F, Verhage D, Audran R, et al. (2010) Randomized double-blind controlled Phase I/IIa trial to assess the efficacy of malaria vaccine PfCS102 to protect against challenge with P. falciparum. Vaccine 28: 6573–6580.
  34. 34. Seder RA, Chang LJ, Enama ME, Zephir KL, Sarwar UN, et al. (2013) Protection against malaria by intravenous immunization with a nonreplicating sporozoite vaccine. Science 341: 1359–1365.
  35. 35. Laurens MB, Billingsley P, Richman A, Eappen AG, Adams M, et al. (2013) Successful human infection with P. falciparum using three aseptic Anopheles stephensi mosquitoes: a new model for controlled human malaria infection. PLoS One 8: e68969.
  36. 36. Richie TL, Charoenvit Y, Wang R, Epstein JE, Hedstrom RC, et al. (2012) Clinical trial in healthy malaria-naive adults to evaluate the safety, tolerability, immunogenicity and efficacy of MuStDO5, a five-gene, sporozoite/hepatic stage Plasmodium falciparum DNA vaccine combined with escalating dose human GM-CSF DNA. Hum Vaccin Immunother 8: 1564–1584.
  37. 37. Lyke KE, Laurens M, Adams M, Billingsley PF, Richman A, et al. (2010) Plasmodium falciparum malaria challenge by the bite of aseptic Anopheles stephensi mosquitoes: results of a randomized infectivity trial. PLoS One 5: e13490.
  38. 38. Ewer KJ, O'Hara GA, Duncan CJ, Collins KA, Sheehy SH, et al. (2013) Protective CD8+ T-cell immunity to human malaria induced by chimpanzee adenovirus-MVA immunisation. Nat Commun 4: 2836.
  39. 39. Sheehy SH, Duncan CJ, Elias SC, Choudhary P, Biswas S, et al. (2012) ChAd63-MVA-vectored blood-stage malaria vaccines targeting MSP1 and AMA1: assessment of efficacy against mosquito bite challenge in humans. Mol Ther 20: 2355–2368.
  40. 40. Duncan CJ, Sheehy SH, Ewer KJ, Douglas AD, Collins KA, et al. (2011) Impact on malaria parasite multiplication rates in infected volunteers of the protein-in-adjuvant vaccine AMA1-C1/Alhydrogel+CPG 7909. PLoS One 6: e22271.
  41. 41. Murphy SC, Prentice JL, Williamson K, Wallis CK, Fang FC, et al. (2012) Real-time quantitative reverse transcription PCR for monitoring of blood-stage Plasmodium falciparum infections in malaria human challenge trials. Am J Trop Med Hyg 86: 383–394.
  42. 42. Douglas AD, Edwards NJ, Duncan CJ, Thompson FM, Sheehy SH, et al. (2013) Comparison of modeling methods to determine liver-to-blood inocula and parasite multiplication rates during controlled human malaria infection. J Infect Dis 208: 340–345.
  43. 43. Hermsen CC, de Vlas SJ, van Gemert GJ, Telgt DS, Verhage DF, et al. (2004) Testing vaccines in human experimental malaria: statistical analysis of parasitemia measured by a quantitative real-time polymerase chain reaction. Am J Trop Med Hyg 71: 196–201.
  44. 44. Wallace PS, MacKay WG (2013) Quality in the molecular microbiology laboratory. Methods Mol Biol 943: 49–79.
  45. 45. World Health Organization; Foundation for Innovative New Diagnostics; TDR (2012) RDT Evaluation Programme.
  46. 46. Chinkhumba J, Skarbinski J, Chilima B, Campbell C, Ewing V, et al. (2010) Comparative field performance and adherence to test results of four malaria rapid diagnostic tests among febrile patients more than five years of age in Blantyre, Malawi. Malar J 9: 209.
  47. 47. McMorrow ML, Masanja MI, Kahigwa E, Abdulla SM, Kachur SP (2010) Quality assurance of rapid diagnostic tests for malaria in routine patient care in rural Tanzania. Am J Trop Med Hyg 82: 151–155.
  48. 48. Aidoo M, Patel JC, Barnwell JW (2012) Dried Plasmodium falciparum-infected samples as positive controls for malaria rapid diagnostic tests. Malar J 11: 239.
  49. 49. Hoffman SL, Billingsley PF, James E, Richman A, Loyevsky M, et al. (2010) Development of a metabolically active, non-replicating sporozoite vaccine to prevent Plasmodium falciparum malaria. Hum Vaccin 6: 97–106.
  50. 50. Sauerwein RW, Roestenberg M, Moorthy VS (2011) Experimental human challenge infections can accelerate clinical malaria vaccine development. Nature reviews Immunology 11: 57–64.
  51. 51. Laurens MB, Duncan CJ, Epstein JE, Hill AV, Komisar JL, et al. (2012) A consultation on the optimization of controlled human malaria infection by mosquito bite for evaluation of candidate malaria vaccines. Vaccine 30: 5302–5304.
  52. 52. Padley DJ, Heath AB, Sutherland C, Chiodini PL, Baylis SA (2008) Establishment of the 1st World Health Organization International Standard for Plasmodium falciparum DNA for nucleic acid amplification technique (NAT)-based assays. Malar J 7: 139.
  53. 53. Alemayehu S, Feghali KC, Cowden J, Komisar J, Ockenhouse CF, et al. (2013) Comparative evaluation of published real-time PCR assays for the detection of malaria following MIQE guidelines. Malar J 12: 277.
  54. 54. Jackson JB, Drew J, Lin HJ, Otto P, Bremer JW, et al. (1993) Establishment of a quality assurance program for human immunodeficiency virus type 1 DNA polymerase chain reaction assays by the AIDS Clinical Trials Group. ACTG PCR Working Group, and the ACTG PCR Virology Laboratories. J Clin Microbiol 31: 3123–3128.
  55. 55. Reichelderfer PS, Jackson JB (1994) Quality assurance and use of PCR in clinical trials. PCR Methods Appl 4: S141–149.
  56. 56. Schirm J, van Loon AM, Valentine-Thon E, Klapper PE, Reid J, et al. (2002) External quality assessment program for qualitative and quantitative detection of hepatitis C virus RNA in diagnostic virology. J Clin Microbiol 40: 2973–2980.
  57. 57. Andrews L, Andersen RF, Webster D, Dunachie S, Walther RM, et al. (2005) Quantitative real-time polymerase chain reaction for malaria diagnosis and its use in malaria vaccine clinical trials. Am J Trop Med Hyg 73: 191–198.
  58. 58. Murphy SC, Daza G, Chang M, Coombs R (2012) Laser cutting eliminates nucleic acid cross-contamination in dried blood spot processing. J Clin Microbiol.
  59. 59. Wang CW, Hermsen CC, Sauerwein RW, Arnot DE, Theander TG, et al. (2009) The Plasmodium falciparum var gene transcription strategy at the onset of blood stage infection in a human volunteer. Parasitol Int 58: 478–480.
  60. 60. Dowling MA, Shute GT (1966) A comparative study of thick and thin blood films in the diagnosis of scanty malaria parasitaemia. Bull World Health Organ 34: 249–267.
  61. 61. Hammami I, Garcia A, Nuel G (2013) Evidence for overdispersion in the distribution of malaria parasites and leukocytes in thick blood smears. Malar J 12: 398.
  62. 62. Fu J, Li D, Xia S, Song H, Dong Z, et al. (2009) Absolute quantification of plasmid DNA by real-time PCR with genomic DNA as external standard and its application to a biodistribution study of an HIV DNA vaccine. Anal Sci 25: 675–680.
  63. 63. Al-Soud WA, Jonsson LJ, Radstrom P (2000) Identification and characterization of immunoglobulin G in blood as a major inhibitor of diagnostic PCR. J Clin Microbiol 38: 345–350.
  64. 64. Al-Soud WA, Radstrom P (2001) Purification and characterization of PCR-inhibitory components in blood cells. J Clin Microbiol 39: 485–493.
  65. 65. Huggett JF, Novak T, Garson JA, Green C, Morris-Jones SD, et al. (2008) Differential susceptibility of PCR reactions to inhibitors: an important and unrecognised phenomenon. BMC Res Notes 1: 70.
  66. 66. Poon LL, Wong BW, Ma EH, Chan KH, Chow LM, et al. (2006) Sensitive and inexpensive molecular test for falciparum malaria: detecting Plasmodium falciparum DNA directly from heat-treated blood by loop-mediated isothermal amplification. Clin Chem 52: 303–306.
  67. 67. Gunderson JH, Sogin ML, Wollett G, Hollingdale M, de la Cruz VF, et al. (1987) Structurally distinct, stage-specific ribosomes occur in Plasmodium. Science 238: 933–937.
  68. 68. Oguike MC, Betson M, Burke M, Nolder D, Stothard JR, et al. (2011) Plasmodium ovale curtisi and Plasmodium ovale wallikeri circulate simultaneously in African communities. Int J Parasitol 41: 677–683.
  69. 69. Sutherland CJ, Tanomsing N, Nolder D, Oguike M, Jennison C, et al. (2010) Two nonrecombining sympatric forms of the human malaria parasite Plasmodium ovale occur globally. J Infect Dis 201: 1544–1550.
  70. 70. Costa DC, Madureira AP, Amaral LC, Sanchez BA, Gomes LT, et al. (2014) Submicroscopic malaria parasite carriage: how reproducible are polymerase chain reaction-based methods? Mem Inst Oswaldo Cruz 109: 21–28.
  71. 71. malERA Consultative Group on Diagnoses and Diagnostics (2011) A research agenda for malaria eradication: diagnoses and diagnostics. PLoS Med 8: e1000396.
  72. 72. Roestenberg M, Teirlinck AC, McCall MB, Teelen K, Makamdop KN, et al. (2011) Long-term protection against malaria after experimental sporozoite inoculation: an open-label follow-up study. Lancet 377: 1770–1776.
  73. 73. Chuang I, Sedegah M, Cicatelli S, Spring M, Polhemus M, et al. (2013) DNA prime/Adenovirus boost malaria vaccine encoding P. falciparum CSP and AMA1 induces sterile protection associated with cell-mediated immunity. PLoS One 8: e55571.
  74. 74. Spring MD, Cummings JF, Ockenhouse CF, Dutta S, Reidler R, et al. (2009) Phase 1/2a study of the malaria vaccine candidate apical membrane antigen-1 (AMA-1) administered in adjuvant system AS01B or AS02A. PLoS One 4: e5254.
  75. 75. Hermsen CC, Konijnenberg Y, Mulder L, Loe C, van Deuren M, et al. (2003) Circulating concentrations of soluble granzyme A and B increase during natural and experimental Plasmodium falciparum infections. Clin Exp Immunol 132: 467–472.
  76. 76. Verhage DF, Telgt DS, Bousema JT, Hermsen CC, van Gemert GJ, et al. (2005) Clinical outcome of experimental human malaria induced by Plasmodium falciparum-infected mosquitoes. Neth J Med 63: 52–58.
  77. 77. Epstein JE, Tewari K, Lyke KE, Sim BK, Billingsley PF, et al. (2011) Live attenuated malaria vaccine designed to protect through hepatic CD8(+) T cell immunity. Science 334: 475–480.