Standardization of cytokine flow cytometry assays

In the first experiment, whole blood from three cytomegalovirus (CMV)-seropositive donors was activated, fixed, and frozen by the method described in Nomura et al.[6]. The blood was incubated for 6 hours in the presence of brefeldin A, either with no stimulus, Staphylococcal enterotoxin B (SEB), or a mixture of overlapping peptides corresponding to the CMV pp65 protein [8‐10]. Aliquots of the frozen activated whole blood were then shipped to 9 laboratories for processing and analysis. The results, as reported by each site and also as determined by central, automated analysis of the raw data files, are summarized in Figure 3. For data reported by each site, the mean inter-lab C.V. was 55% for CD4 T cell responses and 32% for CD8 T cell responses. This is higher than the inter-assay C.V. previously reported for ICS assays performed at a single site [6]. However, when the raw data was centrally analyzed, the inter-lab C.V. was reduced to 24% for both CD4 and CD8 T cell responses, very similar to the inter-assay C.V. previously reported [6]. Thus, a large proportion of the site-to-site variability could be explained by differences in gating of the ICS data.

In a second experiment, whole blood from three CMV-seropositive donors was shipped overnight to 6 U.S. labs for activation, processing, and analysis. This experiment was conducted twice, since the first trial was compromised by shipping delays. The results of the second trial are shown in Figure 4. As in Figure 3, the inter-lab C.V.'s were higher for data reported by each site, although the reduction due to centralized analysis was less dramatic than in the first experiment. Feedback on gating differences was provided to the labs between the first and second experiment, so the smaller effect of centralized analysis could be attributed to a progression of the individual sites toward a more uniform gating scheme. Also, the relatively high C.V. for CD4 T cell responses (42% even after centralized analysis) could be due to the low mean response to CMV peptide mix in two of the donors (donors were not identical in the different experiments). Since the C.V. varies inversely with the mean of the sample population, comparison of C.V.'s between experiments performed on different donors are subject to this confounding variable.

In a third experiment, PBMC were isolated from 6 CMV-seropositive donors and cryopreserved. Replicate cryopreserved vials were sent to each of 7 sites, where they were thawed, rested overnight, stimulated, processed, and analyzed. The post-thaw viability and recovery of the PBMC samples from each site are shown in Figure 5. Mean viabilities were >82%, and mean recoveries were >75% for each sample (determined by trypan blue exclusion). In general, viabilities were quite consistent across labs, while recoveries varied more widely, both between labs and between samples. This could be due in part to imprecise filling of the vials when they were initially frozen, which would impact the apparent recoveries calculated upon thawing. Furthermore, while a common thawing protocol was provided, no attempt was made to standardize counting methods or other factors that may impact the reproducibility of viability and recovery calculations across labs.

The ICS results from cryopreserved PBMC are shown in Figure 6. The inter-lab C.V.'s for this experiment averaged slightly lower than those for the whole blood experiments (25–32% for manual analysis; 23–25% for centralized automated analysis). Like the fresh whole blood experiment, the improvement in C.V. from centralized analysis was relatively small. When outlier samples with unusually low responses were checked for viability and recovery, they were not necessarily low in these parameters as well. In fact, there was no obvious relationship of viability or recovery with response, perhaps because viabilities and recoveries were virtually all within generally acknowledged limits of acceptability (>80% viability, >50% recovery) [11, 12].

To expand upon the results of the third experiment using cryopreserved PBMC, a fourth experiment with this sample type was carried out using an enlarged cohort of participating laboratories (table 1). In addition, a protocol refinement was introduced to attempt to further reduce inter-lab variability: Peptide stimuli together with brefeldin A were provided in lyophilised form in appropriate wells of a microtiter plate, to provide simplified assay set-up; and lyophilised staining antibody cocktails were provided in the corresponding wells of a second microtiter plate. These latter were rehydrated and added to the cells in the first microtiter plate after fixation and permeabilization of the cells. This experiment also sought to compare two different types of staining antibody cocktails: the two cocktails used in the previous three experiments (IFNγ FITC/CD69 PE/CD4 PerCP-Cy5.5/CD3 APC and IFNγ FITC/CD69 PE/CD8 PerCP-Cy5.5/CD3); and one that combined CD4 and CD8 staining in a single sample, as well as adding IL-2 staining (CD4 FITC/IFNγ +IL-2 PE/CD8 PerCP-Cy5.5/CD3 APC).

The results of this experiment are shown in Figure 7 (note the change to a linear scale in this and the following figures). A set of peptides consisting of epitopes from CMV, EBV, and influenza (CEF) [13] was used as a positive control, due to restrictions on international shipment of SEB. CEF was expected to induce CD8, rather than CD4 responses, and indeed the CD4 responses to this control were very low or negative. The CD8 responses, while positive, were considerably lower than those seen with SEB in the previous experiments. Responses to CMV pp65 peptide mix were also not high in the donors used in this experiment, but were detectable in both CD4 and CD8 compartments. Despite the lower response means, the average C.V. for this experiment was roughly similar to the previous experiment, when comparing the same staining antibody cocktails (cocktails 2 and 3). For unknown reasons, the average C.V. for cocktail 1 (CD4/IFNγ +IL-2/CD8/CD3) was higher than that for cocktails 2 and 3, although the mean percentage of cytokine-positive cells was not significantly different. The addition of IL-2 in this cocktail did not significantly increase the mean percentage of cytokine-positive cells, as very few cells responding to CMV produce IL-2 without IFNγ [14]. This would not be expected to be true for all types of responses, however.

When data for this experiment were centrally analysed (Figure 7B), the average C.V.'s were considerably reduced, much like in the first experiment (Figure 3). This could reflect the fact that new laboratory sites had been added that had not yet standardized their gating strategies with the existing sites; thus more benefit was realized by centralized analysis. The difference in average C.V. between cocktail 1 and cocktails 2 and 3 was preserved even after centralized analysis. The mean C.V. for cocktails 2 and 3 was now 18%, the lowest variability seen in any of the experiments. For comparison, the mean inter-lab C.V. of the percent CD4⁺ or CD8⁺ cells in the unstimulated samples from this experiment was 3% and 7%, respectively (data not shown).

As a positive control in Experiment 4, a set of PBMC were SEB-activated, processed, stained, and then lyophilised in certain wells of the lyophilised antibody plates. They were hydrated and transferred to the plate containing activated cells, along with the staining antibodies. These cells served as a control for instrument setup and gating, since all the activation and processing steps were done centrally. The results reported by the individual sites for these cells are shown in the left panel of Figure 7C. Surprisingly, the average C.V. (20.5%) was only slightly lower than that for the rest of Experiment 4, in which cells were activated and processed independently by each site. However, when the control cell data were centrally analysed using a dynamic gating template (right panel), the C.V.'s were reduced to 3–7%. This reinforces the notion that the vast majority of inter-lab variability is due to gating.

"Background" or spontaneous cytokine production (subtracted from all data in Figures 3, 4, 6, and 7) is plotted for all experiments in Figure 8. Backgrounds were generally low. For all experiments combined, the mean CD4 background was 0.02% and the mean CD8 background was 0.05%. 98% of CD4 samples and 84% of CD8 samples had backgrounds =0.1%. There were a significant number of CD8 samples that exhibited high spontaneous cytokine production. However, the mean CD8 background was significantly higher than the mean CD4 background only in the activated, fixed whole blood experiment (p < 0.0005). When centralized automated analysis was applied to the data, backgrounds were not usually reduced. This indicates that the gains in reproducibility seen with centralized analysis were not simply due to reductions in background.

While CD4 backgrounds were very similar between experiments, CD8 backgrounds varied. The median CD8 background in the PBMC experiments was significantly lower than that of the frozen activated whole blood experiment (p < 0.0001) or the fresh whole blood experiment (p < 0.05). The differences were significant after centralized analysis as well. However, this could be due to the fact that different donors were used in the four experiments, rather than being due to any inherent difference between assay types. In experiment 4, the CD4 backgrounds for cocktail 1 were significantly higher than those for cocktail 2 (p < 0.05, data not shown), while there was no significant difference for CD8 backgrounds. This could be due to the inclusion of IL-2 in cocktail 1, which would be expected to be produced by more CD4⁺ than CD8⁺ cells, and thus contribute selectively to the CD4 background.

Heparinized whole blood was collected from healthy CMV seropositive volunteers for experiments 1 and 2. For experiment 1, the blood was activated in 15 mL conical tubes according to the method of Nomura et al.[6]. Activated blood was treated with 2 mM final concentration of EDTA for 15 minutes at room temperature, then 10 volumes of FACS Lysing Solution (BD Biosciences, San Jose, CA) were added. After 10 minutes at room temperature, the tubes were frozen at -80°C, then shipped to participating laboratories on dry ice. The protocol used by each laboratory for handling these samples is provided in Additional File 1.

For experiment 2, 5 mL of heparinized whole blood was overnight shipped in an insulated container at ambient temperature to each participating lab. The protocol used by each lab for handling these samples is provided in Additional File 2.

For experiments 3 and 4, PBMC from leukapheresis of CMV seropositive donors were isolated using Ficoll gradient separation. They were then cryopreserved according to a standard protocol (Disis et al., submitted for publication). These cryopreserved PBMC were shipped to participating labs using liquid nitrogen dry shippers. The protocol used by each lab for thawing and processing of these cells is provided as Additional Files 3 and 4.

The flow cytometry instrumentation used in this study included 12 BD FACS Caliburs (BD Biosciences), 3 BD LSRIIs (BD Biosciences), and 1 CyAn (Dako Cytomation, Fort Collins, CO). Instrument setup was at the discretion of the individual laboratory, and was either manual (using isotype control stained cells to set PMT voltages, and single-stained cells to set compensation) or automated (using BD FACSComp software and BD Calibrite beads (BD Biosciences)). In some labs, automated setup was followed by manual adjustment using stained cells as above.

Original FCS files from each site were sent to BD Biosciences for analysis using a dynamic gating template (Figure 2B). This template was built using "Snap-To Gating" and "Tethering" tools available in CellQuest Pro software (BD Biosciences). The shape of the snap-to gates is determined by a clustering algorithm, and this algorithm allows for their movement from sample to sample in a data-dependent manner. The size and amount of allowable movement of each snap-to gate was adjusted by inspection of a subset of the files to be used, with iterative changes being made until the template performed as desired. The template was then used, without further adjustment, on all the files of a given experiment. Since the template was generated in CellQuest Pro software, only files generated on FACS Calibur instruments were analyzable by this method.

The %CV was calculated as 100*SD/mean for each sample, from the percentage of cytokine-positive cells reported by each laboratory or derived from centralized analysis of that sample. The mean CV for each experiment was taken as the average of all the individual sample CVs. Statistical significance of differences in the average CV between experiments was calculated using a Kruskal-Wallis test, with Dunn's Multiple Comparison test to determine where significant differences were found. The significance of the difference between individually and centrally analyzed data was calculated by comparing the aggregate CVs of all samples from all experiments using a Wilcoxin signed rank test for matched pairs. A two-tailed Student t test was used to calculate significance of differences in background within or between experiments.