Quality assessment of an interferon-gamma release assay for tuberculosis infection in a resource-limited setting

Interferon-gamma concentrations in the plasma were measured by ELISA, using microtiter plate washer and reader (Wellwash Plus Microplate Washer and Multiscan JX Microplate Reader, Thermo Electron Corporation, Vantaa, Finland) with the analysis software provided by the manufacturer (QuantiFERON-TB Gold Analysis Software, ver. 2.50, Cellestis). In this study, interferon-gamma concentrations obtained from this calculation were directly used for further analysis.

Two trials were carried out in Hanoi TB and Lung Disease Hospital, Viet Nam. Between the first and second trial, statistical analysis was made and an intervention was planned to ensure counterchecking and correct questionable manipulations. Each trial consisted of two runs. In each run, three milliliters of blood were collected from volunteers after informed consent had been obtained. Study protocols using IGRA were approved by ethical committees of the Ministry of Health, Viet Nam and International Medical Center of Japan respectively.

Two laboratory members, A and B, performed either plasma harvest or ELISA operation or both: Harvest included labeling and placing plasma storage tubes properly and transferring plasma from centrifuged blood collection tubes to these tubes by pipetting. ELISA was a process including preparing reagents and transferring plasma samples into the microtiter plate. ELISA ended with calculation of interferon-gamma concentration. Because their roles were changed occasionally due to the limited manpower of the laboratory, their performance in both Harvest and ELISA was the subject to be analyzed.

A single specimen obtained from routine blood collection was not sufficient to assess two staff members' performance. Because additional blood sampling was not easily accepted in many countries including Viet Nam, BIBD was attempted to obtain analytical information from small volume of plasma samples in this study: Of four possible combinations of harvester and ELISA operator, two combinations were cyclically chosen, using the limited amount of specimen. Allocation of observed combinations by BIBD in this study was described in Table 1. In each trial, there were two levels of Harvest (two different Harvesters), two levels of ELISA (two different ELISA operators) and 12 levels of Specimen (12 different blood donors).

To identify outliers, Mahalanobis distance D was calculated, which took the distance from the mean and correlation into account [10]. When D > 2.0, the value of that observed pair was regarded as outlier.

To assess effects of factors of interest and error variance, we used a fixed-effect model:

of which,

y_ijk: Interferon-gamma concentration in the plasma

μ: Grand mean of all measurements

α_i: Harvest with i levels: i = 1, 2 (= A and B)

β_j: ELISA with j levels: j = 1, 2 (= A and B)

γ_k: Blood specimen with k levels: k = 1, 2,..., 12

ε_ijk: Within-individual error; following normal distribution with mean = 0 and variance = σ²: N (0, σ²)

In this clinical setting, effects of interaction terms were not considered in the above model, because harvesting plasma and performing ELISA are independent steps and it is unlikely that the exchanging of staff roles in itself could increase the chances of error.

To determine whether individuals of Harvest or ELISA affect within-individual error, we assessed a fixed-effect model with heterogeneous variance of error in the following way:

where error follows the normal distribution N(0, σ_ij ²).

Error variance affected by Harvesters was evaluated in the following formula:

Similarly, the following formula was used for error variance affected by ELISA operators:

Error variance ε_ijk that included sources of Harvest and ELISA was calculated in a simple one-way ANOVA model adjusted by specimen. Based on the following formula, CVs of the two trials were assessed:

CV should not be larger than 20% in any types of immunoassay [8].

To analyze overall within-individual error between the two trials statistically, we used a fixed-effect model with heterogeneous variance of error, under the assumption that α and β were fixed throughout the trials. The effect of each blood specimen γ was expected to be different between the two trials.

of which, ε_ijk1 and ε_ijk2 were within-individual errors of the first and the second trial respectively. On the above assumption, ε_ijk1 and ε_ijk2 would be heterogeneous error between the trials.

Out of 72 measurements obtained from the first trial, seven outliers were identified: One was in Nil condition, three in TB-Antigen and three in Mitogen (Mahalanobis D = 2.64 to 4.69).

To assess effects of individuals for Harvest and ELISA and character of errors involved in the first trial, we first performed three-way ANOVA using a fixed-effect model, in which three factors, Harvest, ELISA and individual blood specimens may have possible effects on the interferon-gamma concentration respectively. This model decomposes the total variance into between-individual error (or bias) and within-individual error (or imprecision). Herein, "between-individual error" indicates deviation in interferon-gamma values caused by the difference between Harvesters or ELISA operators, and "within-individual error" represents fluctuation of interferon-gamma values measured by a single Harvester or ELISA operator.

As shown in Table 2, mean square error indicating magnitude of within-individual error was large in all conditions of the first trial, which was indicated by remarkably large CV (> 20%) for Nil, TB-Antigen and Mitogen. Furthermore, in the condition of Mitogen, the mean-square value directing the effect of ELISA, or "between-individual error", was significantly large (P = 0.017). In the other two conditions, the effects of ELISA and Harvest were also considerably large but did not reach significant levels, as compared with the corresponding mean square errors. These findings indicate that their performance is unstable. Problems specific to ELISA and Harvest should be considered, although not statistically significant in all conditions.

We then analyzed which factor affected within-individual error. Because two laboratory members were involved in each step of this experiment, we assumed that within-individual error, i.e. error variance, could be different depending on the personnel in each step. Thus, we chose a fixed-effect model with heterogeneous variance of error affected by Harvest and ELISA (Table 3).

In Nil condition, difference in error variance was statistically significant between Harvesters A and B (P = 0.0040), when error variance caused by ELISA operator was not considered. Difference of error variance caused by ELISA operators A and B was also significant (P = 0.024), when error variance caused by Harvester was not taken into account. These findings imply that under the model, the error variance was affected significantly by different Harvesters or ELISA operators, respectively.

By means of the above-mentioned statistical analysis of the first trial, we identified several points to be improved: a) there was a considerable number of outliers. Within-individual error was large and between-individual error was also comparably large, and b) within-individual error was affected by both Harvesters and ELISA operators at least when Nil was measured.

Based on these results, an intervention was introduced: 1) reviewing all procedures of Harvest and ELISA, 2) reconsidering and strengthening standard operation procedures, 3) checking working condition of machines, and 4) developing a checklist for countercheck. First, we attempted to find out which procedure of harvesting and ELISA operation would be unstable and all questionable manipulations were listed up. Essential laboratory skills, such as mixing the solution by pipetting, were reviewed. Secondly, standard operation procedures were rechecked and corrected seeing that the laboratory personnel were handling three blood collection tubes and three other plasma storage tubes from each blood donor at a time, they should take every care to identify the tubes during Harvest and ELISA and to confirm the right position of corresponding tubes. Thorough instruction for handling ELISA plates and tubes with manipulation of the pipette was given to avoid carry-over error or contamination. After intensive discussions, more attention was paid to basic laboratory practice and reduction of preventable mistakes. Thirdly, performance of the ELISA plate washer and reader and the quality of distilled water were also checked. Technical requirements from the manufacturer, such as temperature for reagent reservation, time of incubation, were strictly followed. Finally, a checklist for the countercheck of each step was developed for practical use.

To assess the overall improvement after intervention, CV was compared between the two trials. Because variation due to Harvest and ELISA was of interest, CV adjusted by the effect of specimens was calculated and used. The CV had decreased remarkably in each condition of the second trial, as compared with that of the first trial, indicating the overall improvement of test performance after intervention (Table 4).

In the second trial, only one outlier was seen in Nil condition (Mahalanobis D = 2.59); the number of outliers was lower than that of the first trial.

We then proceeded to analyze the change of parameters that had possibly contributed to overall improvement of test performance. As shown in Table 5, both mean square error and mean-square values showing effects of Harvest and of ELISA were markedly lower in the second trial. The former implies the decrease in within-individual error and the latter shows the reduction of between-individual error. The latter change was also clearly shown when differences of least square means between Harvesters and between ELISA operators in each condition of the second trial were compared with those in the first trial (Figure 1).

In contrast to the first trial, there were no significant differences of error variance affected by Harvesters or ELISA operators (Table 6). This finding showed that the heterogeneous error variance indicating personnel-dependent unstableness was small enough in each step of the second trial.

We further evaluated the decrease in overall within-individual error statistically. For this purpose, we used a fixed-effect model with heterogeneous variance between the two trials. Under the assumption that influence of Harvest and ELISA was not changed between the two trials, estimated overall error variances of the two trials were compared. As shown in Table 7, values indicating the overall within-individual error in all conditions had significantly decreased in the second trial (P < 0.0001).