Evaluation of System Accuracy, Precision, Hematocrit Influence, and User Performance of Two Blood Glucose Monitoring Systems Based on ISO 15197:2013/EN ISO 15197:2015

Participants were allowed to read the instructions of use and to familiarize themselves with the system by performing up to three measurements with control solution. No additional instructions, training, or assistance by study personnel were provided.

After the familiarization period, participants performed BG measurements under supervision of the study personnel who documented any handling errors on the basis of the system-specific instructions for use. Prior to measurements with the BGMS, all participants washed their hands; in case they did not remember on their own, they were reminded to do so. Following the BGMS measurement, the participants were asked whether they thought they had performed the measurements correctly. Participants were allowed to repeat the measurement up to three times if they thought the performance was incorrect or if they obtained an error message or no result at all. Handling errors, based on the instructions of use from the manufacturer, which were only observed by the study personnel did not lead to exclusion of measurements from accuracy analysis. Two blood samples for comparator measurements were subsequently collected by the study personnel and hematocrit values were determined. Comparator values were determined in duplicate measurements with a YSI 2300 STAT PLUS (YSI Inc, Yellow Springs, OH) analyzer using the glucose oxidase method on separated plasma samples. A secondary analysis was done using measurement results from the hexokinase-based Cobas Integra 400 plus (Roche Diagnostics GmbH, Mannheim, Germany) (see supplementary material). Demographic data of the 102 subjects included in the evaluation of user performance can be found in the supplementary material (Table S1).

Each BGMS was tested on the basis of ISO 15197 with three different reagent systems lots (test strip lots). A total of 115 subjects were recruited to obtain 100 evaluable datasets per BGMS. Demographic data of the subjects is found in Table S1 in the supplementary material. Subjects were asked to wash and dry their hands before capillary samples were collected by skin puncture on the fingertip. The glucose concentrations in the samples were determined in duplicate with two test meters using test strips from the same vial. This step was repeated with three reagent system lots per BGMS. Comparator measurements were performed with the laboratory analyzers in separated plasma. Stability of glucose concentration was checked by calculating the difference between blood samples for comparator measurements collected before and after the measurements with the BGMS. Hematocrit values, temperature, and humidity were tested to be in the range of the manufacture’s labelling.

According to ISO 15197, BG concentrations of samples were distributed over the clinically relevant concentration range between ≤ 50 mg/dL and > 400 mg/dL (5% ≤ 50 mg/dL, 15% > 50 to 80 mg/dL, 20% > 80 to 120 mg/dL, 30% > 120 to 200 mg/dL, 15% > 200 to 300 mg/dL, 10% > 300 to 400 mg/dL, and 5% > 400 mg/dL). Samples were allowed to be altered through glycolysis or glucose supplementation to achieve glucose concentrations ≤ 50 mg/dL or > 400 mg/dL, respectively. Additionally, up to seven samples between 50 and 80 mg/dL and up to five samples between 300 and 400 mg/dL could be altered but unaltered samples were preferred. In altered samples, partial pressure of oxygen was checked to be comparable to unaltered samples.

Accuracy evaluation of venous samples was only performed for system B since it is labeled for venous blood samples in addition to capillary samples. Accuracy analysis was performed similar to system accuracy, except for using venous blood samples, collected in lithium-heparin tubes which were applied to the test strip using a syringe. Glucose concentrations were distributed as follows: 5% ≤ 50 mg/dL, 90% > 50 to 400 mg/dL, and 5% > 400 mg/dL. Manipulation of samples was only allowed for samples ≤ 50 mg/dL or > 400 mg/dL. Demographic data of the analyzed subjects is given in Table S1 in the supplementary material.

To assess the intermediate measurement precision, ten meters, three reagent system lots, and three control solution samples representing glucose concentrations from 30 to 50 mg/dL, 96 to 144 mg/dL, and 280 to 420 mg/dL were used. Each sample was measured once daily with each combination of meter and reagent lot. Measurements were performed on ten subsequent days by two different users. Comparator measurements with YSI 2300 STAT PLUS (YSI Inc, Yellow Springs, OH) to assess sample stability were performed before and after the daily measurement procedure. Sample temperature was checked to be within 23 ± 5 °C and was not allowed to differ more than ± 2 °C from the starting value.

For the analysis of measurement repeatability, ten meters and three reagent system lots were used on five venous blood samples collected in lithium-heparin tubes. Samples were taken from five different subjects in the following glucose concentration ranges: 30–50 mg/dL, 51–110 mg/dL, 111–150 mg/dL, 151–250 mg/dL, and 251–400 mg/dL. Glucose concentrations of a sample could be adjusted to verify that each concentration range is covered. For each BGMS, 10 measurements were performed with each combination of meter and reagent lot on each sample. Aliquots for plasma separation and subsequent duplicate measurement with a glucose oxidase-based comparator method (YSI 2300 STAT PLUS) were collected before and after the BGMS measurements.

Sample temperature before and after measurements was checked to differ by no more than ± 2 °C. Stability of glucose concentrations within the samples was verified in the same way as for system accuracy.

Venous blood samples from different subjects were used to generate samples in three glucose concentration intervals: 30–50 mg/dL, 96–144 mg/dL, and 280–420 mg/dL. Samples were assigned to the respective glucose concentration category depending on the mean measurement result of the laboratory method. Glucose concentrations of the sample could be adjusted as described above. From each glucose concentration interval, five samples with different hematocrit levels were generated. Hematocrit levels that were tested were 15%, 30%, 42 ± 2%, 55%, and 65% (system A) or 70% (system B). The highest and the lowest hematocrit levels represented the upper and lower limit of the system’s acceptable hematocrit range as indicated in the manufacturer’s labeling. To adjust for different hematocrit values, an aliquot of samples was centrifuged and then plasma and packed cells were separated. Defined volumes of plasma, cells, and whole blood were then mixed to generate the individual hematocrit levels. Each sample was measured within 36 h of sample collection and the sample temperature was confirmed to be 23 ± 5 °C and differing no more than ± 3 °C in between the measurements. Ten measurements were performed on each individual sample with each reagent lot. Partial pressure of oxygen in each sample was determined and checked to be comparable to capillary blood (55–96 mmHg) [5].

Accuracy of measurement results obtained by subjects was determined by comparison to the mean of the respective duplicate comparator measurements in the first blood sample. The second sample was only used for quality control to ensure sample stability. Acceptance criteria based on ISO 15197 were applied which request that ≥ 95% of the individual measured glucose values must be within ± 15 mg/dL (glucose concentrations < 100 mg/dL) or ± 15% (glucose concentrations ≥ 100 mg/dL) of the comparator measurement. For system A one sample had to be rejected from analysis since the acceptance criteria for sample stability were not fulfilled. For system B, one sample was rejected because there was no valid quality control (QC) measurement result of the comparator method.

ISO 15197 requires as first criterion that ≥ 95% of capillary measurement values per reagent system lot must fall within ± 15 mg/dL of the measured comparator glucose values at glucose concentrations < 100 mg/dL or within ± 15% at glucose concentrations ≥ 100 mg/dL. Differences between each of the 200 BGMS measurement results per reagent lot and the corresponding mean comparator results were calculated. Data was only included if the difference between the comparison samples taken before and after the measurements with the BGMS was ≤ 4 mg/dL at glucose concentrations ≤ 100 mg/dL and ≤ 4% at glucose concentrations > 100 mg/dL. For each concentration category, the first valid samples taken were included and the category was closed as soon as the required number of samples was achieved. Accuracy results were visualized in a difference plot. As a second criterion which requires ≥ 99% of data points to be in the clinically acceptable zones A and B of the consensus error grid (CEG) [6] for diabetes, the relative number of data points was calculated for all three reagent lots taken together. In addition, for each reagent system lot, the relative bias of the measurement results was determined according to Bland and Altman [7]. A total of 22 measurements were not included in the analysis as a result of the following reasons: the concentration category was filled already (n = 12), sample stability could not be ensured (n = 6), no valid QC result of the comparator measurement (n = 3), and hemolysis in plasma sample for comparator analysis (n = 1).

Data for venous accuracy assessment was analyzed and is presented the same way as described above for capillary blood in system accuracy assessment. Here, nine samples were excluded from analysis because of the following reasons: concentration category already filled (n = 6), no valid QC result of the comparator measurement (n = 1), glucose concentration of a manipulated sample was < 400 mg/dL (n = 1), and comparator sample was outside the borders used for CEG analysis (n = 1).

For each combination of glucose concentration range and reagent system lot, the mean glucose value, the standard deviation (SD) for samples with mean BGMS result < 100 mg/dL, and the coefficient of variation (CV) for samples ≥ 100 mg/dL were calculated. Variance components analysis was performed for meter, reagent system lot, user, day, reagent system vial, and control solution vial.

The mean BGMS result and the SD (for samples with mean BGMS result < 100 mg/dL) or the CV (samples ≥ 100 mg/dL) were calculated for each glucose concentration and each test strip lot separately.

Mean glucose values determined by the laboratory analyzer, SD, and the bias between the mean glucose value of the BGMS and of the laboratory comparator method were determined for each combination of glucose concentration and hematocrit value. The effects of hematocrit were determined by calculating the difference (normalized bias) of the bias at high and low hematocrit levels and the bias at the mid-level.