European follow-up of incorrect biomarker results for colorectal cancer demonstrates the importance of quality improvement projects

Metastatic colorectal carcinoma (mCRC) is the third most commonly diagnosed malignancy and the fourth leading cause of cancer death worldwide [1]. Besides standard chemotherapy, mCRC patients are currently receiving personalized treatment by anti-epidermal growth factor receptor (EGFR) monoclonal antibodies, which has been shown to significantly increase the median survival time from 18.5 to 23.5 months of mCRC patients [2].

In 2008, mutations in exon 2 of the Kirsten rat sarcoma viral oncogene homolog (KRAS) gene were shown to be a negative predictor for anti-EGFR therapy benefit [2, 3]. In 2013, the same was demonstrated for mutations in KRAS exons 3 and 4, and for the less frequent mutations in neuroblastoma rat sarcoma (NRAS) viral oncogene homolog exon 2–4 [4, 5], resulting in an extension of the drug labels for cetuximab and panitumumab by the European Medicine Agency (EMA) [6, 7]. Consequently, molecular diagnostic laboratories were challenged to include these new test requirements in a correct and timely manner [4].

Since 2009, the European Society of Pathology (ESP) has been involved in the organization of a yearly colon external quality assessment (EQA) scheme to assess and improve RAS biomarker analysis in mCRC [8, 9]. Based on the updated requirements, the 2013 ESP colon EQA scheme was expanded by the assessment of full RAS testing (exon 2, 3, and 4 of both KRAS and NRAS) [10]. Since that same year, laboratories could also optionally test the BRAF (B-Raf proto-oncogene) gene, which has demonstrated prognostic value and is increasingly being analyzed in Europe [11].

Results from the 2013 scheme revealed that full RAS testing was only implemented by half of the laboratories (49.3%, n = 131 laboratories) and that there were numerous errors in testing the new gene segments [10]. In addition, EQA data confirmed that molecular diagnostic laboratories in Europe are using a large variation in methods for (a) the estimation of the neoplastic cell content [12], (b) for DNA extraction [13], and (c) for determining the RAS and BRAF status [13].

A 2016 study showed that the vast majority of samples (97%) tested by laboratories participating to an EQA scheme had been correctly classified. For about 2% of samples tested, an incorrect outcome was obtained that could potentially lead to a different anti-EGFR therapy advice [14]. Given the potential impact of predictive biomarker analyses on patient outcome, it is important to evaluate the exact causes of errors and to provide tailored feedback to diagnostic laboratories for quality improvement [15]. In turn, laboratories are encouraged to implement the necessary corrective and preventive actions (CAPA) as a required by the ISO15189 standard [16] or national equivalents, and the Clinical Laboratory Improvement Amendments of 1988 [17].

In clinical biology and forensics, error causes have been shown to occur mostly during the pre- (46–86%) and post-analytical (18–47%) phases of the total test process (TTP) compared to the analytical phase (7–13%) [18, 19], although the lack of standardization in taxonomy accounts for some of the variation seen in these error rates [20].

Although EQA schemes reflect the performance of diagnostic laboratories, more detailed information is required on the error causes and distribution throughout the TTP for molecular cancer diagnostics, as well as the actions undertaken by laboratories to improve quality in the long-term [21].

Therefore, the objectives of this study were (a) to evaluate the causes, distribution, and follow-up of laboratory errors from laboratories participating to the ESP colon EQA scheme; (b) to provide feedback to laboratories as to how practice can be improved, and (c) to assess potential improvement between 2016 and 2017 EQA schemes.

The 2016 and 2017 ESP colon EQA schemes were organized according to the ISO 17043 standard for proficiency testing [22] and the guideline on the requirements of external quality assessment programs in molecular pathology [23]. Participation to EQA was free of choice and open to all laboratories worldwide. Details on validation, results submission, and feedback provided to the laboratories have been previously described ([13], Supplemental Table 1).

At the end of both EQA schemes, all laboratories with at least one major genotyping error, a score “i,” or technical failure (in which no result could be obtained for a case) in one of the ten provided formalin-fixed paraffin-embedded cases were invited by e-mail to complete an electronic survey with both laboratory-specific (general) questions and case-specific questions for each observed error. A list of definitions was included to clarify all questionnaire terms (Supplemental Table 2). Data was collected for 1 month, laboratories received a first reminder after 14 days and a second the day before the deadline.

All participants to the 2016 EQA scheme were invited to attend a 1.5-day long optional workshop, organized in December 2016 at the Radboud University Medical Center, The Netherlands. Topics were based on the 2016 survey output and included issues occurring in the pre-, post-, and analytical phase as cited by the survey respondents. A separate microscopy session focusing on the estimation of neoplastic cell content was held outside this project (Dufraing et al., submitted for publication).

Response bias was assessed by investigating the difference in laboratory characteristics between survey respondents and non-responders. Missing data were reported in the tables accordingly and not included in the statistical analysis. The reported accreditation statuses and laboratory settings were validated on the websites of the relevant national accreditation bodies and the laboratories’ website, respectively. Comparison of categorical variables was performed using chi-squared (Χ²) tests or a Fisher’s exact (FE) test if one of the row or column cells counted below five. For the difference in the sample amounts tested and people involved in the laboratory between responders and non-respondents, categories were treated as ordinal data following a Mann-Whitney U (MWU) test. For a combination of categorical and continuous variables (e.g., improvement of the average genotyping score between more than two groups) a one-way ANOVA with Tukey’s HSD was performed. Bonferroni corrections were applied when necessary. The significance level was set at α = 0.05. All statistical analyses were performed using SPSS Statistics Subscription version 1.0.0.903 (IBM, Armonk, NY, USA). Graphs were created using Microsoft Excel Professional Plus 2013.

Accurate biomarker tests are crucial to determine appropriate treatment options for mCRC patients. To further improve the standard of biomarker testing, diagnostic laboratories are encouraged [16, 17, 24, 25] to implement measures for continual quality improvement, including CAPAs, education of laboratory personnel, and participation to EQA to compare the laboratory’s performance to peers and identify improvement priorities. This study focused on laboratories that reported an error during the annual EQA assessment. It included an in-depth analysis of EQA results to assess the influence of the exact error causes and their follow-up on performance, which have been reported in other fields besides molecular pathology [15, 18‐20]. It included a survey and evaluation of customized feedback provided to the participant laboratories.

EQA participation has been shown to reflect RAS testing performance in routine practice [14]. This was confirmed by this study by a significant link between recurring errors and a lower average analysis score for participants who performed less in the first EQA scheme. This also suggests that EQA might exert a positive influence on laboratory performance, as previously reported for non-small cell lung cancer [21]. The results of this study demonstrated 6.1% of samples misclassified in 2017. Given the impact of the biomarker status on treatment choice it is important to continue improving biomarker testing.

Active participation to quality improvement projects aids laboratories in the critical evaluation of their results, as shown by the improved performance for workshop participants and survey respondents compared to general participants. Indeed, protocol revisions were frequently reported as CAPAs in the survey, and performing this CAPA type led to less errors and a better score in the next scheme. These revisions might be technical (e.g., a change to the analysis protocol) or general of nature (e.g., a general change to prevent errors from occurring such as building in a second check step when entering the results in to the online system). Moreover, the fact that the type of action performed (instead of type of error) influences the performance in the next scheme, suggests an active role for laboratories in quality improvement.

Survey respondents were a good representation of EQA participants, and error monitoring was not restricted to larger laboratories, laboratories in a research setting or who are accredited for molecular pathology. Although receiving accreditation was not linked to a better performance in this study, it has previously been shown to aid in the successful implementation of a new biomarker [26]. These surveys had the advantage of a standardized taxonomy, which allows to monitor error causes on a longitudinal level. The availability of multiple international laboratories’ data enables to link error causes to specific laboratory characteristics and methodologies. This can reveal systematic shortcomings and critical points in the TTP, eventually guiding molecular diagnostic laboratories.

In terms of continuous education, many laboratories did not perform additional training for results interpretation besides a person’s educational degree. Although training of the staff for a specific methodology should be well documented, and re-evaluated at frequent intervals [16], this was not reflected in the EQA performance.

Analysis of 56.2% technical failures and 40.8% of genotyping errors in both schemes combined, stresses the need of risk analysis in the TTP instead of merely the analytical phase. Consistent with previous results [15, 18], the pre- and post-analytical phases constituted a high fraction of the observed causes. In addition, as pre-cut and pre-labeled slides were provided to participants, an evaluation of the pre-analytical errors in routine practice of prior steps (deparaffinization, cutting, labeling) is advisable [27] as well as of errors at phases outside the laboratory’s responsibility. Namely, errors were reported in the pre-pre-analytical phase (from test request to sample reception at the laboratory) and the post-post-analytical phase (interpretation of the reported results by the clinician and making the appropriate therapy decision), albeit in other fields [15, 18‐20].

This study demonstrates that pre-analytical errors were more likely to result in not obtaining the maximum score in the next scheme, and that a close involvement of the pathologist in results reporting and error follow-up contributes to a better scheme performance and less pre-analytical problems, especially when using a commercial kit, in line with a previous longitudinal study [13]. This stresses the importance for standardization of the neoplastic cell content determination for test outcome interpretation in mCRC [13] (Dufraing et al., submitted for publication). This is supported by the observation that only for 1 out of 12 cases for which the laboratories reported that the tumor tissue was not optimally suited for analysis in 2016, this was good practice, as it was indeed a case without neoplastic cells. Coincidently, survey respondents in 2016 were less likely to perform the analysis at the department of pathology and more frequently outsourced the selection of the neoplastic cell to another laboratory. Therefore, it might be useful to analyze a larger dataset or to evaluate non-conformities in routine practice to evaluate if these responder characteristics might have skewed the data and if non-conformities observed during routine reflect non-conformities reported during EQA, as even more challenging cases might occur in routine.

In 2017, more (analytical) methodological and personnel errors were observed, and more frequently no CAPA was implemented compared to the pre-analytical issues reported in 2016. This is surprising, as this is a requirement of the ISO 15189 and similar quality framework. Also, CAPAs were more likely to be monitored by the laboratory director at the expense of the laboratory technicians and were performed more often before the official release of the EQA results. This might suggest that these analytical problems are considered to be more severe by participants as compared to pre- and post-analytical problems in 2016, and direct follow-up may be more difficult. Indeed, reported causes included (a) unknown factors for which the manufacturer needed to be contacted or (b) a variant that was not included in the method for analysis, not linked to a specific methodology. Surprisingly, in spite of the large number of NGS users, none of the laboratories included a bio-informatician to interpret the results [28]. However, it must be noted that not all survey respondents in 2016 (n = 24) registered again in the next EQA scheme of 2017, as yearly participation is not required to demonstrate high quality performance. Therefore, we contacted those participants (n = 8) to ask for the reasons of refraining from participation. Two laboratories mentioned they only participate once every 2 years. One laboratory merged with another institute and therefore stopped RAS analyses, while another laboratory experienced bureaucratic issues with the payment of the registration fee. One participant did not agree with their awarded analysis score in 2016. The other three laboratories did not respond.

To interpret the error rates in the EQA schemes there are four points that need to be taken into account: (a) More samples were included containing a KRAS variant. However, no differences were observed when re-calculating the error rates based on the number of included samples per gene. (b) The ten distributed samples each had a different origin. Error rates were highest for a case containing the c.436G > A p.(Ala146Thr) variant (15.4%, n = 123) in 2016, and for the c.176C > A p.(Ala59Glu) variant (24.8%, n = 105) in 2017. The reason is that laboratories may be using an analysis method which may not include all necessary codons, consistent with previous EQA schemes [10]. (c) Many laboratories incorrectly analyzed the sample without any neoplastic cells, for which numbers and consequences have been previously described [13]. (d) Pre-defined scoring criteria differed as laboratories with an error in the online datasheet but correct written report received full points in 2017 and no points in 2016. However, this had no influence on the scores in this study.

As a conclusion, quality improvement projects such as the study described here are important to further improve the current high standards of biomarker testing in Europe. To avoid any issues with testing, laboratories need to work according to pre-defined procedures and document any changes. Laboratories need to be aware that reporting and monitoring of errors is required for quality improvement. To assure quality of biomarker analysis, it is thus clear that a holistic approach [29] is needed at all phases in combination with quality improvement projects within the laboratory and organized by EQA providers.