Pilot study on developing a decision support tool for guiding re-administration of chemotherapeutic agent after a serious adverse drug reaction

A total of 854 patients who were treated at the National Cancer Centre Singapore (NCCS) during the period 2004-2009 and experienced ADRs from chemotherapeutic agents were identified using spontaneous ADR reporting forms. Among these patients, 81 experienced serious ADR and 46 of them were rechallenged. These include 23 negative rechallenge cases and 23 positive rechallenge cases. Information about these 46 patients' demographics, relevant medical records and laboratory values were collected from the case notes and electronic medical systems.

The dataset was split into 3 sets: training, testing and validation. A total of 24 cases were randomly selected from the 35 cases that occurred between 2004 to 2008 to form the training set for developing the Naïve Bayes model. All the 35 cases were used as the testing set to test the performance of the model during the GA attribute selection process. The 11 cases that occurred during 2009 were used as an independent validation set to validate the model. These 11 cases were not used during the development of the model or during attribute selection.

Patient's attributes that may have potential associations with rechallenge status and which were readily available were collected for the 46 patients. The attributes, given in Table 1, can be broadly categorized into the following groups: (1) patient demographics, (2) medical conditions, (3) medications usage, (4) laboratory parameters. Genetic variables were not included in the study as genetic testing is not commonly conducted in NCCS.

Discrete categories of attributes were derived from some of the continuous attributes. For example, age was expressed as a continuous variable and a discrete variable with 2 categories: elderly (65 years old and above) and non-elderly (less than 65 years old) [6]. The number of concurrent medications was also categorized into a binomial attribute to test for possible associations between polypharmacy and rechallenge status. The definition of "more than 5 concomitant drugs" for serious polypharmacy [7] was adopted in this study. In addition to the presence of comorbidities, the number of comorbidities for each patient was also calculated. These comorbidities include hypertension, diabetes, hyperlipidemia, psoriasis, gastroestrophageal reflux disease (GERD), asthma and other allergic disorders. Other additional attributes constructed include the class of chemotherapeutic agents, types and malignancy of cancer, and types of ADRs experienced.

Laboratory parameters used include full blood count (white blood cell, red blood cell, platelet, neutrophil, lymphocyte, monocyte, eosinophil and basophil levels), renal indices (serum creatinine) and liver indices (aspartate aminotransferase, alanine aminotransferase and alkaline phosphatase). These are routinely analyzed for all patients in NCCS and the laboratory panels were obtained from the most recent measurements before the reported ADR incident. Additional attributes representing the presence or absence of abnormal laboratory parameters were also constructed.

The total number of attributes used for data mining was 53. Detailed information on these 53 attributes is provided in Additional File 1: Appendices 1 to 3.

GA was used to rank and select attributes that are useful for predicting patients' rechallenge status. Figure 1 shows an overview of the GA attribute selection process. The process involved creating multiple 'generations' of attribute subsets. Each attribute subset contained between 1 to 53 attributes that were initially randomly chosen. During each 'generation', a Naïve Bayes model was developed using the training set for each of the 80 attribute subsets in the population. A fitness value was determined for each model by using the testing set to compute the area under the receiver operating characteristic curve (AUC). The roulette wheel method was used as the selection operator for reproduction and population replacement. Crossover and mutation rates of 95% and 5% were used respectively to construct new attribute subsets for the next 'generation'. This process was repeated until 100 'generations' was reached. The GA attribute selection process was repeated 5 times using different starting 'generation' of attribute subsets. The best attribute subset with the highest AUC from these 5 runs was chosen. The clinical predictors were then selected from this attribute subset by expert assessment of the Naïve Bayes distribution table and clinical significance of the predictors. The final Naïve Bayes model was then developed using the 35 cases that occurred between 2004-2008 and the selected clinical predictors.

Naïve Bayes is a simple probabilistic classification algorithm based on Bayes' theorem and has the assumption of conditional independence of the predictive attributes [8, 9]. Given a set of patient attributes, F = {F₁, F₂... F_n}, the posterior probability of a patient having negative rechallenge (NR) under the assumption of independence of the attributes can be computed as follows:

The performance of the model was measured using AUC, which is frequently used to evaluate prediction models in the biomedical informatics field [10, 11]. In addition, the sensitivity and specificity of the model were calculated. Sensitivity refers to the proportion of patients with negative rechallenge who are predicted to have negative rechallenge. Specificity refers to the proportion of patients with positive rechallenge who are predicted to have positive rechallenge.

The set of clinical predictors identified by GA and refined by expert assessment is listed in Table 2. Patients' rechallenge status was found to be associated with age. The mean age for positive rechallenge cases were higher (55.55 ± 14.18) compared to negative rechallenge cases (47.71 ± 14.87). The type of chemotherapeutic agents administered was also found to be a significant clinical predictor, with patients who were given carboplatin, paclitaxel, docetaxel and cetuximab being more likely to have positive rechallenge. The mean albumin level for patients with positive rechallenge was 30.9 g/l, compared to 35.7 g/l for patients with negative rechallenge. Mean platelet and red blood cell (RBC) levels were found to be lower in patients who have positive rechallenge. Patients with abnormal levels of alanine aminotransferase (ALT), alkaline phosphatase (AP) and white blood cell (WBC) were more likely to have positive rechallenge.

The Naïve Bayes model developed using the 35 cases that occurred between 2004-2008 and the selected clinical predictors had an AUC of 0.767 for the validation set.

A platform independent version and an online version of the model (PaDEL-Rechallenge) is available at http://​padel.​nus.​edu.​sg/​software/​padelrechallenge​. This will facilitate independent validation of the model by clinicians.

It is important to note that the identified clinical predictors in this study were only found to be associated with rechallenge status. However, association does not imply causality. It is also essential to note that most of these were weak associations, but when considered together in the Naïve Bayes model, they were found to be useful for predicting patients' rechallenge status. Detailed discussion on the individual predictors can be found in Additional File 1: Appendix 4.

Since there are no similar studies that develop models for predicting patients' rechallenge status, we will assess the performance of our model by making tentative comparisons with other models developed in other biomedical fields. A Naïve Bayes and Radial Basis Function method for predicting implantation potentials of IVF embryos reported superior performance with an AUC of 0.712 [12]. Another model that used artificial neural network model to differentiate between patients with and without prostate cancer had an AUC range of 0.77 to 0.81 [13]. Thus, our Naïve Bayes model with an AUC of 0.767 can be considered to have acceptable prediction performance.

It is important to note that the rechallenge status of those patients who were not rechallenged will never be known. Since our model was not trained using this group of patients, it is not justifiable to replace clinical judgement with our model for predicting the rechallenge status for all patients. Instead, a more suitable application for our model will be to assist in subsequent clinical decision making after clinicians had identified a group of patients who are likely to have negative rechallenge.

The 13 serious ADR cases that occurred in 2009 will be used to illustrate the potential usefulness of our model for this type of application. Initial clinical judgement identified 11 cases as potentially negative rechallenge cases. Out of these, 6 were negative rechallenge cases and 5 were positive rechallenge cases. Thus initial clinical judgement had a sensitivity of 100% and specificity of 0%. Our Naïve Bayes model can be used to improve the prediction accuracy of the clinicians by providing a score for each case. A threshold for the score can be set and patients with scores above or below this threshold will be predicted by the model as potential negative or positive rechallenge cases respectively. Clinicians can choose different thresholds for the score according to their treatment objectives for the patients. For example, a low threshold is more important for patients who are undergoing curative treatment so that they are not deprived of a useful drug treatment. This is a significant issue in chemotherapeutic treatment because there are limited choices of effective chemotherapeutic drugs available to the patients. Thus, the benefits from using the first line drugs usually outweigh the potential risks caused by any serious ADRs. The sensitivity and specificity of our model using 0.01 as the threshold is 100% and 20% respectively. Conversely, a high threshold would be more useful for patients undergoing palliative chemotherapeutic treatment as the key priority is to prevent them from experiencing unnecessary serious ADRs caused by rechallenge. A threshold of 0.8 would result in a sensitivity of 67% and specificity of 80% for our model.

Despite analyzing 6 years worth of data, the size of the dataset used to develop and validate our model is rather small. This is due to the limited number of ADR cases reported and collected in NCCS. Thus, this study is only a pilot study and there is a need for further validation of the accuracy and reproducibility of this model. A study is currently ongoing to validate the model using 2010 to 2014 data. In addition, a visual aid will be added to improve the interpretability of the results by clinicians. Additional discussion on other limitations of this study can be found in Additional File 1: Appendix 4.