Development and validation of a patient-reported questionnaire assessing systemic therapy induced diarrhea in oncology patients

In the first step of hypothesizing the conceptual framework, the population of interest and intended application of use were conceptualized by the investigators. Item generation was completed from a literature review, interviews with clinicians and patient interviews. A literature search was conducted to determine published diarrhea assessment tools used in any patient population, and to identify potential bowel assessment dimensions that can be used to construct the assessment tool. The search was conducted in PubMed and EMBASE using [diarrhea AND assess* AND question*] OR [diarrhea AND clinical assessment tool*] as MeSH words. References of selected papers were scanned for additional tools that may not have surfaced from the search. The literature search was also extended to include any unpublished tools that other institutions or associations may have developed and made available for clinical use. The search was limited to studies in the English language. The tools were initially selected by the primary author (ML) and subsequently reviewed by the last author (CDA). Studies that provided only general guidelines, criteria or algorithm of diarrhea assessment rather than the questions from a tool were excluded.

An expert panel of oncology healthcare practitioners reviewed the findings from the literature review to select bowel assessment dimensions important for bowel habit assessment, develop questions for each dimension that clinicians collectively perceived as important for bowel habits assessment and agree upon appropriate wording and format of questions. Furthermore, the panel reviewed how well the questions individually and together evaluated the presence and severity of diarrhea. This process was used to develop the preliminary framework, as well as its face validity and construct validity from the clinicians’ perspective.

To adjust the preliminary framework, the investigators conducted patient interviews to evaluate face and content validity of the preliminary instrument from the patient perspective. Actively treated patients over eighteen years of age with a cancer diagnosis who had a history of STID within the last seven days were eligible to be interviewed. Patients experiencing diarrhea before starting treatment, had a gastrointestinal stoma or had cognitive impairment were excluded. A series of questions were asked regarding each item by the primary author to elicit information about patients’ understanding of the questions’ meaning, clarity of questions, criteria perceived to be important or unimportant for the assessment of STID and other questions that should be added or excluded from the questionnaire. Face validity was evaluated by questions like, “Would these questions be important for us to better understand your diarrhea?” while content validity was evaluated by questions like, “What do you think this question is trying to ask you?” Patients were instructed to answer the questions based on the preliminary instrument, which was reviewed during the interview.

Answers were transcribed and compared by two investigators to identify themes that would be used for revisions of the draft STIDAT. The authors performed thematic analyses by finding repetitions, similarities and differences within the answers of the patient interviews. [18] More patients were recruited if additional ideas were elicited from the most recent interview. Once saturation has been reached, no more patients were recruited and all revisions were made to the preliminary instrument. The final version was subsequently used for internal validation in the validation phase.

Given the ratio of observations to each item is generally accepted to be at least 5 to 1 to prevent overfitting [19], the number of dimensions assessed in the STIDAT was expected to be ten based on the preliminary instrument, and a dropout rate of 40%, the number of patients who needed to be recruited was calculated to be 70 and the number of patients who needed to complete the validation phase was 50.

Upon completion of data collection, results of the STIDAT were compared with results of the diaries and EORTC QLQ-C30 for each patient. Internal consistency was evaluated by calculating Cronbach’s alpha. A questionnaire with an alpha coefficient of greater than 0.7 was interpreted to have good internal consistency. Furthermore, inter-rater diagnostic reliability was assessed by determining Fleiss’ kappa for binary parameters of bowel habit assessment, Pearson’s coefficient of correlation for continuous parameters and Kendall’s tau for ordinal parameters. A Fleiss’ kappa of at least 0.70 suggests acceptable inter-rater reliability among binary parameters, a large value of Pearson’s coefficient of correlation represents good correlation of continuous variables, and a high Kendall’s tau signifies high probability of concordant pairs of ordinal data. Concurrent validity of the STIDAT was evaluated by using Chi-Square test to compare STID occurrence during treatment reported in the STIDAT and EORTC QLQ-C30.

A weighted scoring system of the STIDAT was constructed using exploratory factor analysis (EFA) using the minimal residual method using “Promax” as the oblique rotation method. [20, 21] The number of factors were initially determined using eigenvalues greater than one method and confirmed using parallel analysis, with eigenvalues generated using principal components analysis (PCA). EFA was used to determine the total weight that each dimension had on the STIDAT’s ability to detect the occurrence of diarrhea. The score of each dimension was calculated by multiplying the weight with the value that corresponds to the result of its STIDAT question(s) and the total STIDAT score was determined by summing the scores of each dimension.

The median STIDAT scores of patients with and without diarrhea, as well as those by each level of patient-reported STID severity were determined. The Welch’s T-test was used to determine whether STIDAT scores of patients with and without diarrhea were significantly different from each other. To evaluate the predictive ability of the STIDAT, a receiver-operating characteristic (ROC) curve was constructed to determine the threshold (minimum STIDAT score) that predicts for STID occurrence with the highest possible values of sensitivity and specificity. The corresponding values of threshold score, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) are also reported. All statistical analyses were performed using R (version 3.3.1).

As a post-hoc analysis to determine how patients perceive severity of diarrhea, we investigated whether there were any differences in quality of life among patients with no diarrhea, mild diarrhea, moderate diarrhea and severe diarrhea. Multiple comparisons were adjusted using the Bonferroni correction method. Furthermore, we determined whether there was a difference in diarrheal episodes among patients with different severities of diarrhea using the Dunnett-Tukey-Kramer multiple comparison test that was adjusted for unequal variances and unequal sample sizes.

From the 16 tools uncovered from the literature review (Fig. 1), the expert panel determined the following dimensions to be relevant to assessing the presence and severity of diarrhea: patient’s perception of having diarrhea, daily number of bowel movements, daily number of diarrhea episodes, use of antidiarrheal medications, the presence of urgency, abdominal pain, abdominal spasms, fecal incontinence, patient’s perception of diarrhea severity, and quality of life. Quality of life was evaluated based on the impact of having diarrhea on the patient’s social life, mood, family life, ability to perform daily activities of living and energy level. In addition, the expert panel had included a checklist of foods and beverages that were thought to influence bowel habits by either helping or exacerbating ongoing diarrhea, with the thought that the presence of such items in patients’ diets may influence the severity of their diarrhea. A list of additional symptoms indicative of severe diarrhea, including the presence of nausea, fever and melena, was included in the questionnaire, as the members perceived these symptoms to be important in assessing severity of diarrhea.

Five patients were interviewed; saturation was achieved from their collective data. Patients generally understood why the questions were asked and correctly interpreted that they were to assess different “aspects” of their diarrhea, therefore confirming the tool’s face validity. With respect to the clarity of questions, all patients perceived the wording to be clear and provided interpretations akin to the intent of each question. Patients agreed unanimously that the questionnaire had included all the dimensions required to evaluate the incidence and severity of STID and was thus perceived to be comprehensive in STID assessment. All patients answered that each question was important to assess for the presence and severity of diarrhea, and that they would not eliminate any question from the questionnaire. Furthermore, none of the patients had additional questions to the questionnaire.

In addition, 80% of patients agreed that the definition of diarrhea was mostly driven by the presence of watery stool rather than by stool frequency. Patient 1, who reported having 5 to 6 bowel movements daily, commented that he “would have diarrhea every day if the definition was based on number of stools”, but understands that “they [were] still normal bowel movements given that [he] has normal stool each time”. Furthermore, 2 patients commented that evaluation of stool form should include forms associated with constipation in addition to diarrhea.

While the panel members initially perceived a list of foods and beverages that help or exacerbate diarrhea included in the questionnaire to be helpful to patients to evaluate their bowel habits, most patients found it difficult determining how their diet could have influenced their bowel habits. Furthermore, patients also found the checklist of symptoms of severe diarrhea to be confusing with other treatment-related toxicities. Therefore, both sections were eliminated from the preliminary instrument. The final instrument is illustrated in Fig. 2.

Eighty–nine patients were enrolled and 52 patients completed the STIDAT (Table 1). 12 withdrew consent, 20 did not complete the questionnaire and 5 were lost to follow-up.

Table 2 summarizes the reliability and consistency of the STIDAT. The questionnaire has an overall raw Cronbach’s alpha of 0.78 (95% CI = 0.74–0.83). Regarding the reliability of STIDAT, high kappa values were calculated between the questionnaire and diaries for the patient’s perception of diarrhea (κ = 0.934) and a positive history of antidiarrheal medication use (κ = 0.952) and a low kappa value was calculated for the dimension of abdominal spasms (κ = 0.0455). Good correlation between the questionnaire and diaries was noted for the dimensions of diarrhea severity and quality of life. Furthermore, STID occurrence reported in the STIDAT and EORTC QLQ-C30 are significantly related to each other (χ² = 79.71, p < 0.05).

A breakdown of signs and symptoms based on the severity of diarrhea (no to severe diarrhea) is available in Table 3. Of note, the mean overall quality of life scores (calculated as the average of the five quality of life dimensions measured in the STIDAT) were 8.95 (95% CI = 8.50–9.40) in the no diarrhea group, 8.12 (95% CI = 7.38–8.87, p = 0.6164) in the mild diarrhea group, 7.1 (95% CI = 6.18–8.02) in the moderate diarrhea group and 4.44 (95% CI = 1.53–7.35) in the severe diarrhea group.

Using the Dunnett-Tukey-Kramer test, it was determined that there were significant decreases in quality of life experienced by patients with moderate diarrhea (−1.86, 95% CI = −3.05, −0.67, p < 0.001) and severe diarrhea (−3.64, 95% CI = −5.91, −1.38, p < 0.001) compared to patients with no diarrhea. There was a trend towards a decrease in quality of life in patients with severe diarrhea compared to those with mild diarrhea (−2.82, 95% CI = −5.3, −0.37, p = 0.017). There was no difference in quality of life between mild and no diarrhea, mild and moderate diarrhea, and moderate and severe diarrhea patients (Table 4).

Using the eigenvalues greater than one rule, the number of factors was determined to be 4. This was confirmed with the parallel analysis approach. Using the minimal residual method with Promax rotation, the factor loadings of each dimension to each factor were generated. The dimensions were grouped into four clusters. The dimensions of patient’s report of the occurrence and severity of diarrhea, and the presence of urgency were related conceptually by the patient’s perception of having diarrhea, or Cluster 1. Cluster 2 – frequency of diarrhea – comprised of total number of bowel movements, number of diarrhea episodes, quality of life and medication use. Fecal incontinence is the sole dimension of Cluster 3. Finally, abdominal spasms and abdominal pain were grouped together as abdominal symptoms – Cluster 4. Figure 3 and Table 5 shows the relationships of the dimensions assessed by the STIDAT.

Table 6 is the weighted scoring system from EFA using an eigenvalue of 4, as per the initial PCA. The multiplication of each component score with its corresponding component weight, followed by the addition of an adjustment factor, yields a total score. Patients without diarrhea had a mean score of 0.72 (SD = 0.21), while patients with diarrhea had a significantly higher score of 2.25 (SD = 0.73) (t = 15.04, 95% CI = 1.33–1.73).

ROC analysis yielded an area-under-the-curve (AUC) of 0.977 (95% CI = 0.941–1.013) at the cut-off score of 1.35 in predicting the occurrence of diarrhea (Fig. 4). Sensitivity, specificity, PPV and NPV were 0.982, 0.966, 0.964 and 0.983, respectively.

Table 7 and Fig. 5 show STIDAT scores stratified by diarrhea severity. Median scores of no, mild, moderate and severe diarrhea cohorts were 0.64, 1.54, 2.38 and 3.63, respectively. Based on the ranges of scores for each severity (Table 7 and Fig. 5), clinicians may categorize scores from 0 to 1.1 as no diarrhea, greater than 1.1 to 2 as mild diarrhea, greater than 2 to 3 as moderate diarrhea and greater than 3 as severe diarrhea.

The patient interviews revealed that most patients defined diarrhea based on the presence of watery stool without mention of increased stool frequency, which is a departure to how the CTCAE assesses diarrhea mostly through an increase in stool frequency. This discrepancy among the clinician and patient definitions of diarrhea may in part account for the difference in the evaluation of diarrhea severity by patients and the clinicians via the CTCAE. This finding suggests the need to choose one definition of diarrhea that patients use when reporting the occurrence and severity of STID to ensure clarity and standardization in the assessment of STID. To address this discrepancy, the STIDAT uses the patient definition in which diarrhea was identified based on the presence of watery stool.

The STIDAT was observed to have good internal consistency with a raw Cronbach’s alpha of 0.78, which far exceeded 0.7. It has similar degrees of internal consistency with other validated diarrhea assessment tools used in other disease states, including the SSC-HIV (α = 0.749), Irritable Bowel Syndrome Quality of Life (α = 0.963), DAPonDEN (α = 0.67–0.717) and Birmingham IBS symptom questionnaire (α = 0.74–0.90). [22‐24] It also had good inter-rater reliability, particularly in the dimensions of patient’s report of the occurrence and severity of diarrhea, urgency, fecal incontinence and antidiarrheal medication use, suggesting that the STIDAT is particularly robust in evaluating diarrhea based on those dimensions. The STIDAT exhibited poor reliability in reporting abdominal spasms, suggesting that patients may not understand what spasms are, or are not able to differentiate between spasms and abdominal discomfort.

The weighted scoring system is based on how much each component contributes towards the incidence of diarrhea as predicted by EFA. The score of each question corresponding to its component would be multiplied by its component weight. Scores of binary variables (yes/no questions) would be either 0 (no) or 1 (yes), scores for diarrhea severity vary from 0 (no diarrhea) to 3 (severe diarrhea) and scores of questions pertaining to episode frequency would be an integer starting from 0. The quality of life score is the average of the five scores corresponding to the five dimensions of quality of life in the STIDAT. The component score for quality of life has a negative sign because the quality of life questions are ranked in reverse. The total STIDAT score is the sum of all component scores and the adjustment factor (0.48), which offsets the minimum quality of life score that a patient may reach to prevent a negative STIDAT score.

The STIDAT dimensions were grouped based on their relationships with each other. A heightened increased sense of urgency to have a bowel movement increases the perception of having diarrhea that may be more severe, hence the potential rationale for urgency, patient report of diarrhea occurrence and severity to be connected via Cluster 1. While it is apparent that total number of bowel movements and number of diarrhea episodes are related to Cluster 2, medication use can reduce bowel movement frequency and quality of life can be negatively affected from increased frequency of bowel movements. Therefore, medication use and quality of life are also grouped together into Cluster 2. While fecal incontinence is a diarrhea symptom along with abdominal spasms and pain, the former may happen independently of diarrhea occurrence. This is observed by the low component weight of fecal incontinence in the scoring system in determining the presence and severity of diarrhea in patients.

A score of 1.35 is the minimum for the STIDAT to detect whether a patient objectively has diarrhea or not with excellent sensitivity and specificity. Furthermore, the STIDAT scoring system can identify the severity level of STIDAT of patients. There was a distinct separation of scores in patients of each group of diarrhea severity. While there is some overlap between mild and moderate diarrhea, it is sufficiently small such that the potential error in differentiation is small. Because the management of mild and moderate diarrhea is similar, the potential harm on the patient is not expected to be significant. Therefore, this STIDAT can accurately identify the occurrence and severity of diarrhea through a comprehensive assessment and appropriate weighting of various bowel habit dimensions.

The development of the STIDAT also highlighted the significant differences in quality of life among patients with different severities of diarrhea. Patients with severe diarrhea had significantly worse quality of life compared to patients with no or mild diarrhea. While there was no difference among patients with moderate and mild diarrhea, a significant decrease in quality of life was observed among patients with moderate and no diarrhea. Therefore, clinicians should aim to improve diarrhea management of patients with moderate to severe diarrhea with the therapeutic goal to decrease it to at least mild diarrhea, not simply to prevent dehydration and potential treatment delays, but to maintain patients’ quality of life throughout their treatment journey.

The STIDAT is superior to the CTCAE as an oncology-based clinical bowel assessment tool. As a patient-reported questionnaire, it circumvents potential underreporting or inaccuracies in clinician-based assessments in the occurrence and severity of diarrhea. This increases the likelihood of patients’ STID in being assessed and managed appropriately, which may decrease negative consequences associated with STID, including dehydration-associated hospitalizations, treatment dose reductions or delays, and decreases of quality of life associated with undermanaged STID. It assesses beyond stool frequency and presence of fecal incontinence to include frequency of diarrhea episodes, quality of life, stool form and other symptoms of diarrhea to provide comprehensive insight on the presence and severity of diarrhea. This enables clinicians to better understand the toxicity and provide more appropriate management for it. It also uses a weighted scoring system to objectively assess the presence and severity of STID using patient-reported data, which contrasts sharply from the arbitrarily set stool frequency intervals that define each CTCAE severity grade. The use of a weighted scoring system allows for objective stratification of diarrhea severity that considers multiple dimensions. In addition to providing a total score that characterizes the severity of diarrhea, it provides a score for each item to provide insight for what may drive the score, which the CTCAE cannot do with an overall grade. The STIDAT was also developed using the presence of watery stool as the definition of diarrhea – as per patient interviews – compared to the presence of increased stool frequency in the CTCAE. The use of stool form to define diarrhea seems effective in detecting diarrhea occurrence, as reflected by its excellent specificity and sensitivity. Furthermore, the STIDAT has undergone a formal development and validation process, unlike the CTCAE.

This study has some limitations. The validation phase was conducted in two Centres with a small, albeit varied, patient population that lacked patients with hematologic malignancies. While around 60% of recruited patients had completed the questionnaire, these patients may be particularly motivated and self-efficacious and therefore, the use of the questionnaire may be biased towards such individuals. It is not designed to be used in patients with an ostomy; thus, the STIDAT cannot assess the bowel habits of these patients. It is restricted for patients with adequate working English and cognitive ability to self-report his or her bowel habits. Finally, this assessment requires external validation to further characterize validity and predictive ability. Future validation efforts will be focused on including more centres, amassing a larger sample size, including hematology patients and modifying the questionnaire to accommodate patients with ostomies and patients speaking other languages.