Equivalence testing of a newly developed interviewer-led telephone script for the EORTC QLQ-C30

The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life-Core Questionnaire (QLQ-C30) [1] is currently one of the most widely used self-report measures of health-related quality of life (HRQOL) for cancer patients [2]. Patient reports using the EORTC QLQ-C30 are carried out using paper-and-pencil administration or through electronic methods. However, to increase the accessibility of the questionnaire across different research settings and populations (e.g. in patients that could be at risk of exclusion because of illiteracy), and to minimise the need for otherwise unnecessary clinic trips, the EORTC Quality of Life Group (QLG) set out to develop and validate a voice script for phone administration of the questionnaire. The voice script would incorporate all relevant elements of the QLQ-C30, along with additional instructions and text, where necessary, to facilitate its interviewer-led administration.

The EORTC QLQ-C30 was first released in 1987 by Aaronson et al. [3], and underwent further revisions leading to the development of its third version [1], which is still in use today. Comprised of 30 different items (questions), it is made up of eight multi-item functional (physical, role, emotional, cognitive, and social) and symptom (fatigue, pain, and nausea) scales, one global health status and quality of life (QOL) scale, and six single items (dyspnoea, insomnia, appetite loss, constipation, diarrhoea, and financial difficulties). Covering the majority of the core symptoms recommended for patient-reported outcome (PRO) measurement in cancer clinical trials [4], it is available for use in over 110 language versions, having undergone extensive testing to demonstrate its psychometric [5] and cultural [6] validity.

In addition to its frequent use in cancer research and clinical trials [2, 7], the QLQ-C30 is being increasingly used for monitoring purposes in clinical practice [8]. By providing a direct means of measuring core symptoms and issues from the patient’s perspective, the QLQ-C30 provides clinically meaningful information, distinct from that offered by clinical markers and clinicians’ ratings [2, 7, 9]. In 2018, the EORTC QLG published guidelines to help facilitate the use and migration of EORTC questionnaires into electronic PRO (ePRO) formats (e.g. computer, tablet) [10]. A computerised adaptive testing (CAT) version of the QLQ-C30, the EORTC CAT Core [11], is also available, and consists of dynamic item banks which correspond with the QLQ-C30’s 14 functional and symptom domains.

The purpose of this study was to pilot test the provisional QLQ-C30 phone script through cognitive debriefing interviews to ensure its acceptability and relevance, amending it if needed, and to subsequently validate the QLQ-C30 phone-administered version by carrying out equivalence testing between the paper and phone administration modes in a population of patients actively undergoing cancer treatment. An intraclass correlation coefficient (ICC) of ≥ 0.70, the recommended threshold to demonstrate equivalence between various modes of administration, was employed in this study for the purpose of equivalence testing [12]. Previous research supports the use of ICC ≥ 0.70, as demonstrated in studies by Lundy et al. [13, 14], in which an interactive voice response (IVR) version of the QLQ-C30 was developed. Similarly, in an equivalence study aimed at comparing tablet computer, IVR, and paper-based administration of the PRO-CTCAE [15], the degree of mode equivalence was assessed using ICC ≥ 0.70.

Although previous work conducted by Lundy et al. demonstrated the equivalence of an IVR version of the QLQ-C30 to its paper administration [13, 14], this is the first project aimed at validating a voice script for phone administration of the QLQ-C30 by an interviewer. A considerable body of research comparing paper to screen-based (e.g. tablet, computer) administration of PROs has demonstrated high levels of reliability between both modes [16‐18] but less work has compared paper administration to auditory modes (e.g. IVR, phone interview). Still, the existing research suggests that equivalence can be established between paper and phone PRO administration [13, 15].

Patient recruitment and data collection, management, and analysis were subcontracted to contract research organisation (CRO) Mapi/ICON plc who provided a final report to the EORTC detailing the methodology and findings. Study approval was obtained in the United Kingdom (UK) by the Quorum Review independent review board.

This study aimed to develop and validate a voice script for phone administration of the QLQ-C30 and evaluate its equivalence to paper administration in a sample of patients actively undergoing cancer treatment. During pilot-testing, the voice script was deemed understandable and relevant with minimal comments received from patients.

Results from the final sample of patients included in the equivalence testing indicated good equivalence between paper and phone administration modes, with all total ICC scores above the 0.70 threshold, ranging from 0.72 to 0.90. In the evaluation of paper administration first, two ICC scores were found to be below the 0.70 threshold, for nausea and vomiting (ICC 0.55; 95% CI 0.24–0.76) and financial difficulties (ICC 0.60; 95% CI 0.31–0.79). When comparing differences in means at the domain score level, the differences were still well below 10 points for the comparison of both administration modes, suggesting minimal clinically meaningful differences despite the ICCs [25]. Failure to reach the 0.70 ICC threshold for nausea and vomiting may also reflect the possibility of more ambiguity surrounding the rating of nausea. While vomiting is a more concrete occurrence, and it is unlikely that a patient’s recollection would change over a 2-day timeframe, nausea may be subject to broader interpretation. Moreover, medications are generally readily available to patients, which help to resolve these symptoms on a day-to-day basis, thus indicating that those symptoms can change within a 2-day period. For financial difficulties, the potential source of discrepancy is less clear. There may have been an issue that was not detectable based on the scope of this study, or it may simply be due to random error or noise.

A more general limitation of using ICC to assess equivalence is that the absolute size of a given ICC is dependent on the variation observed within the sample. As such, minimal variation in nausea and vomiting and financial difficulties scores may have contributed to the lower ICCs. Still, the ICCs for nausea and vomiting and financial difficulties were well above 0.50 for paper administration first, indicating that they remain within the minimally acceptable range, especially since the total ICC scores for both scales were over 0.70. It is worth noting that the nausea and vomiting domain score has performed poorly in a previous test–retest study carried out by Hjermstad et al. [26], so there may be other factors influencing that scale, which were not identified in this study. Such factors could also account for the lower ICC score found for nausea and vomiting, when the paper version was administered first.

Differences in mean scores at the domain score level were uniformly minimal, suggesting that, overall, results from both administration modes were equivalent. The relatively short completion time of 8.6 ± 1.9 min for the voice script suggests that it can be integrated into a study protocol with relative ease and minimal patient burden.

Following guidelines from ISPOR’s PRO Mixed Modes Good Research Practices Task Force, and drawing on methodology used in similar PRO equivalence studies [13, 15], this study had a number of strengths. The randomised cross-over design helped to minimise the potential for bias in either one of the administration modes, and the inclusion criteria ensured that the voice script was evaluated and tested by patients for whom it would be relevant and feasible (i.e. those actively undergoing treatment, with the appropriate language level). The final sample of patients was diverse, and sufficiently well-balanced in terms of demographic and clinical characteristics, helping to ensure representativeness across patients and disease types. Analyses were also strengthened by the fact that there were no missing data in either the paper or phone-administered versions of the questionnaire, making the results more easily interpretable.

The decision to decrease the initial ICC threshold from ≥ 0.90 to ≥ 0.70 following the study amendment to include a second wave of testing, is well-supported by robust evidence in the literature, for which an ICC of ≥ 0.70 has also been used to evaluate equivalence in similar studies [13‐15].

Moreover, the total ICCs for both waves were largely similar. Significant differences were only observed for nausea and vomiting, which was lower in the first wave, and constipation, which was lower in the second wave of testing. Although the same recruitment procedures and inclusion criteria were applied, and demographic and clinical characteristics were largely comparable between groups, differences in gender and age distribution were found between the two waves of testing. In light of these differences, sensitivity analyses were carried out across all participants by age and gender. While most scores were similar across groups, differences were found in ICC scores for the nausea and vomiting and diarrhoea domain scores by group, with younger patients scoring lower on these domains compared to older patients. Males also scored lower than females on both the nausea and vomiting and physical functioning scales.

Despite these findings, when examining all other ICCs and comparing them across subgroups, no consistent pattern was identified which would support a potential correlation between lower or higher ICCs and the age or gender of patients. Moreover, the limited sample size makes it difficult to draw robust conclusions at the subgroup level. Factors other than age and gender may be related to the experience of disease and treatment, and may help to account for the differences observed; however, such interpretations are beyond the scope of this study, which is limited to the available demographic and clinical data.

Overall, sensitivity analyses showed that differences observed between the two waves of testing are minimal, thereby further supporting the equivalence of paper-and-pen and phone administration modes.

Results from this study support the equivalence of paper and phone administration modes of the QLQ-C30. In addition to its initial source language (UK English) development, the QLQ-C30 voice script is now available in multiple other languages, with more translations anticipated in the future. By providing an alternative means of questionnaire completion, the QLQ-C30 voice script helps to ensure that the questionnaire remains accessible in multiple formats across a wide range of patients.