Development of computerised adaptive testing (CAT) for the EORTC QLQ-C30 dimensions – General approach and initial results for physical functioning

https://doi.org/10.1016/j.ejca.2010.02.011Get rights and content

Abstract

Background

Health-related quality of life (HRQOL) questionnaires should ideally be adapted to the individual patient and at the same time scores should be directly comparable across patients. This is achievable using a computerised adaptive test (CAT). Basing the CAT on an existing instrument enables measurement within an established HRQOL framework and allows backward-compatibility with studies using the original instrument. Because of these advantages the EORTC Quality of Life Group (QLG) has initiated a project to develop a CAT version of the widely used EORTC QLQ-C30.

Methods

We present the EORTC QLG’s strategy for developing a CAT. For each dimension of the EORTC QLQ-C30 our approach includes literature search and conceptualisation, formulation of new items, expert and patient evaluations, field-testing, and psychometric analyses of the items. The strategy is illustrated with the initial results of the development of CAT for physical functioning (PF).

Results

We identified 975 PF items in the literature. Of these, 407 items were deemed relevant, i.e. measured one of the PF aspects measured by the QLQ-C30. Based on these items we developed 86 new items. Review by the EORTC CAT-project group reduced this to 66 items. Based on expert and patient evaluations several items were revised and the list was further reduced to 51 items.

Conclusions

Based on the findings for PF, we believe that our approach will generate item pools that are relevant and appropriate for cancer patients. These will form the basis for a backward-compatible CAT assessing the HRQOL dimensions of the EORTC QLQ-C30.

Introduction

The primary source of information about patients’ health-related quality of life (HRQOL) is self-report questionnaires, also termed patient-reported outcomes (PROs). These instruments have typically been developed and implemented using classical methods like sum scoring (adding the scores to individual items into scales). However, these classical methods have some limitations. For example, all patients have to answer the same set of items for sum scores to be comparable. This means that PRO instruments often constitute a compromise between optimal measurement (requiring longer instruments) and reasonable response burden (requiring shorter instruments).

In recent years there has been an increasing interest in methods based on item response theory (IRT)1, 2 for measuring HRQOL (see e.g.3, 4, 5, 6, 7, 8, 9, 10). IRT is a statistical framework for assessing the characteristics of the items in multi-item scales. IRT models estimate among other things the ‘difficulty’ of each item. For example, taking a long walk is more demanding/difficult than taking a short walk. Hence, asking patients with good physical functioning about taking a long walk may be highly informative while for patients with poor functioning an item about taking a short walk may be more informative. IRT-based methods take this into account.

One of the major advantages of IRT methods compared to classical methods, and one of the reasons for the interest in these methods, is that when a set of items has been calibrated (estimated) to an IRT model all scores based on any subset of the items are on the same metric. That is, even if two patients answer different subsets of items from the same item pool, their scores are directly comparable. This unique feature means that a questionnaire can be adapted to the individual to optimise the measurement properties, yet at the same time comparability of scores are maintained across patients. This possibility for adapting the instrument is fully utilised in a computerised adaptive test (CAT)11 to construct individualised instruments: based on the responses to the preceding items, a computer programme evaluates which item should be asked next to obtain maximal information. The additional items are administered until a predefined level of information (i.e. precision) has been reached or until a predefined number of items has been administered. Fig. 1 shows two simple examples of how a CAT measuring physical functioning could proceed. If the patient reports few problems on an item, the next item will concern a more demanding task, while if severe problems are reported the following item will concern a less demanding task. In this way the questionnaire is individualised, using the most informative items for each patient. In contrast, traditional questionnaires ask the same items to all, and hence, generally need more items to obtain the same level of precision.

CAT measurement has several advantages compared to traditional questionnaires including: increased measurement precision and/or reduced response burden, increased flexibility as the questionnaire can be adapted to the individual study or patient, avoidance of asking uninformative questions, and immediate calculation and presentation of results.

Because of the clear advantages of CAT measurement a number of research groups are developing CAT’s for measurement of HRQOL, including the Patient-Reported Outcomes Measurement Information System (PROMIS)12, QualityMetric13, and the European Palliative Care Research Collaborative.14

The European Organisation for Research and Treatment of Cancer Quality of Life Group (EORTC QLG) has initiated a CAT-project for the HRQOL dimensions measured by the EORTC Quality of Life Questionnaire (EORTC QLQ-C30).15 The EORTC QLQ-C30 is one of the most widely used quality of life questionnaires in cancer research.16, 17 It consists of 30 items measuring 15 aspects of HRQOL: five functional measures, nine symptom measures and one measure of overall health/quality of life.18 At present the EORTC QLQ-C30 exists only in a traditional version where all patients are asked the same 30 items.

The aim of the EORTC CAT-project is to measure the same 15 HRQOL dimensions as measured with the QLQ-C30, but with higher efficiency and precision. This requires new items supplementing the existing items, filling ‘gaps’, e.g. new items enabling assessment of patients in very poor condition. Initial investigations indicated that it might be difficult to develop enough relevant but different items for ‘overall quality of life’. Therefore, the project focuses on developing CAT for the other 14 HRQOL dimensions of the QLQ-C30. Basing the CAT on the QLQ-C30 ensures backward-compatibility with the substantial literature of studies using the QLQ-C30. The CAT instrument aims to measure the same well-validated and well-known HRQOL dimensions with significantly improved precision. Most other CATs do not have such a direct link to an existing instrument. Instead they are developed ‘from scratch’ and are unconstrained by existing conceptual frameworks, resulting in a completely new instrument. This gives greater freedom in the development process, but may also result in unfamiliar measures.

The purpose of the current paper is to provide an overview of the methods used to develop the new EORTC QLQ-C30 CAT instrument, illustrated by the development of CAT for physical functioning (PF).

Section snippets

Methods

The development of the EORTC CAT is conducted by members of the EORTC QLG. This international CAT-project group includes oncologists, psychologists, statisticians, and others with considerable experience in developing HRQOL instruments.

The development of the item pools forming the basis for the CAT can be divided into four phases: (1) literature search, (2) operationalisation (selection and formulation of items), (3) pre-testing (patient interviews), and (4) field-testing (data collection and

Results

We illustrate the methodology using the results of the first three phases of the development of the item pool for PF. The field-testing is in progress and the results of this and the psychometric analyses (phase 4) will be described in detail in a separate paper.

Discussion

CAT measurement has several advantages compared to traditional questionnaire measurement. In particular, it can improve measurement precision without increasing the response burden for the patients. To utilise these new methods to improve the measurement of cancer patients’ HRQOL the EORTC QLG has initiated a project to develop CAT for the widely used EORTC QLQ-C30 questionnaire.

The EORTC CAT development can be divided into four phases: literature search, operationalisation, pre-testing, and

Sources of support

The study was funded by grants from the EORTC Quality of Life Group.

Conflict of interest statement

None declared.

Acknowledgement

This study was funded by grants from the EORTC Quality of Life Group.

References (32)

  • M. Orlando-Edelen et al.

    Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement

    Qual Life Res

    (2007)
  • H. Wainer

    Computerized adaptive testing: a primer

    (2000)
  • The Patient-Reported Outcomes Measurement Information System (PROMIS). <http://www.nihpromis.org> [accessed June...
  • Dynamic Computer Adaptive (DYNHA®) System. <http://www.qualitymetric.com/products/dynhadetails.aspx> [accessed June...
  • S. Kaasa et al.

    Symptom assessment in palliative care: a need for international collaboration

    J Clin Oncol

    (2008)
  • N.K. Aaronson et al.

    The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology

    J Natl Cancer Inst

    (1993)
  • Cited by (55)

    View all citing articles on Scopus
    1

    Grant National Science Council, Taiwan, No. 95-2314-B-002-266-MY2, 97-2314-B-002-020-MY3.

    View full text