Skip to main content
Erschienen in: Quality of Life Research 9/2018

05.05.2018

Grooming a CAT: customizing CAT administration rules to increase response efficiency in specific research and clinical settings

verfasst von: Michael A. Kallen, Karon F. Cook, Dagmar Amtmann, Elizabeth Knowlton, Richard C. Gershon

Erschienen in: Quality of Life Research | Ausgabe 9/2018

Einloggen, um Zugang zu erhalten

Abstract

Purpose

To evaluate the degree to which applying alternative stopping rules would reduce response burden while maintaining score precision in the context of computer adaptive testing (CAT).

Data

Analyses were conducted on secondary data comprised of CATs administered in a clinical setting at multiple time points (baseline and up to two follow ups) to 417 study participants who had back pain (51.3%) and/or depression (47.0%). Participant mean age was 51.3 years (SD = 17.2) and ranged from 18 to 86. Participants tended to be white (84.7%), relatively well educated (77% with at least some college), female (63.9%), and married or living in a committed relationship (57.4%). The unit of analysis was individual assessment histories (i.e., CAT item response histories) from the parent study. Data were first aggregated across all individuals, domains, and time points in an omnibus dataset of assessment histories and then were disaggregated by measure for domain-specific analyses. Finally, assessment histories within a “clinically relevant range” (score ≥ 1 SD from the mean in direction of poorer health) were analyzed separately to explore score level-specific findings.

Method

Two different sets of CAT administration rules were compared. The original CAT (CATORIG) rules required at least four and no more than 12 items be administered. If the score standard error (SE) reached a value < 3 points (T score metric) before 12 items were administered, the CAT was stopped. We simulated applying alternative stopping rules (CATALT), removing the requirement that a minimum four items be administered, and stopped a CAT if responses to the first two items were both associated with best health, if the SE was < 3, if SE change < 0.1 (T score metric), or if 12 items were administered. We then compared score fidelity and response burden, defined as number of items administered, between CATORIG and CATALT.

Results

CATORIG and CATALT scores varied little, especially within the clinically relevant range, and response burden was substantially lower under CATALT (e.g., 41.2% savings in omnibus dataset).

Conclusions

Alternate stopping rules result in substantial reductions in response burden with minimal sacrifice in score precision.
Literatur
1.
Zurück zum Zitat Ahmed, S., Ware, P., Gardner, W., Witter, J., Bingham, C. O. 3rd, Kairy, D., et al. (2017) Montreal Accord on patient-reported outcomes use series-paper 8: Patient-reported outcomes in electronic health records can inform clinical and policy decisions. Journal of Clinical Epidemiology, 89, 160–167.CrossRefPubMed Ahmed, S., Ware, P., Gardner, W., Witter, J., Bingham, C. O. 3rd, Kairy, D., et al. (2017) Montreal Accord on patient-reported outcomes use series-paper 8: Patient-reported outcomes in electronic health records can inform clinical and policy decisions. Journal of Clinical Epidemiology, 89, 160–167.CrossRefPubMed
2.
Zurück zum Zitat Ameur, H., Ravaud, P., Fayard, F., Riveros, C., & Dechartres, A. (2017). Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. Journal of Clinical Epidemiology, 84, 70–77.CrossRefPubMed Ameur, H., Ravaud, P., Fayard, F., Riveros, C., & Dechartres, A. (2017). Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. Journal of Clinical Epidemiology, 84, 70–77.CrossRefPubMed
3.
Zurück zum Zitat Broderick, J. E., DeWitt, E. M., Rothrock, N., Crane, P. K., & Forrest, C. B. (2013) Advances in patient-reported outcomes: The NIH PROMIS((R)) measures. EGEMS (Wash DC), 1, 1015. Broderick, J. E., DeWitt, E. M., Rothrock, N., Crane, P. K., & Forrest, C. B. (2013) Advances in patient-reported outcomes: The NIH PROMIS((R)) measures. EGEMS (Wash DC), 1, 1015.
4.
Zurück zum Zitat Health USDo, Human Services FDACfDE, Research, Health USDo, Human Services FDACfBE, Research, et al. (2006). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims: Draft guidance. Health and Quality of Life Outcomes, 4, 79.CrossRef Health USDo, Human Services FDACfDE, Research, Health USDo, Human Services FDACfBE, Research, et al. (2006). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims: Draft guidance. Health and Quality of Life Outcomes, 4, 79.CrossRef
5.
Zurück zum Zitat Noonan, V. K., Lyddiatt, A., Ware, P., Jaglal, S. B., Riopelle, R. J., & Bingham, C. O. 3rd, et al. (2017) Montreal Accord on patient-reported outcomes use series-paper 3: Patient-reported outcomes can facilitate shared decision-making and guide self-management. Journal of Clinical Epidemiology, 89, 125–135CrossRefPubMed Noonan, V. K., Lyddiatt, A., Ware, P., Jaglal, S. B., Riopelle, R. J., & Bingham, C. O. 3rd, et al. (2017) Montreal Accord on patient-reported outcomes use series-paper 3: Patient-reported outcomes can facilitate shared decision-making and guide self-management. Journal of Clinical Epidemiology, 89, 125–135CrossRefPubMed
6.
Zurück zum Zitat Petrillo, J., Cano, S. J., McLeod, L. D., & Coon, C. D. (2015). Using classical test theory, item response theory, and Rasch measurement theory to evaluate patient-reported outcome measures: A comparison of worked examples. Value Health, 18, 25–34.CrossRefPubMed Petrillo, J., Cano, S. J., McLeod, L. D., & Coon, C. D. (2015). Using classical test theory, item response theory, and Rasch measurement theory to evaluate patient-reported outcome measures: A comparison of worked examples. Value Health, 18, 25–34.CrossRefPubMed
7.
Zurück zum Zitat Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63, 1179–1194.CrossRefPubMedPubMedCentral Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63, 1179–1194.CrossRefPubMedPubMedCentral
8.
Zurück zum Zitat Cappelleri, J. C., Jason Lundy, J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36, 648–662.CrossRefPubMedPubMedCentral Cappelleri, J. C., Jason Lundy, J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36, 648–662.CrossRefPubMedPubMedCentral
9.
Zurück zum Zitat Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-42.CrossRefPubMed Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-42.CrossRefPubMed
10.
Zurück zum Zitat Cook, K. F., O’Malley, K. J., & Roddey, T. S. (2005). Dynamic assessment of health outcomes: Time to let the CAT out of the bag? Health Services Research, 40, 1694–1711.CrossRefPubMedPubMedCentral Cook, K. F., O’Malley, K. J., & Roddey, T. S. (2005). Dynamic assessment of health outcomes: Time to let the CAT out of the bag? Health Services Research, 40, 1694–1711.CrossRefPubMedPubMedCentral
11.
Zurück zum Zitat Choi, S. W., Grady, M. W., & Dodd, B. G. (2010). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 70, 1–17.PubMedPubMedCentral Choi, S. W., Grady, M. W., & Dodd, B. G. (2010). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 70, 1–17.PubMedPubMedCentral
12.
Zurück zum Zitat Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., Cella, D., et al. (2011). Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS(R)): Depression, anxiety, and anger. Assessment, 18, 263–283.CrossRefPubMedPubMedCentral Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., Cella, D., et al. (2011). Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS(R)): Depression, anxiety, and anger. Assessment, 18, 263–283.CrossRefPubMedPubMedCentral
13.
Zurück zum Zitat Christodoulou, C., Junghaenel, D. U., DeWalt, D. A., Rothrock, N., & Stone, A. A. (2008). Cognitive interviewing in the evaluation of fatigue items: Results from the patient-reported outcomes measurement information system (PROMIS). Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 17, 1239–1246. Christodoulou, C., Junghaenel, D. U., DeWalt, D. A., Rothrock, N., & Stone, A. A. (2008). Cognitive interviewing in the evaluation of fatigue items: Results from the patient-reported outcomes measurement information system (PROMIS). Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 17, 1239–1246.
14.
Zurück zum Zitat Noonan, V. K., Cook, K. F., Bamer, A. M., Choi, S. W., Kim, J., & Amtmann, D. (2012). Measuring fatigue in persons with multiple sclerosis: Creating a crosswalk between the Modified Fatigue Impact Scale and the PROMIS fatigue short form. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 21, 1123–1133. Noonan, V. K., Cook, K. F., Bamer, A. M., Choi, S. W., Kim, J., & Amtmann, D. (2012). Measuring fatigue in persons with multiple sclerosis: Creating a crosswalk between the Modified Fatigue Impact Scale and the PROMIS fatigue short form. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 21, 1123–1133.
15.
Zurück zum Zitat Revicki, D. A., Chen, W. H., Harnam, N., Cook, K. F., Amtmann, D., Callahan, L. F., et al. (2009). Development and psychometric analysis of the PROMIS pain behavior item bank. Pain, 146, 158–169.CrossRefPubMedPubMedCentral Revicki, D. A., Chen, W. H., Harnam, N., Cook, K. F., Amtmann, D., Callahan, L. F., et al. (2009). Development and psychometric analysis of the PROMIS pain behavior item bank. Pain, 146, 158–169.CrossRefPubMedPubMedCentral
16.
Zurück zum Zitat Amtmann, D., Cook, K. F., Jensen, M. P., Chen, W. H., Choi, S., Revicki, D., et al. (2010). Development of a PROMIS item bank to measure pain interference. Pain, 150, 173–182.CrossRefPubMedPubMedCentral Amtmann, D., Cook, K. F., Jensen, M. P., Chen, W. H., Choi, S., Revicki, D., et al. (2010). Development of a PROMIS item bank to measure pain interference. Pain, 150, 173–182.CrossRefPubMedPubMedCentral
17.
Zurück zum Zitat Fries, J. F., Cella, D., Rose, M., Krishnan, E., & Bruce, B. (2009). Progress in assessing physical function in arthritis: PROMIS short forms and computerized adaptive testing. The Journal of Rheumatology, 36, 2061–2066.CrossRefPubMed Fries, J. F., Cella, D., Rose, M., Krishnan, E., & Bruce, B. (2009). Progress in assessing physical function in arthritis: PROMIS short forms and computerized adaptive testing. The Journal of Rheumatology, 36, 2061–2066.CrossRefPubMed
18.
Zurück zum Zitat Flynn, K. E., Shelby, R. A., Mitchell, S. A., Fawzy, M. R., Hardy, N. C., Husain, A. M., et al. (2010). Sleep-wake functioning along the cancer continuum: Focus group results from the patient-reported outcomes measurement information system (PROMIS((R))). Psychooncology, 19, 1086–1093.CrossRefPubMedPubMedCentral Flynn, K. E., Shelby, R. A., Mitchell, S. A., Fawzy, M. R., Hardy, N. C., Husain, A. M., et al. (2010). Sleep-wake functioning along the cancer continuum: Focus group results from the patient-reported outcomes measurement information system (PROMIS((R))). Psychooncology, 19, 1086–1093.CrossRefPubMedPubMedCentral
19.
Zurück zum Zitat Hahn, E. A., Devellis, R. F., Bode, R. K., Garcia, S. F., Castel, L. D., Eisen, S. V., et al. (2010). Measuring social health in the patient-reported outcomes measurement information system (PROMIS): Item bank development and testing. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 19, 1035–1044. Hahn, E. A., Devellis, R. F., Bode, R. K., Garcia, S. F., Castel, L. D., Eisen, S. V., et al. (2010). Measuring social health in the patient-reported outcomes measurement information system (PROMIS): Item bank development and testing. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 19, 1035–1044.
20.
Zurück zum Zitat Amtmann, D., Kim, J., Chung, H., Askew, R. L., Park, R., & Cook, K. F. (2016). Minimally important differences for patient reported outcomes measurement information system pain interference for individuals with back pain. Journal of Pain Research, 9, 251–255.CrossRefPubMedPubMedCentral Amtmann, D., Kim, J., Chung, H., Askew, R. L., Park, R., & Cook, K. F. (2016). Minimally important differences for patient reported outcomes measurement information system pain interference for individuals with back pain. Journal of Pain Research, 9, 251–255.CrossRefPubMedPubMedCentral
21.
Zurück zum Zitat Pilkonis, P. A., Yu, L., Dodds, N. E., Johnston, K. L., Maihoefer, C. C., & Lawrence, S. M. (2014). Validation of the depression item bank from the patient-reported outcomes measurement information system (PROMIS) in a three-month observational study. Journal of Psychiatric Research, 56, 112–119.CrossRefPubMedPubMedCentral Pilkonis, P. A., Yu, L., Dodds, N. E., Johnston, K. L., Maihoefer, C. C., & Lawrence, S. M. (2014). Validation of the depression item bank from the patient-reported outcomes measurement information system (PROMIS) in a three-month observational study. Journal of Psychiatric Research, 56, 112–119.CrossRefPubMedPubMedCentral
22.
Zurück zum Zitat Ware, J. E. Jr., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlof, C. G., et al. (2003) Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 12, 935–952. Ware, J. E. Jr., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlof, C. G., et al. (2003) Applications of computerized adaptive testing (CAT) to the assessment of headache impact. Quality of life research: An international journal of quality of life aspects of treatment. Care and Rehabilitation, 12, 935–952.
23.
Zurück zum Zitat Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1, 307–310.CrossRefPubMed Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1, 307–310.CrossRefPubMed
Metadaten
Titel
Grooming a CAT: customizing CAT administration rules to increase response efficiency in specific research and clinical settings
verfasst von
Michael A. Kallen
Karon F. Cook
Dagmar Amtmann
Elizabeth Knowlton
Richard C. Gershon
Publikationsdatum
05.05.2018
Verlag
Springer International Publishing
Erschienen in
Quality of Life Research / Ausgabe 9/2018
Print ISSN: 0962-9343
Elektronische ISSN: 1573-2649
DOI
https://doi.org/10.1007/s11136-018-1870-z

Weitere Artikel der Ausgabe 9/2018

Quality of Life Research 9/2018 Zur Ausgabe