A Comparison of the Reliabilities of Four Types of Difference Scores for Five Cognitive Assessment Batteries
Abstract
Summary: In recent years, studies have been conducted comparing the reliability of simple difference scores to differences computed via reliable component analysis (RCA). For the five cognitive assessment batteries examined here, RCA difference scores have been superior to simple difference scores in this regard. In the present study, the five cognitive assessment batteries are considered again, including a comparison of two alternative methods of difference score construction, the residualized and base-free difference methods, in order to determine if RCA outperforms the other methods as well. Results indicated that the RCA difference scores were more reliable in 68/70 comparisons between the four methods. When difference scores are a central part of test interpretation, clinicians should use the RCA scoring method because of the high reliability of the difference scores.
References
Bracken, B.A., McCallum, R.S. (1998). Universal nonverbal intelligence test examiner's manual. . Itsaka, IL: Riverside Publishing.Caruso, J.C. (2001a). Reliable component analysis of the Stanford-Binet: Fourth edition for 2- to 6-year-olds.. Psychological Assessment, 13, 261– 266Caruso, J.C. (2001b). Increasing the reliability of the fluid/crystallized difference score from the Kaufman Adolescent and Adult Intelligence Scale with reliable component analysis.. Assessment, 8, 155– 166Caruso, J.C., Cliff, N. (2000). Increasing the reliability of WISC-III difference scores with reliable component analysis.. Psychological Assessment, 12, 89– 96Caruso, J.C., Cliff, N. (in press). Reliable component analysis: A new/old method for exploratory data reduction.. To be published in B. Thompson's (Ed.), Advances in social science methodology. Thousand Oaks, CA: Sage.Caruso, J.C., Witkiewitz, K. (2001). Memory and reasoning abilities assessed by the universal nonverbal intelligence test: A reliable component (RCA) study.. Educational and Psychological Measurement, 61, 5– 22Caruso, J.C., Witkiewitz, K. (2002). Increasing the reliability of ability - achievement difference scores: An example using the Kaufman assessment battery for children.. Journal of Educational Measurement, 39, 39– 58Cattell, R.B. (1982). The clinical use of difference scores: Some psychometric problems.. Multivariate Experimental Clinical Research, 6, 87– 98Cliff, N., Caruso, J.C. (1998). Reliable component analysis through maximizing component reliability.. Psychological Methods, 3, 291– 308Cronbach, L.J., Furby, L. (1970). How we should measure “change” - or should we?. Psychological Bulletin, 74, 68– 80Gridley, B.E., McIntosh, D.E. (1991). Confirmatory factor analysis of the Stanford-Binet: Fourth edition for a normal sample.. Journal of School Psychology, 29, 237– 248Kaplan, S.L., Alfonso, V.C. (1997). Confirmatory factor analysis of the Stanford-Binet: Fourth edition with preschoolers with developmental delays.. Journal of Psychoeducational Assessment, 15, 226– 237Kaufman, A.S., Kaufman, N.L. (1983). Kaufman assessment battery for children: Administration and scoring manual. . Circle Pines, MN: American Guidance Service.Kaufman, A.S., Kaufman, N.L. (1993). The Kaufman adolescent and adult intelligence test manual. . Circle Pines, MN: American Guidance Service.Lord, F.M., Novick, M.R. (1968). Statistical theories of mental test scores. . Reading, MA: Addison-Wesley.Malgady, R.G., Colon-Malgady, G. (1991). Comparing the reliability of difference scores and residuals in analysis of covariance.. Educational and Psychological Measurement, 51, 803– 807Reynolds, C.R. (1984-1985). Critical measurement issues in learning disabilities.. Journal of Special Education, 18, 451– 476Reynolds, C.R. Kamphaus, R.W., Rosenthal, B.L. (1988). Factor analysis of the Stanford-Binet Fourth Edition for ages 2 years through 23 years.. Measurement and Evaluation in Counseling and Development, 21, 52– 63Thorndike, R.L. (1990). Would the real factors of the Stanford-Binet fourth edition please come forward?. Journal of Psychoeducational Assessment, 8, 412– 435Thorndike, R.L. Hagen, E.P., Sattler, J.M. (1986). Guide for administering and scoring, the Stanford-Binet intelligence scale: Fourth edition. . Chicago: Riverside Publishing.Tucker, L.R. Damarin, F., Messick, S. (1966). A base-free measure of change.. Psychometrika, 31, 457– 473Wechsler, D. (1991). Wechsler Intelligence Scale for children - third edition. . San Antonio, TX: Psychological Corporation.Williams, R.H., Zimmerman, D.W. (1977). The reliability of difference scores when errors are correlated.. Educational and Psychological Measurement, 37, 679– 689Williams, R.H., Zimmerman, D.W. (1982-1983). The comparative reliability of simple and residualized difference scores.. Journal of Experimental Education, 51, 94– 97Williams, R.H. Zimmerman, D.W., Mazzagatti, R.D. (1987). Large sample estimates of the reliability of simple, residualized, and base-free gain scores.. Journal of Experimental Education, 55, 116– 118
