Abstract
Whenever one uses a composite scale score from item responses, one is tacitly assuming that the scale is dominantly unidimensional. Investigating the unidimensionality of item response data is an essential component of construct validity. Yet, there is no universally accepted technique or set of rules to determine the number of factors to retain when assessing the dimensionality of item response data. Typically factor analysis is used with the eigenvalues-greater-than-one rule, the ratio of first-to-second eigenvalues, parallel analysis, root-mean-square-error-of-approximation, or hypothesis testing approaches involving chi-square tests from Maximum Likelihood or Generalized Least Squares estimation. The purpose of this study was to investigate how these various procedures perform individually, and in combination, when assessing the unidimensionality of item response data via a computer simulated design. Conditions such as sample size, magnitude of communality, distribution of item responses, proportion of communality on second factor, and the number of items with non-zero loadings on the second factor were varied. Results indicate that there was no one individual decision-making method that identified unidimensionality under all conditions manipulated. Given the low communalities, all individual decision-making methods failed to detect unidimensionality for the combination where sample size was small, magnitude of communality was low, and item distributions were skewed. A set of guidelines and a new statistical methodology are provided for researchers.
Similar content being viewed by others
References
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for education and psychological testing. Washington, DC: American Psychological Association.
Aron, A., & Aron, E. (2002). Statistics for psychology (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
Boyd, K. C., & Gorsuch, R. L. (2003). Factor replication, factor invariance, and salient loadings: Three objective criteria for number of factors. Manuscript submitted for publication.
Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230–258.
Bryne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum Associates.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Crawford, C. B. (1975). Determining the number of interpretable factors. Psychological Bulletin, 82, 226–237.
Crawford, C. B., & Koopman, P. (1973). A note on Horn’s test for the number of factors in factor in analysis. Multivariate Behavioral Research, 8, 117–125.
Davison, M. L., & Sireci, S. G. (2000). Multidimensional scaling. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematics modeling (pp. 323–352). New York: Academic Press.
DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.
Gessaroli, M. E., & De Champlain, A. F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying responses to a set of items. Journal of Educational Measurement, 33, 157–179.
Glenberg, A. (1996). Learning from data: An introduction to statistical reasoning. New York, NY: Lawrence Erlbaum Associates.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 1–10.
Hattie, J. (1984). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 20, 1–14.
Hattie, J. (1985). An empirical study of the various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49–78.
Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology, 123(3), 207–215.
Humphreys, L. G. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.
Humphreys, L. G. (1962). The organization of human abilities. American Psychologist, 17, 475–483.
Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education and National Council on Measurement in Education.
Lissitz, R. W. (Ed.). (2009). The concept of validity: Revisions, new directions and applications. Charlotte: Information Age Publishing.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. New Jersey: Lawrence Erlbaum.
MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955–966.
Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 33–45). Hillsdale, NJ: Lawrence Erlbaum.
Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis. Thousand Oaks, CA: Sage.
Preacher, K. J., & MacCallum, R. C. (2002). Exploratory factor analysis in behavior genetics research: Factor recovery with small sample sizes. Behavior Genetics, 32(2), 13–43.
Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28(2), 1629–1646.
Schonemann, P. H. (1981). Power as a function of communality in factor analysis. Bulletin of the Psychonomic Society, 17, 57–60.
Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.
Slocum-Gori, S. L., Zumbo, B. D., Michalos, A. C., & Diener, E. (2009). A note on the dimensionality of quality of life scales: An illustration with the Satisfaction With Life Scale (SWLS). Social Indicators Research: An International Interdisciplinary Journal for Quality of Life Measurement, 92, 489–496.
Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589–617.
Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins College Publishers.
Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: An historical overview and some guidelines. Educational and Psychological Measurement, 56, 213–224.
Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45–79). The Netherlands: Elsevier Science B.V.
Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.
Zumbo, B. D., Gelin, M. N., & Hubley, A. M. (2002). The construction and use of psychological tests and measures. In the Psychology theme of the Encyclopedia of Life Support Systems (EOLSS). Oxford, UK: Eolss Publishers. http://www.educ.ubc.ca/faculty/zumbo/talks/Zumbo_Gelin_Hubley_EOLSS.pdf.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Slocum-Gori, S.L., Zumbo, B.D. Assessing the Unidimensionality of Psychological Scales: Using Multiple Criteria from Factor Analysis. Soc Indic Res 102, 443–461 (2011). https://doi.org/10.1007/s11205-010-9682-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11205-010-9682-8