Skip to main content
Log in

Assessing the Unidimensionality of Psychological Scales: Using Multiple Criteria from Factor Analysis

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

Whenever one uses a composite scale score from item responses, one is tacitly assuming that the scale is dominantly unidimensional. Investigating the unidimensionality of item response data is an essential component of construct validity. Yet, there is no universally accepted technique or set of rules to determine the number of factors to retain when assessing the dimensionality of item response data. Typically factor analysis is used with the eigenvalues-greater-than-one rule, the ratio of first-to-second eigenvalues, parallel analysis, root-mean-square-error-of-approximation, or hypothesis testing approaches involving chi-square tests from Maximum Likelihood or Generalized Least Squares estimation. The purpose of this study was to investigate how these various procedures perform individually, and in combination, when assessing the unidimensionality of item response data via a computer simulated design. Conditions such as sample size, magnitude of communality, distribution of item responses, proportion of communality on second factor, and the number of items with non-zero loadings on the second factor were varied. Results indicate that there was no one individual decision-making method that identified unidimensionality under all conditions manipulated. Given the low communalities, all individual decision-making methods failed to detect unidimensionality for the combination where sample size was small, magnitude of communality was low, and item distributions were skewed. A set of guidelines and a new statistical methodology are provided for researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for education and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • Aron, A., & Aron, E. (2002). Statistics for psychology (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Boyd, K. C., & Gorsuch, R. L. (2003). Factor replication, factor invariance, and salient loadings: Three objective criteria for number of factors. Manuscript submitted for publication.

  • Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230–258.

    Article  Google Scholar 

  • Bryne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Crawford, C. B. (1975). Determining the number of interpretable factors. Psychological Bulletin, 82, 226–237.

    Article  Google Scholar 

  • Crawford, C. B., & Koopman, P. (1973). A note on Horn’s test for the number of factors in factor in analysis. Multivariate Behavioral Research, 8, 117–125.

    Article  Google Scholar 

  • Davison, M. L., & Sireci, S. G. (2000). Multidimensional scaling. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematics modeling (pp. 323–352). New York: Academic Press.

    Chapter  Google Scholar 

  • DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.

    Article  Google Scholar 

  • Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.

    Article  Google Scholar 

  • Gessaroli, M. E., & De Champlain, A. F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying responses to a set of items. Journal of Educational Measurement, 33, 157–179.

    Article  Google Scholar 

  • Glenberg, A. (1996). Learning from data: An introduction to statistical reasoning. New York, NY: Lawrence Erlbaum Associates.

    Google Scholar 

  • Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 1–10.

    Article  Google Scholar 

  • Hattie, J. (1984). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 20, 1–14.

    Article  Google Scholar 

  • Hattie, J. (1985). An empirical study of the various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49–78.

    Article  Google Scholar 

  • Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology, 123(3), 207–215.

    Article  Google Scholar 

  • Humphreys, L. G. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.

    Article  Google Scholar 

  • Humphreys, L. G. (1962). The organization of human abilities. American Psychologist, 17, 475–483.

    Article  Google Scholar 

  • Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education and National Council on Measurement in Education.

    Google Scholar 

  • Lissitz, R. W. (Ed.). (2009). The concept of validity: Revisions, new directions and applications. Charlotte: Information Age Publishing.

    Google Scholar 

  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. New Jersey: Lawrence Erlbaum.

    Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637.

    Article  Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.

    Article  Google Scholar 

  • Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955–966.

    Article  Google Scholar 

  • Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 33–45). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Preacher, K. J., & MacCallum, R. C. (2002). Exploratory factor analysis in behavior genetics research: Factor recovery with small sample sizes. Behavior Genetics, 32(2), 13–43.

    Article  Google Scholar 

  • Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28(2), 1629–1646.

    Article  Google Scholar 

  • Schonemann, P. H. (1981). Power as a function of communality in factor analysis. Bulletin of the Psychonomic Society, 17, 57–60.

    Google Scholar 

  • Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.

    Google Scholar 

  • Slocum-Gori, S. L., Zumbo, B. D., Michalos, A. C., & Diener, E. (2009). A note on the dimensionality of quality of life scales: An illustration with the Satisfaction With Life Scale (SWLS). Social Indicators Research: An International Interdisciplinary Journal for Quality of Life Measurement, 92, 489–496.

    Google Scholar 

  • Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589–617.

    Article  Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins College Publishers.

    Google Scholar 

  • Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: An historical overview and some guidelines. Educational and Psychological Measurement, 56, 213–224.

    Google Scholar 

  • Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45–79). The Netherlands: Elsevier Science B.V.

    Google Scholar 

  • Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.

    Google Scholar 

  • Zumbo, B. D., Gelin, M. N., & Hubley, A. M. (2002). The construction and use of psychological tests and measures. In the Psychology theme of the Encyclopedia of Life Support Systems (EOLSS). Oxford, UK: Eolss Publishers. http://www.educ.ubc.ca/faculty/zumbo/talks/Zumbo_Gelin_Hubley_EOLSS.pdf.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Suzanne L. Slocum-Gori.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Slocum-Gori, S.L., Zumbo, B.D. Assessing the Unidimensionality of Psychological Scales: Using Multiple Criteria from Factor Analysis. Soc Indic Res 102, 443–461 (2011). https://doi.org/10.1007/s11205-010-9682-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-010-9682-8

Keywords

Navigation