Abstract
In this paper, the state of research on the assessment of competencies in higher education is reviewed. Fundamental conceptual and methodological issues are clarified by showing that current controversies are built on misleading dichotomies. By systematically sketching conceptual controversies, competing competence definitions are unpacked (analytic/trait vs. holistic/real-world performance) and commonplaces are identified. Disagreements are also highlighted. Similarly, competing statistical approaches to assessing competencies, namely item-response theory (latent trait) versus generalizability theory (sampling error variance), are unpacked. The resulting framework moves beyond dichotomies and shows how the different approaches complement each other. Competence is viewed along a continuum from traits that underlie perception, interpretation, and decision-making skills, which in turn give rise to observed behavior in real-world situations. Statistical approaches are also viewed along a continuum from linear to nonlinear models that serve different purposes. Item response theory (IRT) models may be used for scaling item responses and modeling structural relations, and generalizability theory (GT) models pinpoint sources of measurement error variance, thereby enabling the design of reliable measurements. The proposed framework suggests multiple new research studies and may serve as a “grand” structural model.
References
(2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125–154.
(2011). An evidence-based model of work engagement. Current Directions in Psychological Science, 20, 265–269.
(1980). Life-span developmental psychology. Annual Review of Psychology, 31, 65–110.
(2013). The principles and logic of competency testing in higher education. In , Modeling and measuring competencies in higher education: Tasks and challenges (pp. 127–136). Boston, MA: Sense.
(1993). The validity and reliability of assessments and self-assessments of workbased learning. Assessment & Evaluation in Higher Education, 18, 83–94.
(2011). Black-white differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review. Journal of Applied Psychology, 96, 881–906.
(2004). Competence-based VET in The Netherlands: Backgrounds and pitfalls. Journal of Vocational Education and Training, 56, 523–538.
(2007). Evaluating the assessment: Sources of evidence for quality assurance. Studies in Educational Evaluation, 33, 29–49.
(2014). Entwicklung von Lehrpersonen in den ersten Berufsjahren: Längsschnittliche Vorhersage von Unterrichtswahrnehmung und Lehrerreaktionen durch Ausbildungsergebnisse
[Teacher development during the initial years in the profession: Longitudinal prediction of lesson perception and teacher reactions by means of training results] . Zeitschrift für Erziehungswissenschaft, 17, 509–542.(2013). Call for papers: Assessment of competencies in higher education – a topical issue of the Zeitschrift für Psychologie . Zeitschrift für Psychologie, 221, 202.
. (2013). Modeling and measuring competencies in higher education: Tasks and challenges. Rotterdam, The Netherlands: Sense.
(1982). The competent manager. New York, NY: Wiley.
(2001). Generalizability theory. New York, NY: Springer.
(2001). Organizational behavior: Affect in the workplace. Annual Review of Psychology, 53, 279–307.
(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
(1971). Abilities: Their structure, growth, and action. New York, NY: Houghton Mifflin.
(2002). Remaking the concept of aptitude: Extending the legacy of R. E. Snow. Mahwah, NJ: Erlbaum.
(1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
(1999). Intraindividual variability in affect: Reliability, validity, and personality correlates. Journal of Personality and Social Psychology, 76, 662–676.
(2002). Defining and assessing professional competence. JAMA, 287, 226–235.
(2003). National vocational qualifications in England: Description and analysis of an alternative qualification system. In , Zertifizierung non-formell und informell erworbener beruflicher Kompetenzen. Münster, Germany: Waxmann.
(2015). Validating test score interpretations by cross-national comparison: Comparing the results of students from Japan and Germany on an American test of economic knowledge in higher education. Zeitschrift für Psychologie, 223, 14–23. doi: 10.1027/2151-2604/a000195
(2013). Evaluation eines videobasierten Trainingsseminars zur Förderung der professionallen Wahrnehmung von Klassenführung im Grundschulunterricht
[Evaluation of a video-based training program to enhance professional perception of classroom leadership in primary school education] . Zeitschrift für Pädagogische Psychologie, 27, 141–155.(1979). On competence: A critical analysis of competence-based reforms in higher education. San Francisco, CA: Jossey-Bass.
(2002). Measurement from a hierarchical point of view. In , The role of constructs in psychological and educational measurement (pp. 73–95). London, UK: Erlbaum.
(2010). Unidimensionality and interpretability of psychological instruments. In , Measuring psychological constructs: Advances in model-based approaches. Washington, DC: American Psychological Association.
(2012). Do burnout and work engagement predict depressive symptoms and life satisfaction? A three-wave seven-year prospective study. Journal of Affective Disorders, 141, 415–424.
(2003). Assessment of competencies in England’s National Vocational Qualification system. In , Zertifizierung non-formell und informell erworbener beruflicher Kompetenzen. Münster, Germany: Waxmann.
(2015). The relationship of mathematical competence and mathematics anxiety: An application of latent state-trait theory. Zeitschrift für Psychologie, 223, 31–38. doi: 10.1027/2151-2604/a000197
(2001). The job satisfaction-job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376–407.
(1992). An argument-based approach to validation. Psychological Bulletin, 112, 527–535.
(2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73.
(2006). Innovative technological performance. In , Defining technological literacy: Towards an epistemological framework (pp. 159–179). Basingstoke, UK: Palgrave.
(2014). Is teachers’ general pedagogical knowledge a premise for noticing and interpreting classroom situations? A video-based assessment approach. Teaching and Teacher Education, 38, 76–88.
(2008). Current issues in competence modeling and assessment. Zeitschrift für Psychologie, 216, 61–73. doi: 10.1027/0044-3409.216.2.61
(1970). Discipline and group management in classrooms. New York, NY: Holt, Rinehart, & Winston.
(2002). Cognitive abilities and motivational processes in high school students’ situational engagement and achievement in science. Educational Assessment, 8, 139–162.
(1973). Testing for competence rather than testing for “intelligence”. American Psychologist, 28, 1–14.
(2000). Finite mixture models. New York, NY: Wiley.
(2003). Portfolios and assessment of competence: A review of the literature. Journal of Advanced Nursing, 41, 283–294.
(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
(1990). The assessment of clinical skills/competence/performance. Academic Medicine: Journal of the Association of American Medical Colleges, 65, 63–67.
(2009). The new competence concept in higher education: Error or enrichment? Journal of European Industrial Training, 33, 755–770.
(2002). Beyond SEM: General latent variable modeling. Behaviormetrika, 29, 81–117.
(2013). “I know how to do it, but I can’t do it”: Modeling competence profiles for future teachers and trainers. In , Modeling and measuring competencies in higher education: Tasks and challenges (pp. 45–60). Rotterdam, The Netherlands: Sense.
(2010). Studying the reliability of group-level measures with implications for statistical power: A six-step paradigm [Working Paper]. Chicago, IL: University of Chicago.
(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696.
(2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185–205.
(2013). Making competent judgments of competence. In , Modeling and measuring competencies in higher education: Tasks and challenges (pp. 13–27). Rotterdam, The Netherlands: Sense.
(2010). How we think: A theory of goal-oriented decision making and its educational applications. New York, NY: Routledge.
(2010). On the measurement of competency. Empirical Research in Vocational Education and Training, 1, 43–65.
(2012). An approach to testing and modeling competencies. In , Modeling and measuring competencies in higher education: Tasks and challenges. Rotterdam, The Netherlands: Sense.
(1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
(1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57, 1–22.
(2004). Generalized latent variable modeling: Multilevel longitudinal, and structural equation models. London, UK: Chapman and Hall/CRC.
(1994). Abilities in academic tasks. In , Mind in context: Interactionist perspectives on human intelligence. New York, NY: Cambridge University Press.
(1993). Competency requirement forecasting: Issues for international selection and assessment. International Journal of Selection and Assessment, 1, 50–58.
(1993). Competence at work: Models for superior performance. New York, NY: Wiley.
(2009). What intelligence tests miss: The psychology of rational thought. New Haven, CT: Yale University Press.
(2003). The psychology of abilities, competencies, and expertise. Cambridge, MA: Cambridge University Press.
(1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13, 389–408.
(2012). Declarative knowledge and professional vision in teacher education: Effect of courses in teaching and learning. British Journal of Educational Psychology, 83, 467–483.
(2007). Testlet response theory and its applications. Cambridge, UK: Cambridge University Press.
(2001). Assessment of clinical competence. Lancet, 357, 945–949.
(2001). Concept of competence: A conceptual clarification. In , Defining and selecting key competencies (pp. 45–66). Göttingen, Germany: Hogrefe.
