Assessing Longitudinal Change: Adjustment for Regression to the Mean Effects

Abstract

Pascarella (J Coll Stud Dev 47:508–520, 2006) has called for an increase in use of longitudinal data with pretest-posttest design when studying effects on college students. However, such designs that use multiple measures to document change are vulnerable to an important threat to internal validity, regression to the mean. Herein, we discuss a brief history of regression to the mean and illustrate a straightforward procedure to make adjustments to initial pretest scores for regression to the mean effects utilizing a method developed by Roberts (in: G. Echternacht (Guest ed.) New directions for testing and measurement, 1980). Analyses are shown with both unadjusted and adjusted pretest scores, illustrating dramatic differences in conclusions about whether students change across time.

Appendices

Appendix 1

Salient Attributes of the Six Model Environments from Holland’s Theory ¹

Investigative environments emphasize analytical or intellectual activities aimed at the creation and use of knowledge. Such environments devote little attention to persuasive, social, and repetitive activities. These behavioral tendencies in investigative environments lead, in turn, to the acquisition of analytical, scientific, and mathematical competencies and to a deficit in persuasive and leadership abilities. People in investigative environments are encouraged to perceive themselves as cautious, critical, complex, curious, independent, precise, rational, and scholarly. Investigative environments reward people for skepticism and persistence in problem solving, documentation of new knowledge, and understanding solutions of common problems. Biology, civil engineering, and mathematics are representative of “consistent” disciplines in investigative environments; pharmacy, economics, and sociology are representative of “inconsistent” disciplines in investigative environments.

Artistic environments emphasize ambiguous, free, and unsystematized activities that involve emotionally expressive interactions with others. These environments devote little attention to explicit, systematic, and ordered activities. These behavioral tendencies in Artistic environments lead, in turn, to the acquisition of innovative and creative competencies––language, art, music, drama, writing––and to a deficit in clerical and business system competencies. People in Artistic environments are encouraged to perceive themselves as having unconventional ideas or manners and possessing aesthetic values. Artistic environments reward people for imagination in literary, artistic, or musical accomplishments. English language and literature and philosophy are representative of “consistent” disciplines in Artistic environments; journalism and drama/theater arts are representative of “inconsistent” disciplines in Artistic environments.

Social environments emphasize activities that involve the mentoring, treating, healing, or teaching of others. These environments devote little attention to explicit, ordered, systematic activities involving materials, tools, or machines. These behavioral tendencies in social environments lead, in turn, to the acquisition of interpersonal competencies and to a deficit in manual and technical competencies. People in social environments are encouraged to perceive themselves as cooperative, empathetic, generous, helpful, idealistic, responsible, tactful, understanding, and having concern for the welfare of others. Social environments reward people for the display of empathy, humanitarianism, sociability, and friendliness. Elementary education and social work are representative of “consistent” disciplines in social environments; nursing is representative of an “inconsistent” discipline in social environments.

Enterprising environments emphasize activities that involve the manipulation of others to attain organizational goals or economic gain. These environments devote little attention to observational, symbolic, and systematic activities. These behavioral tendencies in enterprising environments lead, in turn, to an acquisition of leadership, interpersonal, speaking, and persuasive competencies and to a deficit in scientific competencies. People in enterprising environments are encouraged to perceive themselves as aggressive, ambitious, domineering, energetic, extroverted, optimistic, popular, self-confident, sociable, and talkative. Enterprising environments reward people for the display of initiative in the pursuit of financial or material accomplishments, dominance, and self-confidence. Finance, market, and law are representative of “consistent” disciplines in enterprising environments; business administration and business management are representative of “inconsistent” disciplines in enterprising environments.

Realistic environments emphasize concrete, practical activities and the use of machines, tools, and materials. These behavioral tendencies of realistic environments lead, in turn, to the acquisition of mechanical and technical competencies and to a deficit in human relation skills. People in realistic environments are encouraged to perceive themselves as having practical, productive, and concrete values. Realistic environments reward people for the display of conforming behavior and practical accomplishment. Agriculture and archaeology are examples of realistic majors.

Conventional environments emphasize activities that involve the explicit, ordered, systematic manipulation of data to meet predictable organizational demands or specified standards. The behavioral tendencies in conventional environments lead, in turn, to the acquisition of clerical, computational, and business system competencies necessary to meet precise performance standards and to a deficit in artistic competencies. People in conventional environments are encouraged to perceive themselves as having a conventional outlook and concern for orderliness and routines. Conventional environments reward people for the display of dependability, conformity, and organizational skills. Secretarial science is an example of a conventional major.

Appendix 2

Table 2 1986 and 1990 investigative student ability and interest scales