Effect size guidelines for individual differences researchers

https://doi.org/10.1016/j.paid.2016.06.069Get rights and content

Highlights

  • Cohen's correlation guidelines (0.10, 0.30, 0.50) were evaluated empirically.

  • A pool of 708 meta-analytically derived correlations were collated.

  • The 25th, 50th, and 75th percentiles corresponded to r = 0.10, r = 0.20, r = 0.30.

  • Researchers are recommended to consider these normative guidelines.

Abstract

Individual differences researchers very commonly report Pearson correlations between their variables of interest. Cohen (1988) provided guidelines for the purposes of interpreting the magnitude of a correlation, as well as estimating power. Specifically, r = 0.10, r = 0.30, and r = 0.50 were recommended to be considered small, medium, and large in magnitude, respectively. However, Cohen's effect size guidelines were based principally upon an essentially qualitative impression, rather than a systematic, quantitative analysis of data. Consequently, the purpose of this investigation was to develop a large sample of previously published meta-analytically derived correlations which would allow for an evaluation of Cohen's guidelines from an empirical perspective. Based on 708 meta-analytically derived correlations, the 25th, 50th, and 75th percentiles corresponded to correlations of 0.11, 0.19, and 0.29, respectively. Based on the results, it is suggested that Cohen's correlation guidelines are too exigent, as < 3% of correlations in the literature were found to be as large as r = 0.50. Consequently, in the absence of any other information, individual differences researchers are recommended to consider correlations of 0.10, 0.20, and 0.30 as relatively small, typical, and relatively large, in the context of a power analysis, as well as the interpretation of statistical results from a normative perspective.

Introduction

Researchers in the behavioural and cognitive sciences have been recommended to report and interpret effect sizes in their research papers (Wilkinson & the APA Task Force on Statistical Inference, 1999, p. 599). Cohen, 1988, Cohen, 1992 provided guidelines for the purposes of interpreting the magnitude of effect sizes across a number of statistical analyses. Individual differences researchers very commonly report correlation coefficients to represent the magnitude of the association between two continuously scored variables. Cohen, 1988, Cohen, 1992 recommended Pearson r values of 0.10, 0.30, and 0.50 to demarcate small, medium, and large effects, respectively.1 Cohen's effect size guidelines were based upon the notion that a medium effect should be noticeable to the naked eye of a careful observer (Cohen, 1988). Additionally, Cohen, 1988, Cohen, 1992 suggested that a medium effect is about the average effect observed in the literature across various disciplines. However, Cohen, 1988, Cohen, 1992 impression of an average effect was not based on a systematic, quantitative analysis of data.

More recently, Hemphill (2003) provided quantitatively-based guidelines for the purposes of interpreting correlation coefficients on the basis of a review of two meta-meta-analyses. Hemphill (2003) found that that one third of the correlations were < 0.20, one third were between 0.20 and 0.30, and one third were > 0.30. Consequently, Hemphill (2003) suggested a revision of Cohen, 1988, Cohen, 1992 guidelines: small < 0.20; medium = 0.20 to 0.30, and large > 0.30.

Although Hemphill's (2003) recommendations may be considered an advancement over Cohen, 1988, Cohen, 1992 guidelines, approximately 80% of the correlations included in the Hemphill (2003) review were derived from treatment/therapy experiments, all of which reported Cohen's d values. Hemphill (2003) converted the Cohen's d values into correlations for the purposes of his investigation. Arguably, the effects observed in treatment/experiments may not be valid representations of the effect sizes that might be expected in individual differences research for a number of reasons. First, one of the variables associated with a basic experiment is manipulated. By contrast, a typical individual differences hypothesis is tested by the estimation of the association between two continuously scored variables in the absence of any manipulation. Secondly, a correlation derived from a Cohen's d value is essentially a point-biserial correlation, rather than a Pearson correlation. By contrast, individual differences researchers tend to report Pearson correlations to represent the association between their variables. Thirdly, Hemphill's (2003) investigation was also limited in that the distribution of the correlations was not reported, nor was a relatively complete percentile breakdown of the results provided. Finally, Hemphill (2003) reported only observed correlations, rather than both observed correlations and correlations disattenuated for imperfect reliability (i.e., true score correlations).

Consequently, the principal purpose of this investigation was to collate a large number of meta-analytically derived correlations across the broad area of differential psychology. The sample of correlations (observed score and true score) would then allow for the determination of empirically-based normative guidelines for individual differences researchers.

Section snippets

Search procedure

Meta-analytic publications were sought across six journals known to publish research relevant to individual differences: Personality and Individual Differences, Psychological Bulletin, Journal of Research in Personality, Journal of Personality and Social Psychology, Journal of Personality, and Intelligence. Google Scholar was used to identify the meta-analytic publications by restricting the search results to the titles above. Additionally, journal article title keyword search terms included

Results

As can be seen in Table 1 (left-hand side), the 25th, 50th, and 75th percentiles corresponded to correlations equal to 0.11, 0.19, and 0.29, respectively. Although not reported in Table 1, only 2.7% of the correlations were 0.50 or greater. Furthermore, approximately 55% of the correlations were ≤ 0.21. As can be seen in Fig. 1, the distribution of the correlations was skewed positively (skew = 0.95, z = 10.29, p < 0.001; kurtosis = 1.56, z = 8.51, p < 0.001).

As can be seen in Table 1 (right-hand side), the

Discussion

The results of this investigation suggest that Cohen, 1988, Cohen, 1992 commonly cited guidelines for interpreting correlations are too exigent. Specifically, in contrast to Cohen's impression-based guidelines of 0.10, 0.30, and 0.50 for small, medium, and large correlations, the results of this quantitative investigation suggest that normative guidelines should be closer to 0.10, 0.20, and 0.30, respectively. A correlation as large as 0.50 may be expected to occur in only 2.7% of cases. The

Conclusion

It is doubtful the emphasis on reporting and interpreting effect sizes in research will abate. Consequently, it is important to have a guidelines for the interpretation of effect sizes that are based on good quality, representative data, rather than subjective impressions. Differential psychologists will be arguably better served by applying the normative correlation guidelines reported in this investigation (0.10, 0.20, and 0.30), rather than those reported by Cohen, 1988, Cohen, 1992, or even

References (0)

Cited by (0)

View full text