Elsevier

Developmental Review

Volume 33, Issue 4, December 2013, Pages 357-370
Developmental Review

Sampling in developmental science: Situations, shortcomings, solutions, and standards

https://doi.org/10.1016/j.dr.2013.08.003Get rights and content

Highlights

  • Sampling is a key feature of every study in developmental science.

  • We evaluate four prominent sampling strategies in developmental science.

  • We judge these sampling strategies by criteria, such as representativeness and generalizability.

  • We tally the use of the four sampling strategies in five prominent developmental science journals.

  • Finally, we make recommendations about best practices for sample selection and reporting.

Abstract

Sampling is a key feature of every study in developmental science. Although sampling has far-reaching implications, too little attention is paid to sampling. Here, we describe, discuss, and evaluate four prominent sampling strategies in developmental science: population-based probability sampling, convenience sampling, quota sampling, and homogeneous sampling. We then judge these sampling strategies by five criteria: whether they yield representative and generalizable estimates of a study’s target population, whether they yield representative and generalizable estimates of subsamples within a study’s target population, the recruitment efforts and costs they entail, whether they yield sufficient power to detect subsample differences, and whether they introduce “noise” related to variation in subsamples and whether that “noise” can be accounted for statistically. We use sample composition of gender, ethnicity, and socioeconomic status to illustrate and assess the four sampling strategies. Finally, we tally the use of the four sampling strategies in five prominent developmental science journals and make recommendations about best practices for sample selection and reporting.

Introduction

When we undertake to study some phenomenon, we wish to know something about that phenomenon in a population, but in practice we study the phenomenon in a group of individuals who purportedly represent the target or reference population to whom we wish our results to generalize. That is, we sample the population. We sample because we normally do not command the resources (time, money, or personnel) to assess the entire population of interest. Sampling is therefore a key feature of every study in developmental science, and sampling has far-reaching implications in all studies. This article is concerned with sampling in developmental science. As we point out, different sampling strategies exist, and each has its implications. Employing sub-optimal sampling strategies is far too common in developmental research, compromises the validity and utility of the research, renders replication and cross-study comparisons difficult, and most generally impedes progress in the field of developmental science.

In this article, we briefly describe and illustrate four prominent strategies that answer the sampling challenge, and we evaluate each in terms of some fundamental, meaningful, and practical criteria. The four strategies include (a) population-based probability sampling as well as nonprobability sampling strategies such as (b) convenience sampling, (c) quota sampling, and (d) homogeneous sampling. The five criteria by which we appraise these sampling strategies include (a) whether they yield representative and generalizable estimates of a study’s target population (e.g., estimates of intelligence among the population when all sociodemographic groups are collapsed), (b) whether they yield representative and generalizable estimates of sociodemographic group differences within a study’s target population (e.g., how estimates of intelligence vary across a population’s ethnic groups), (c) the recruitment efforts and costs they entail, (d) whether they provide sufficient power to detect sociodemographic group differences, and (e) whether they introduce noise related to variation in sociodemographic factors and whether that noise can be accounted for statistically. After overviewing the four sampling strategies, we examine how the sociodemographic composition of a sample in terms of gender, ethnicity, and SES can compromise a study’s findings – regardless of the study goals. We then recount the use of each prominent sampling strategy in five high-profile journals in contemporary developmental science. On these bases, we arrive at conclusions and recommendations about best practices and practical considerations, including ethical issues, and discuss the importance of weighing the research question when considering the merits of various sampling strategies.

This article is not comprehensive, and we have not assumed some related burdens. By now demographers, sociologists, and others in many disciplines have weighed the pros and cons of different sampling strategies (Davis-Kean and Jager, 2011, Henry, 1990, Onwuegbuzie and Collins, 2007, Sue, 1999, Watters and Biernacki, 1989). This article does not provide a tutorial on sampling (see http://stattrek.com/statistics/data-collection-methods.aspx?Tutorial=Stat). We also eschew technical details in favor of highlighting “big picture” issues of design and practicality in an accessible way. Although our examples and arguments are applicable to any single sociodemographic factor or set of sociodemographic factors, here we limit our focus to gender, ethnicity, and SES. Also, although we fully recognize that gender, ethnicity, and SES are non-independent (ethnicity and SES in particular) but interact in myriad complex ways, when discussing the implications of these three sociodemographic factors we typically limit our examples to a single factor for the sake of conceptual clarity. Finally, for the purposes of this exposition about sampling, we combine “race” and “ethnicity” as used by the U.S. Government and its agencies and define six ethnic categories (see Table 1). We acknowledge that these ethnic groups are also heterogeneous in that each group contains people who originated from many different countries with different cultural practices.

Section snippets

Common sampling strategies in developmental science

Here we describe four of the most used sampling strategies, and we assess their advantages, disadvantages, and limitations. How each of the four sampling strategies fares on the five criteria is summarized in Table 2.

Why the sociodemographic composition of samples matters

Given the advantages and disadvantages of the four sampling strategies, it is important to note how sociodemographic characteristics can affect study outcomes and the interpretation of study results. Gender, ethnicity, and SES variation in many characteristics—physical and mental health (Adkins et al., 2009, American Public Health Association, 2004, Breslau et al., 2005, Crimmins and Saito, 2001, National Center for Health Statistics, 2011), beliefs and cognitions (Bornstein and Cote, 2004,

How sampling is used in developmental science

To determine the frequency with which the four different sampling methods are used in contemporary developmental science, we surveyed five years (2007–2011) of five high-profile developmental journals. The five journals included two that generally focus on abnormal development, (Journal of the American Academy of Child and Adolescent Psychiatry and Development and Psychopathology), two that generally focus on normative development (Developmental Psychology and Child Development), and one that

Implications and recommendations for reporting sample characteristics

The analysis of published articles reveals several inconsistencies and inadequacies in basic reporting of demographic characteristics of study samples in developmental science. Overall, 41.4% of articles published in five high-profile developmental science journals failed to report ethnicity at all or only reported that the sample was “predominantly White” or “about half minority”, which is not detailed enough to draw conclusions about the representativeness of the sample to a particular

Summary and conclusions

Here we have recounted four common sampling strategies (population-based probability sampling, convenience sampling, quota sampling, and homogenous sampling) and evaluated each by five meaningful criteria. We find that by far the most common sampling strategy (convenience sampling) is the least desirable in terms of representativeness, generalizability, and noise. Convenience samples require less practical investment in recruitment costs and efforts, but this advantage does not offset its

References (46)

  • Bornstein, M. H. (2013). The specificity principle in parenting and child development. Unpublished manuscript, Eunice...
  • M.H. Bornstein et al.

    Mothers’ parenting cognitions in cultures of origin, acculturating cultures, and cultures of destination

    Child Development

    (2004)
  • M.H. Bornstein et al.

    Play in two societies: Pervasiveness of process, specificity of structure

    Child Development

    (1999)
  • M.H. Bornstein et al.

    Functional analysis of the contents of maternal speech to infants of 5 and 13 months in four cultures: Argentina, France, Japan, and the United States

    Developmental Psychology

    (1992)
  • M.H. Bornstein et al.

    Ideas about parenting in Argentina, France, and the United States

    International Journal of Behavioral Development

    (1996)
  • J. Breslau et al.

    Specifying race-ethnic differences in risk for psychiatric disorder in a USA national sample

    Psychological Medicine

    (2005)
  • G.L. Burke et al.

    Correlates of obesity in young Black and White women: The CARDIA study

    American Journal of Public Health

    (1992)
  • N.A. Card et al.

    Direct and indirect aggression during childhood and adolescence: A meta-analytic review of gender differences, intercorrelations, and relations among adjustment

    Child Development

    (2008)
  • W.G. Cochran

    Sampling techniques

    (1977)
  • W.H. Courtenay et al.

    Gender and ethnic differences in health beliefs and behaviors

    Journal of Health Psychology

    (2002)
  • P. Davis-Kean et al.

    The use of large-scale data sets for the study of developmental science

  • C. Diala et al.

    Racial differences in attitudes towards professional mental health care and the use of services

    American Journal of Orthopsychiatry

    (2000)
  • F. Faul et al.

    G∗Power 3: A flexible statistical power analysis for the social, behavioral, and biomedical sciences

    Behavior Research Methods

    (2007)
  • Cited by (364)

    View all citing articles on Scopus

    Authorship is alphabetical; all authors contributed equally to this paper. We thank A. Bradley, O.M. Haynes, P. Horn, A. Mahler, C. Padilla, and C. Yuen. Supported by the Intramural Research Program of the NIH, NICHD.

    View full text