Sampling in developmental science: Situations, shortcomings, solutions, and standards☆
Introduction
When we undertake to study some phenomenon, we wish to know something about that phenomenon in a population, but in practice we study the phenomenon in a group of individuals who purportedly represent the target or reference population to whom we wish our results to generalize. That is, we sample the population. We sample because we normally do not command the resources (time, money, or personnel) to assess the entire population of interest. Sampling is therefore a key feature of every study in developmental science, and sampling has far-reaching implications in all studies. This article is concerned with sampling in developmental science. As we point out, different sampling strategies exist, and each has its implications. Employing sub-optimal sampling strategies is far too common in developmental research, compromises the validity and utility of the research, renders replication and cross-study comparisons difficult, and most generally impedes progress in the field of developmental science.
In this article, we briefly describe and illustrate four prominent strategies that answer the sampling challenge, and we evaluate each in terms of some fundamental, meaningful, and practical criteria. The four strategies include (a) population-based probability sampling as well as nonprobability sampling strategies such as (b) convenience sampling, (c) quota sampling, and (d) homogeneous sampling. The five criteria by which we appraise these sampling strategies include (a) whether they yield representative and generalizable estimates of a study’s target population (e.g., estimates of intelligence among the population when all sociodemographic groups are collapsed), (b) whether they yield representative and generalizable estimates of sociodemographic group differences within a study’s target population (e.g., how estimates of intelligence vary across a population’s ethnic groups), (c) the recruitment efforts and costs they entail, (d) whether they provide sufficient power to detect sociodemographic group differences, and (e) whether they introduce noise related to variation in sociodemographic factors and whether that noise can be accounted for statistically. After overviewing the four sampling strategies, we examine how the sociodemographic composition of a sample in terms of gender, ethnicity, and SES can compromise a study’s findings – regardless of the study goals. We then recount the use of each prominent sampling strategy in five high-profile journals in contemporary developmental science. On these bases, we arrive at conclusions and recommendations about best practices and practical considerations, including ethical issues, and discuss the importance of weighing the research question when considering the merits of various sampling strategies.
This article is not comprehensive, and we have not assumed some related burdens. By now demographers, sociologists, and others in many disciplines have weighed the pros and cons of different sampling strategies (Davis-Kean and Jager, 2011, Henry, 1990, Onwuegbuzie and Collins, 2007, Sue, 1999, Watters and Biernacki, 1989). This article does not provide a tutorial on sampling (see http://stattrek.com/statistics/data-collection-methods.aspx?Tutorial=Stat). We also eschew technical details in favor of highlighting “big picture” issues of design and practicality in an accessible way. Although our examples and arguments are applicable to any single sociodemographic factor or set of sociodemographic factors, here we limit our focus to gender, ethnicity, and SES. Also, although we fully recognize that gender, ethnicity, and SES are non-independent (ethnicity and SES in particular) but interact in myriad complex ways, when discussing the implications of these three sociodemographic factors we typically limit our examples to a single factor for the sake of conceptual clarity. Finally, for the purposes of this exposition about sampling, we combine “race” and “ethnicity” as used by the U.S. Government and its agencies and define six ethnic categories (see Table 1). We acknowledge that these ethnic groups are also heterogeneous in that each group contains people who originated from many different countries with different cultural practices.
Section snippets
Common sampling strategies in developmental science
Here we describe four of the most used sampling strategies, and we assess their advantages, disadvantages, and limitations. How each of the four sampling strategies fares on the five criteria is summarized in Table 2.
Why the sociodemographic composition of samples matters
Given the advantages and disadvantages of the four sampling strategies, it is important to note how sociodemographic characteristics can affect study outcomes and the interpretation of study results. Gender, ethnicity, and SES variation in many characteristics—physical and mental health (Adkins et al., 2009, American Public Health Association, 2004, Breslau et al., 2005, Crimmins and Saito, 2001, National Center for Health Statistics, 2011), beliefs and cognitions (Bornstein and Cote, 2004,
How sampling is used in developmental science
To determine the frequency with which the four different sampling methods are used in contemporary developmental science, we surveyed five years (2007–2011) of five high-profile developmental journals. The five journals included two that generally focus on abnormal development, (Journal of the American Academy of Child and Adolescent Psychiatry and Development and Psychopathology), two that generally focus on normative development (Developmental Psychology and Child Development), and one that
Implications and recommendations for reporting sample characteristics
The analysis of published articles reveals several inconsistencies and inadequacies in basic reporting of demographic characteristics of study samples in developmental science. Overall, 41.4% of articles published in five high-profile developmental science journals failed to report ethnicity at all or only reported that the sample was “predominantly White” or “about half minority”, which is not detailed enough to draw conclusions about the representativeness of the sample to a particular
Summary and conclusions
Here we have recounted four common sampling strategies (population-based probability sampling, convenience sampling, quota sampling, and homogenous sampling) and evaluated each by five meaningful criteria. We find that by far the most common sampling strategy (convenience sampling) is the least desirable in terms of representativeness, generalizability, and noise. Convenience samples require less practical investment in recruitment costs and efforts, but this advantage does not offset its
References (46)
Cross-cultural developmental comparisons: The case of Japanese–American infant and mother activities and interactions. What we know, what we need to know, and why we need to know
Developmental Review
(1989)- et al.
Trends in health life expectancy in the United States, 1970–1990: Gender, racial, and educational differences
Social Science and Medicine
(2001) - et al.
Body mass index and depressive symptoms in older adults: The moderating roles of race, sex, and socioeconomic status
American Journal of Geriatric Psychiatry
(2007) - et al.
Structure and stress: Trajectories of depressive symptoms across adolescence and young adulthood
Social Forces
(2009) Eliminating health disparities: Toolkit
(2004)- et al.
Racial differences in adolescent drug use: The impact of religion
Substance use and Misuse
(1996) - et al.
The national longitudinal study of adolescent health: Research design
(1997) - et al.
The study of culture, ethnicity, and race in American Psychology
American Psychologist
(1993) - et al.
The effects of race/ethnicity, income, and family structure on adolescent risk behaviors
American Journal of Public Health
(2000)
Mothers’ parenting cognitions in cultures of origin, acculturating cultures, and cultures of destination
Child Development
Play in two societies: Pervasiveness of process, specificity of structure
Child Development
Functional analysis of the contents of maternal speech to infants of 5 and 13 months in four cultures: Argentina, France, Japan, and the United States
Developmental Psychology
Ideas about parenting in Argentina, France, and the United States
International Journal of Behavioral Development
Specifying race-ethnic differences in risk for psychiatric disorder in a USA national sample
Psychological Medicine
Correlates of obesity in young Black and White women: The CARDIA study
American Journal of Public Health
Direct and indirect aggression during childhood and adolescence: A meta-analytic review of gender differences, intercorrelations, and relations among adjustment
Child Development
Sampling techniques
Gender and ethnic differences in health beliefs and behaviors
Journal of Health Psychology
The use of large-scale data sets for the study of developmental science
Racial differences in attitudes towards professional mental health care and the use of services
American Journal of Orthopsychiatry
G∗Power 3: A flexible statistical power analysis for the social, behavioral, and biomedical sciences
Behavior Research Methods
Cited by (364)
Association between residential greenspace and mental health among cancer survivors in Shanghai, China
2023, Environmental ResearchA Room with an Overview: Toward Meaningful Transparency for the Consumer Internet of Things
2024, IEEE Internet of Things JournalA repeated cross-sectional study on the implementation of electronic medication management system
2024, Information Technology and ManagementGender essentialism predicts prejudice against gender nonconformity in two cultural contexts
2024, Social DevelopmentDifferences and variability of physical and technical characteristics among rugby union small-sided games performed within a preseason
2024, International Journal of Sports Science and Coaching
- ☆
Authorship is alphabetical; all authors contributed equally to this paper. We thank A. Bradley, O.M. Haynes, P. Horn, A. Mahler, C. Padilla, and C. Yuen. Supported by the Intramural Research Program of the NIH, NICHD.