Skip to main content
Original Article

Experimental Vignette Studies in Survey Research

Published Online:https://doi.org/10.1027/1614-2241/a000014

Vignette studies use short descriptions of situations or persons (vignettes) that are usually shown to respondents within surveys in order to elicit their judgments about these scenarios. By systematically varying the levels of theoretically important vignette characteristics a large population of different vignettes is typically available – too large to be presented to each respondent. Therefore, each respondent gets only a subset of vignettes. These subsets may either be randomly selected in following the tradition of the factorial survey or systematically selected according to an experimental design. We show that these strategies in selecting vignette sets have strong implications for the analysis and interpretation of vignette data. Random selection strategies result in a random confounding of effects and heavily rely on the assumption of no interaction effects. In contrast, experimental strategies systematically confound interaction effects with main or set effects, thereby preserving a meaningful interpretation of main and important interaction effects. Using a pilot study on attitudes toward immigrants we demonstrate the implementation and analysis of a confounded factorial design.

References

  • Alexander, C. S. , & Becker, H. J. (1978). The use of vignettes in survey research. Public Opinion Quarterly, 42, 93–104. First citation in articleCrossrefGoogle Scholar

  • Alves, W. M. , Rossi, P. H. (1978). Who should get what? Fairness judgments of the distribution of earnings. American Journal of Sociology, 84, 541–564. First citation in articleCrossrefGoogle Scholar

  • Atzmüller, C. (2006). Die Vignettenanalyse als empirisches Messmodell: Methodische und methodologische Neuerungen gezeigt am Beispiel der Untersuchung normativer (Vor-)Urteile unter Jugendlichen zum Thema Integration von MigrantInnen [The vignette analysis as empirical model: Methodological improvements shown as an example for a normative prejudice/judgement survey of adolecents with the topic “integration of immigrants”]. University of Vienna: Vienna Unpublished doctorial dissertation. First citation in articleGoogle Scholar

  • Auspurg, K. , Hinz, T. , & Liebig, S. (2009). Komplexität von Vignetten, Lerneffekte und Plausibilität im Faktoriellen Survey [Complexity, learning effects and plausibility of vignettes in the factorial survey design]. Methoden – Daten – Analysen, 1, 59–96. First citation in articleGoogle Scholar

  • Barrera, D. , Buskens, V. (2002). Imitation and learning under uncertainty: A vignette experiment. ICS/Department of Sociology, Utrecht University. First citation in articleGoogle Scholar

  • Beck, M. , Opp, K. (2001). Der faktorielle Survey und die Messung von Normen [The factorial survey and the measurement of norms]. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 53, 283–306. First citation in articleCrossrefGoogle Scholar

  • Birnbaum, M. H. (1999). How to show that 9 > 221: Collect judgments in a between-subjects design. Psychological Methods, 4, 243–249. First citation in articleCrossrefGoogle Scholar

  • Cochran, W. G. , & Cox, G. M. (1950). Experimental designs (2 nd ed.). New York: Wiley. First citation in articleGoogle Scholar

  • Cook, F. L. (1977). Differences in public support for seven social welfare groups: Description and explanation University of Chicago Doctoral dissertation. First citation in articleGoogle Scholar

  • Cook, F. L. (1979). Who should be helped? Public support for social services. London: Sage. First citation in articleGoogle Scholar

  • Dülmer, H. (2001). Bildung und der Einfluss von Argumenten auf das moralische Urteil. Eine empirische Analyse zur moralischen Entwicklungstheorie Kohlbergs [Education and the influence of arguments for moral judgement: An empirical analysis of Kohlberg’s moral development theory]. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 53, 1–27. First citation in articleCrossrefGoogle Scholar

  • Dülmer, H. (2007). Experimental plans in factorial surveys. Random or quota design?. Sociological Methods & Research, 35, 382–409. First citation in articleCrossrefGoogle Scholar

  • Hox, J. J. , Kreft, I. G. G. , Hermkens, P. L. J. (1991). The analysis of factorial Surveys. Sociological Methods & Research, 19, 493–510. First citation in articleCrossrefGoogle Scholar

  • Hunter, C. , McClelland, K. (1991). Honoring accounts for sexual harassment: A factorial survey analysis. Sex Roles, 24, 725–752. First citation in articleCrossrefGoogle Scholar

  • Jasso, G. (2006). Factorial survey methods for studying beliefs and judgments. Sociological Methods & Research, 34, 334–423. First citation in articleCrossrefGoogle Scholar

  • Jasso, G. , Webster, M. Jr. (1999). Assessing the gender gap in just earnings and its underlying mechanisms. Social Psychology Quarterly, 62, 367–380. First citation in articleCrossrefGoogle Scholar

  • Jones, C. , Aronson, E. (1973). Attribution of fault to a rape victim as a function of respectability of the victim. Journal of Personality and Social Psychology, 26, 415–419. First citation in articleCrossrefGoogle Scholar

  • Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole. First citation in articleGoogle Scholar

  • Liebig, S. , Mau, S. (2005). Wann ist ein Steuersystem gerecht? Einstellungen zu allgemeinen Prinzipien der Besteuerung und zur Gerechtigkeit der eigenen Steuerlast [When is a system of taxation fair? Attitudes towards general principles of taxation and fairness of the own tax burden]. Duisburger Beiträge zur Soziologischen Forschung, 1. First citation in articleGoogle Scholar

  • Ludwick, R. , Zeller, R. A. (2001). The factorial survey: An experimental method to replicate real world problems. Nursing Research, 50, 129–133. First citation in articleCrossrefGoogle Scholar

  • Mäs, M. , Mühler, K. , Opp, K. (2005). Wann ist man Deutsch? Empirische Ergebnisse eines faktoriellen Surveys [When are you German? Empirical results of a factorial survey]. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 57, 112–134. First citation in articleCrossrefGoogle Scholar

  • Pinheiro, J. , Bates, D. , DebRoy, S. , & Sarkar, D. (2007). nlme: Linear and nonlinear mixed effects models. R package version, 3, 1–83. First citation in articleGoogle Scholar

  • Pizarro, D. A. , Uhlmann, E. , Bloom, P. (2003). Causal deviance and the attribution for moral responsibility. Journal of experimental social psychology, 39, 653–660. First citation in articleCrossrefGoogle Scholar

  • R Development Core Team . (2007). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved, April 15, 2007, from www.R-project.org. First citation in articleGoogle Scholar

  • Raudenbush, S. W. , & Bryk, A. S. (2002). Hierarchical linear models. Applications and data analysis methods (2nd ed.). London: Sage. First citation in articleGoogle Scholar

  • Rossi, P. H. (1979). Vignette analysis: Uncovering the normative structure of complex judgments. In R. K. Merton, J. S. Coleman, P. H. Rossi, (Eds.), Qualitative and quantitative social research. Papers in honor of Paul F. Larzarsfeld (pp. 176–186). New York: Free Press. First citation in articleGoogle Scholar

  • Rossi, P. H. , Anderson, A. B. (1982). The factorial survey approach: An introduction. In P. H. Rossi, S. L. Nock, (Eds.), Measuring social judgments: The factorial survey approach (pp. 15–67). Beverly Hills, CA: Sage. First citation in articleGoogle Scholar

  • Rossi, P. H. , Nock, S. L. (1982). Measuring social judgments: The factorial survey approach. Beverly Hills, CA: Sage. First citation in articleGoogle Scholar

  • Rossi, P. H. , Sampson, W. A. , Bose, C. E. , Jasso, G. , Passel, J. (1974). Measuring household social standing. Social Science Research, 3, 169–190. First citation in articleCrossrefGoogle Scholar

  • Rossi, P. H. , Simpson, J. E. , Miller, J. L. (1985). Beyond crime seriousness: Fitting the punishment to the crime. Journal of Quantitative Criminology, 1, 59–90. First citation in articleCrossrefGoogle Scholar

  • Rossi, P. H. , Waite, E. , Bose, C. E. , Berk, R. E. (1974). The seriousness of crimes: Normative structures and individual differences. American Sociological Review, 39, 224–237. First citation in articleCrossrefGoogle Scholar

  • Searle, S. R. (1987). Linear models for unbalanced data. New York: Wiley. First citation in articleGoogle Scholar

  • Seguin, C. A. , & Ambrosio, A. L. (2002). Multicultural vignettes for teacher preparation. Multicultural Perspectives, 4, 10–16. First citation in articleCrossrefGoogle Scholar

  • Shlay, A. B. , Tran, H. , Weinraub, M. , Harmon, M. (2004). How parents evaluate child care: A factorial survey analysis of perceptions of child care quality, fair market price and willingness to pay by low-income, African American mothers. Philadelphia: Temple University. First citation in articleGoogle Scholar

  • Sniderman, P. M. , & Grob, D. B. (1996). Innovations in experimental design in attitude surveys. Annual Review of Sociology, 22, 377–399. First citation in articleCrossrefGoogle Scholar

  • Steiner, P. M. , & Atzmüller, C. (2006). Experimentelle Vignettendesigns in faktoriellen Surveys [Experimental vignette designs in factorial surveys]. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 58, 117–146. First citation in articleCrossrefGoogle Scholar

  • Thurman, Q. C. , Lam, J. A. , & Rossi, P. H. (1988). Sorting out the cuckoo’s nest: A factorial survey approach to the study of popular conceptions of mental illness. Sociological Quarterly, 29, 565–588. First citation in articleCrossrefGoogle Scholar

  • Veal, W. R. (2002). Content specific vignettes as tools for research and teaching. Electronic Journal of Science Education, 6. First citation in articleGoogle Scholar

  • Walster, E. (1966). Assignment of responsibility for an accident. Journal of Personality and Social Psychology, 3, 73–79. First citation in articleCrossrefGoogle Scholar

  • Wason, K. D. , Polonsky, M. J. , Hyman, M. R. (2002). Designing vignette studies in marketing. Australasian Marketing Journal, 10, 41–58. First citation in articleCrossrefGoogle Scholar