Skip to main content
SpringerLink
Log in

Statistical Analysis of Choice Experiments and Surveys

  • Published:
Marketing Letters Aims and scope Submit manuscript

Abstract

Measures of households' past behavior, their expectations with respect to future events and contingencies, and their intentions with respect to future behavior are frequently collected using household surveys. These questions are conceptually difficult. Answering them requires elaborate cognitive and social processes, and often respondents report only their “best” guesses and/or estimates, using more or less sophisticated heuristics. A large body of literature in psychology and survey research shows that as a result, responses to such questions may be severely biased. In this paper, (1) we describe some of the problems that are typically encountered, (2) provide some empirical illustrations of these biases, and (3) develop a framework for conceptualizing survey response behavior and for integrating structural models of response behavior into the statistical analysis of the underlying economic behavior.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adams, P., M. D. Hurd, D. McFadden, A. Merrill, and T. Ribeiro. (2003). “Healthy, Wealthy, and Wise? Tests for Direct Causal Paths Between Health and Socioeconomic Status,” Journal of Econometrics 112(1), 3–56.

    Google Scholar 

  • Alexander, C. S. and H. J. Becker. (1978). “The Use of Vignettes in Survey Research,” Public Opinion Quarterly 42(1), 93–104.

    Article  Google Scholar 

  • Allais, M. (1953). “Le Comportement De L'homme Rationnel Devant Le Risque: Critique Des Postulats Et Axiomes De L'école Américaine,” Econometrica 21(4), 503–546.

    Google Scholar 

  • Banks, J., A. Kapteyn, J. P. Smith, and A. van Soest. (2004). “International Comparisons of Work Disability,” Working Paper No. 155, Labor and Population Group, RAND, Santa Monica, CA.

  • Bemmaor, A. C. (1995). “Predicting Behavior from Intention-to-buy Measures: The Parametric Case,” Journal of Marketing Research 32(May), 176–191.

    Google Scholar 

  • Ben-Akiva, M., D. McFadden, M. Abe, U. BÖckenholt, D. Bolduc, D. Gopinath, T. Morikawa, V. Ramaswamy, V. Rao, D. Revelt, and D. Steinberg. (1997). “Modeling Methods for Discrete Choice Analysis,” Marketing Letters 8(3), 273–286.

    Article  Google Scholar 

  • Ben-Akiva, M., D. McFadden, T. Gärling, D. Gopinath, D. Bolduc, A. BÖrsch-Supan, P. Delquié, O. Larichev, T. Morikawa, A. Polydoropoulou, and V. Rao. (1999). “Extended Framework for Modeling Choice Behavior,” Marketing Letters 10(3), 187–203.

    Article  Google Scholar 

  • Ben-Akiva, M., D. McFadden, K. Train, J. Walker, C. Bhat, M. Bierlaire, D. Bolduc, A. BÖrsch-Supan, D. Brownstone, D. S. Bunch, A. Daly, A. de Palma, D. Gopinath, A. Karlstrom, and M. Munizaga. (2002). “Hybrid Choice Models: Progress and Challenges,” Marketing Letters 13(3), 163–175.

    Article  Google Scholar 

  • Bergadaá, M. M. (1990). “The role of Time in the Action of the Consumer,” Journal of Consumer Research 17(3), 289–302.

    Google Scholar 

  • Bound, J., C. Brown, and N. Mathiowetz. (2001). “Measurement Error in Survey Data.” In J. J. Heckman and E. Leamer (eds.), Handbook of Econometrics, Volume 5, Amsterdam: Elsevier, pp. 3705–3843.

    Google Scholar 

  • Bruine de Bruin, W., P. S. Fischbeck, N. A. Stiber, and B. Fischhoff. (2002). “What Number is Fifty-Fifty? Redistributing Excessive 50% Responses in Elicited Probabilities,” Risk Analysis 22(4), 713–723.

    Google Scholar 

  • Buchanan, B. and G. Morrison. (1987). “Sampling Properties of Rate Questions with Implications for Survey Research,” Marketing Science 16(3), 286–298.

    Google Scholar 

  • Chandon, P., V. G. Morwitz, and W. J. Reinartz. (2005). “Do intentions Really Predict Behavior? Self-generated Validity Effects in Survey Research,” Journal of Marketing 69(2), 1–14.

    Article  Google Scholar 

  • Cowing, T. and D. McFadden. (1984). Microeconomic Modeling and Policy Analysis: Studies in Residential Energy Demand. New York: Academic Press.

    Google Scholar 

  • Crossley, T. F. and S. Kennedy. (2002). “The Reliability of Self-assessed Health Status,” Journal of Health Economics 21(4), 643–658.

    Article  Google Scholar 

  • Davidson, D., P. Suppes, and S. Siegel. (1956). Decision Making: An Experimental Approach. Stanford, CA: Stanford University Press.

    Google Scholar 

  • Dex, S. (1995). “The Reliability of Recall Data: A literature Review,” Bulletin de Méthodologie Sociologique, 49, December, 58–89.

    Google Scholar 

  • Dominitz, J., and C. F. Manski. (1997). “Using Expectations Data to Study Subjective Income Expectations,” Journal of the American Statistical Association 92(439), 855–867.

    Google Scholar 

  • Dominitz, J. and C. F. Manski. (2004). “How Should We Measure Consumer Confidence?” Journal of Economic Perspectives 18(2), 51–66.

    Article  Google Scholar 

  • Feldman, J. M. and J. G. Lynch Jr. (1988). “Self-generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior,” Journal of Applied Psychology 73(3), 421–435.

    Article  Google Scholar 

  • Gan, L., M. D. Hurd, and D. McFadden. (2003). “Individual Subjective Survival Curves,” Working Paper No. 9480, National Bureau of Economic Research (NBER), Cambridge, MA.

    Google Scholar 

  • Gardiner, J. M., and A. Richardson-Klavehn. (2000). “Remembering and Knowing,” in E. Tulving and F. I. M. Craik (eds.), Handbook of Memory, New York: Oxford University Press, pp. 229–244.

    Google Scholar 

  • Granbois, D. H. and J. O. Summers. (1975). “Primary and Secondary Validity of Consumer Purchase Probabilities,” Journal of Consumer Research 1(4), 31–38.

    Article  Google Scholar 

  • Harris, K. M. and M. P. Keane. (1999). “A Model of Health Plan Choice: Inferring Preferences and Perceptions from a Combination of Revealed Preference and Attitudinal Data,” Journal of Econometrics 89(1–2), 131–157.

    Google Scholar 

  • Hill, D., M. Perry, and R. J. Willis. (2004). “Estimating Knightian Uncertainty from Probability Questions on the HRS,” unpublished manuscript, University of Michigan.

  • Hsiao, C. and B.-H. Sun. (1999), “Modeling Response Bias – With an Analysis of the Demand for an Advanced Electronic Device,” Journal of Econometrics 89(1–2), 15–39.

    Google Scholar 

  • Hurd, M. D., D. McFadden, H. Chand, L. Gan, A. Merrill, and M. Roberts. (1998). “Consumption and Savings Balances of the Elderly: Experimental Evidence on Survey Response Bias.” in D. Wise (ed.), Frontiers in the Economics of Aging, Chicago, IL: University of Chicago Press, pp. 353–387.

    Google Scholar 

  • Hurd, M. D. and K. McGarry. (1995). “Evaluation of the Subjective Probabilities of Survival in the Health and Retirement Study,” Journal of Human Resources 30(Supplement), S268–S292.

    Google Scholar 

  • Hurd, M. D. and K. McGarry. (2002). “The Predictive Validity of Subjective Probabilities of Survival,” Economic Journal 112(482), 966–985.

    Article  Google Scholar 

  • Infosino, W. J. (1986). “Forecasting New Product Sales from Likelihood of Purchase Ratings,” Marketing Science 5(4), 372–384.

    Google Scholar 

  • Kalwani, M. U. and A. J. Silk. (1982). “On the Reliability and Predictive Validity of Purchase Intention Measures,” Marketing Science 1(3), 243–286.

    Google Scholar 

  • King, G., C. Murray, J. A. Salomon, and A. Tandon. (2004). “Enhancing the Validity and Cross-cultural Comparability of Measurement in Survey Research,” American Political Science Review 98(1), 191–207.

    Article  Google Scholar 

  • Klein, R. and R. Sherman. (1997). “Estimating New Product Demand from Biased Survey Data,” Journal of Econometrics 76(1–2), 53–76.

    Google Scholar 

  • Lewbel, A. (2004). “Simple Estimators for Hard Problems: Endogeneity in Binary Choice and Other Nonlinear Models,” unpublished manuscript, Boston College.

  • Louviere, J. J., D. A. Hensher, and J. D. Swait (2000). Stated Choice Methods: Analysis and Application. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Louviere, J. J., D. Street, R. Carson, A. Ainslie, J. R. Deshazo, T. Cameron, D. Hensher, R. Kohn, and T. Marley. (2002). “Dissecting the Random Component of Utility”, Marketing Letters 13(3), 163–176.

    Article  Google Scholar 

  • Manski, C. F. (1999). “Analysis of Choice Expectations in Incomplete Scenarios,” Journal of Risk and Uncertainty 19(1–3), 49–65.

    Google Scholar 

  • Manski, C. F. (2004). “Measuring Expectations,” Econometrica 72(5), 1329–1376.

    Article  Google Scholar 

  • Matzkin, R. (2004). “Unobservable Instruments,” unpublished manuscript, Northwestern University.

  • McFadden, D. (2004). “Foundations of Economic Survey Research,” unpublished manuscript, Berkeley: University of California.

  • McFadden, D., N. Schwarz, and J. Winter. (2004), “Measuring Perceptions and Behavior in Household Surveys,” paper presented at the Annual Meeting of the American Economic Association, San Diego, CA.

  • McFadden, D. and K. Train (2000), “Mixed MNL Models for Discrete Response,” Journal of Applied Econometrics 15(5), 447–470.

    Article  Google Scholar 

  • McGarry, K. (2004). “Health and Retirement: Do Changes in Health Affect Retirement Expectations?” Journal of Human Resources 39(3), 624–648.

    Google Scholar 

  • Menon, G., O. Raghubir, and N. Schwarz. (1997). “How Much Will I Spend? Factors Affecting Consumers' Estimates of Future Expense,” Journal of Consumer Psychology 6(2), 141–164.

    Article  Google Scholar 

  • Morwitz, V. G. and G. J. Fitzsimons. (2004). “The Mere Measurement Effect: Why Does Measuring Intentions Change Actual Behavior?” Journal of Consumer Psychology 14(1–2), 64–74.

    Google Scholar 

  • Molinari, F. (2005). “Cognitive Processes and Self/proxy Response Status: Evidence from HRS 2000,” unpublished manuscript, Cornell University.

  • Poterba, J. and L. Summers (1986). “Reporting Errors and Labor Market Dynamics,” Econometrica 54(6), 1319–1338.

    Google Scholar 

  • Schwarz, N. (2003). “Self-Reports in Consumer Research: The Challenge of Comparing Cohorts and Cultures,” Journal of Consumer Research 29(4), 588–594.

    Article  Google Scholar 

  • Schwarz, N., R. M. Groves, and H. Schuman. (1998). “Survey Methods.” In N. Schwarz, R. Groves, and H. Schuman (eds.), Handbook of Social Psychology, 4th Edition, New York: McGraw-Hill, pp. 143–179.

    Google Scholar 

  • Schwarz, N., H. J. Hippler, B. Deutsch, and F. Strack. (1985). “Response Categories: Effects on Behavioral Reports and Comparative Judgements,” Public Opinion Quarterly 49(3), 388–395.

    Article  Google Scholar 

  • Smith, V. K., D. H. Taylor, and F. A. Sloan. (2001). “Longevity Expectations and Death: Can People Predict Their Own Demise?” American Economic Review 91(4), 1126–1134.

    Google Scholar 

  • Tourangeau, R., L. J. Rips, and K. Rasinski. (2000). The Psychology of Survey Response. New York and Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Tversky, A. and D. Kahneman (1974). “Judgement Under Uncertainty: Heuristics and Biases,” Science 185(4157), 1124–1131.

    Google Scholar 

  • Wansbeek, T. and E. Meijer (2000). Measurement Error and Latent Variables in Econometrics, Amsterdam: Elsevier.

    Google Scholar 

  • Wason, K., Polonsky, M., and Hyman, M. (2002). “Designing Vignette Studies in Marketing,” Australasian Marketing Journal 10(3), 41–58.

    Google Scholar 

  • Winter, J. (2002). “Bracketing Effects in Categorized Survey Questions and the Measurement of Economic Quantities,” Discussion Paper No. 02-35, Sonderforschungsbereich 504, University of Mannheim.

Download references

Author information

Authors and Affiliations

  1. Econometrics Laboratory, University of California, 549 Evans Hall, Berkeley, CA, 04720-3880

    Daniel L. McFadden (Workshop chair)

  2. ESSEC, BP 50105, 95021, Cergy-Pontoise Cedex, France

    Albert C. Bemmaor

  3. Gerontology Institute, University of Massachusetts Boston, 100 Morrissey Blvd., Boston, MA, 02125

    Francis G. Caro

  4. H. John Heinz III, School of Public Policy and Management, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213

    Jeff Dominitz

  5. Econometrics Laboratory, University of California, 549 Evans Hall, Berkeley, CA, 04720-3880

    Byung-Hill Jun

  6. Department of Economics, Boston College, 140 Commonwealth Avenue, Chestnut Hill, MA, 02467

    Arthur Lewbel

  7. Department of Economics, Northwestern University, 2001 Sheridan Road, Evanston, IL, 60208

    Rosa L. Matzkin

  8. Department of Economics, Cornell University, 492 Uris Hall, Ithaca, NY, 14853-7601

    Francesca Molinari

  9. Institute for Social Research, University of Michigan, 426 Thompson St., Ann Arbor, MI, 48106-1248

    Norbert Schwarz

  10. Department of Economics and Institute for Social Research, University of Michigan, 426 Thompson St., Ann Arbor, MI, 48104-1248

    Robert J. Willis

  11. Department of Economics, University of Munich, 80539, Munich, Germany

    Joachim K. Winter

Authors
  1. Daniel L. McFadden
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Albert C. Bemmaor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francis G. Caro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeff Dominitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Byung-Hill Jun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Arthur Lewbel
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rosa L. Matzkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Francesca Molinari
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Norbert Schwarz
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Robert J. Willis
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Joachim K. Winter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joachim K. Winter.

Rights and permissions

Reprints and permissions

About this article

Cite this article

McFadden, D.L., Bemmaor, A.C., Caro, F.G. et al. Statistical Analysis of Choice Experiments and Surveys. Market Lett 16, 183–196 (2005). https://doi.org/10.1007/s11002-005-5884-2

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11002-005-5884-2

Keywords

Search

Navigation