Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

3D shape perception from combined depth cues in human visual cortex

Abstract

Our perception of the world's three-dimensional (3D) structure is critical for object recognition, navigation and planning actions. To accomplish this, the brain combines different types of visual information about depth structure, but at present, the neural architecture mediating this combination remains largely unknown. Here, we report neuroimaging correlates of human 3D shape perception from the combination of two depth cues. We measured fMRI responses while observers judged the 3D structure of two sequentially presented images of slanted planes defined by binocular disparity and perspective. We compared the behavioral and fMRI responses evoked by changes in one or both of the depth cues. fMRI responses in extrastriate areas (hMT+/V5 and lateral occipital complex), rather than responses in early retinotopic areas, reflected differences in perceived 3D shape, suggesting 'combined-cue' representations in higher visual areas. These findings provide insight into the neural circuits engaged when the human brain combines different information sources for unified 3D visual perception.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Stimulus and design.
Figure 2: Regions of interest within the visual cortex.
Figure 3: fMRI results from Experiment 1.
Figure 4: Regressions of fMRI responses on psychophysical responses: fMRI data from Experiment 1 (asterisks, dashed lines) against corresponding psychophysical response index.
Figure 5: Design and results from Experiment 2.
Figure 6: Vergence control experiment and response time data.

Similar content being viewed by others

References

  1. Landy, M.S., Maloney, L.T., Johnston, E.B. & Young, M. Measurement and modeling of depth cue combination – in defense of weak fusion. Vision Res. 35, 389–412 (1995).

    CAS  Google Scholar 

  2. Clark, J.J. & Yuille, A.L. Data fusion for sensory information processing systems (Kluwer Academic, Boston, 1990).

    Google Scholar 

  3. Hillis, J.M., Ernst, M.O., Banks, M.S. & Landy, M.S. Combining sensory information: mandatory fusion within, but not between, senses. Science 298, 1627–1630 (2002).

    CAS  Google Scholar 

  4. Nguyenkim, J.D. & DeAngelis, G.C. Disparity-based coding of three-dimensional surface orientation by macaque middle temporal neurons. J. Neurosci. 23, 7117–7128 (2003).

    CAS  Google Scholar 

  5. Backus, B.T., Fleet, D.J., Parker, A.J. & Heeger, D.J. Human cortical activity correlates with stereoscopic depth perception. J. Neurophysiol. 86, 2054–2068 (2001).

    CAS  Google Scholar 

  6. Tsao, D.Y. et al. Stereopsis activates V3A and caudal intraparietal areas in macaques and humans. Neuron 39, 555–568 (2003).

    CAS  Google Scholar 

  7. Tsutsui, K., Sakata, H., Naganuma, T. & Taira, M. Neural correlates for perception of 3D surface orientation from texture gradient. Science 298, 409–412 (2002).

    CAS  Google Scholar 

  8. Liu, Y., Vogels, R. & Orban, G.A. Convergence of depth from texture and depth from disparity in macaque inferior temporal cortex. J. Neurosci. 24, 3795–3800 (2004).

    CAS  Google Scholar 

  9. Grill-Spector, K. & Malach, R. fMR-adaptation: a tool for studying the functional properties of human cortical neurons. Acta Psychol. (Amst.) 107, 293–321 (2001).

    CAS  Google Scholar 

  10. Kourtzi, Z. & Kanwisher, N. Representation of perceived object shape by the human lateral occipital complex. Science 293, 1506–1509 (2001).

    CAS  Google Scholar 

  11. Knill, D.C. & Saunders, J.A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003).

    Google Scholar 

  12. Grill-Spector, K., Kourtzi, Z. & Kanwisher, N. The lateral occipital complex and its role in object recognition. Vision Res. 41, 1409–1422 (2001).

    CAS  Google Scholar 

  13. Zeki, S. et al. A direct demonstration of functional specialization in human visual cortex. J. Neurosci. 11, 641–649 (1991).

    CAS  Google Scholar 

  14. Tootell, R.B. et al. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging. J. Neurosci. 15, 3215–3230 (1995).

    CAS  Google Scholar 

  15. Kourtzi, Z., Buelthoff, H.H., Erb, M. & Grodd, W. Object-selective responses in the human motion area MT/MST. Nat. Neurosci. 5, 17–18 (2002).

    CAS  Google Scholar 

  16. Orban, G.A. et al. Similarities and differences in motion processing between the human and macaque brain: evidence from fMRI. Neuropsychologia 41, 1757–1768 (2003).

    Google Scholar 

  17. Janssen, P., Vogels, R. & Orban, G.A. Selectivity for 3D shape that reveals distinct areas within macaque inferior temporal cortex. Science 288, 2054–2056 (2000).

    CAS  Google Scholar 

  18. Neri, P., Bridge, H. & Heeger, D.J. Stereoscopic processing of absolute and relative disparity in human visual cortex. J. Neurophysiol. 92, 1880–1891 (2004).

    Google Scholar 

  19. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    CAS  Google Scholar 

  20. Enright, J.T. Art and the oculomotor system: perspective illustrations evoke vergence changes. Perception 16, 731–746 (1987).

    CAS  Google Scholar 

  21. Ress, D., Backus, B.T. & Heeger, D.J. Activity in primary visual cortex predicts performance in a visual detection task. Nat. Neurosci. 3, 940–945 (2000).

    CAS  Google Scholar 

  22. Thomas, O.M., Cumming, B.G. & Parker, A.J. A specialization for relative disparity in V2. Nat. Neurosci. 5, 472–478 (2002).

    CAS  Google Scholar 

  23. Cumming, B.G. & Parker, A.J. Responses of primary visual cortical neurons to binocular disparity without depth perception. Nature 389, 280–283 (1997).

    CAS  Google Scholar 

  24. Watanabe, M., Tanaka, H., Uka, T. & Fujita, I. Disparity-selective neurons in area V4 of macaque monkeys. J. Neurophysiol. 87, 1960–1973 (2002).

    Google Scholar 

  25. Nienborg, H., Bridge, H., Parker, A.J. & Cumming, B.G. Receptive field size in V1 neurons limits acuity for perceiving disparity modulation. J. Neurosci. 24, 2065–2076 (2004).

    CAS  Google Scholar 

  26. Taira, M., Tsutsui, K.I., Jiang, M., Yara, K. & Sakata, H. Parietal neurons represent surface orientation from the gradient of binocular disparity. J. Neurophysiol. 83, 3140–3146 (2000).

    CAS  Google Scholar 

  27. Shikata, E. et al. Surface orientation discrimination activates caudal and anterior intraparietal sulcus in humans: an event-related fMRI study. J. Neurophysiol. 85, 1309–1314 (2001).

    CAS  Google Scholar 

  28. Poggio, G.F., Gonzalez, F. & Krause, F. Stereoscopic mechanisms in monkey visual cortex: binocular correlation and disparity selectivity. J. Neurosci. 8, 4531–4550 (1988).

    CAS  Google Scholar 

  29. von der Heydt, R., Zhou, H. & Friedman, H.S. Representation of stereoscopic edges in monkey visual cortex. Vision Res. 40, 1955–1967 (2000).

    CAS  Google Scholar 

  30. Hinkle, D.A. & Connor, C.E. Three-dimensional orientation tuning in macaque area V4. Nat. Neurosci. 5, 665–670 (2002).

    CAS  Google Scholar 

  31. Bakin, J.S., Nakayama, K. & Gilbert, C.D. Visual responses in monkey areas V1 and V2 to three-dimensional surface configurations. J. Neurosci. 20, 8188–8198 (2000).

    CAS  Google Scholar 

  32. Tanaka, H., Uka, T., Yoshiyama, K., Kato, M. & Fujita, I. Processing of shape defined by disparity in monkey inferior temporal cortex. J. Neurophysiol. 85, 735–744 (2001).

    CAS  Google Scholar 

  33. Uka, T., Tanaka, H., Yoshiyama, K., Kato, M. & Fujita, I. Disparity selectivity of neurons in monkey inferior temporal cortex. J. Neurophysiol. 84, 120–132 (2000).

    CAS  Google Scholar 

  34. Janssen, P., Vogels, R., Liu, Y. & Orban, G.A. Macaque inferior temporal neurons are selective for three-dimensional boundaries and surfaces. J. Neurosci. 21, 9419–9429 (2001).

    CAS  Google Scholar 

  35. Gilaie-Dotan, S., Ullman, S., Kushnir, T. & Malach, R. Shape-selective stereo processing in human object-related visual areas. Hum. Brain Mapp. 15, 67–79 (2002).

    Google Scholar 

  36. Bradley, D.C., Chang, G.C. & Andersen, R.A. Encoding of three-dimensional structure-from-motion by primate area MT neurons. Nature 392, 714–717 (1998).

    CAS  Google Scholar 

  37. Palanca, B.J. & DeAngelis, G.C. Macaque middle temporal neurons signal depth in the absence of motion. J. Neurosci. 23, 7647–7658 (2003).

    CAS  Google Scholar 

  38. Dodd, J.V., Krug, K., Cumming, B.G. & Parker, A.J. Perceptually bistable three-dimensional figures evoke high choice probabilities in cortical area MT. J. Neurosci. 21, 4809–4821 (2001).

    CAS  Google Scholar 

  39. Sary, G., Vogels, R. & Orban, G.A. Cue-invariant shape selectivity of macaque inferior temporal neurons. Science 260, 995–997 (1993).

    CAS  Google Scholar 

  40. Grill-Spector, K., Kushnir, T., Edelman, S., Itzchak, Y. & Malach, R. Cue-invariant activation in object-related areas of the human occipital lobe. Neuron 21, 191–202 (1998).

    CAS  Google Scholar 

  41. Albright, T.D. Cortical processing of visual motion. Rev. Oculomot. Res. 5, 177–201 (1993).

    CAS  Google Scholar 

  42. Treue, S. & Andersen, R.A. Neural responses to velocity gradients in macaque cortical area MT. Vis. Neurosci. 13, 797–804 (1996).

    CAS  Google Scholar 

  43. Murray, S.O., Olshausen, B.A. & Woods, D.L. Processing shape, motion and three-dimensional shape-from-motion in the human cortex. Cereb. Cortex 13, 508–516 (2003).

    Google Scholar 

  44. Paradis, A.L. et al. Visual perception of motion and 3-D structure from motion: an fMRI study. Cereb. Cortex 10, 772–783 (2000).

    CAS  Google Scholar 

  45. Taira, M., Nose, I., Inoue, K. & Tsutsui, K. Cortical areas related to attention to 3D surface structures based on shading: an fMRI study. Neuroimage 14, 959–966 (2001).

    CAS  Google Scholar 

  46. Sereno, M.E., Trinath, T., Augath, M. & Logothetis, N.K. Three-dimensional shape representation in monkey cortex. Neuron 33, 635–652 (2002).

    CAS  Google Scholar 

  47. van Ee, R., van Dam, L.C. & Erkelens, C.J. Bi-stability in perceived slant when binocular disparity and monocular perspective specify different slants. J. Vis. 2, 597–607 (2002).

    Google Scholar 

  48. Backus, B.T. in Advances in Neural Information Processing Systems 14 (eds. Dietterich, T.G., Becker, S. & Ghahramani, Z.) (MIT Press, Cambridge, Massachusetts, 2002).

    Google Scholar 

  49. Todd, J.T. The visual perception of 3D shape. Trends Cogn. Sci. 8, 115–121 (2004).

    Google Scholar 

  50. DeAngelis, G.C. & Newsome, W.T. Organization of disparity-selective neurons in macaque area MT. J. Neurosci. 19, 1398–1415 (1999).

    CAS  Google Scholar 

Download references

Acknowledgements

Preliminary reports of this work were presented at the VisionSciences Society's 2003 meeting and at the Society for Neuroscience's 2003 meeting. Thanks to S. Maier and J. Lam for help with data collection, and N. Logothetis, N. Kanwisher, J. Harris, S. McDonald, M. Ernst and R. Fleming for helpful discussions and comments. Thanks also to R. van Ee for advice on stimulus generation. Supported by an Alexander von Humboldt Fellowship to A.E.W., the Max-Planck Society and DFG grant TH812/1-1.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zoe Kourtzi.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Fig. 1

fMRI responses separated to account for between-subjects variability in cue weights. (PDF 79 kb)

Supplementary Fig. 2

Eye movement controls. (PDF 367 kb)

Supplementary Fig. 3

Results from an experiment on consistent-cue stimuli. (PDF 204 kb)

Supplementary Fig. 4

Additional example flatmaps showing examined ROIs. (PDF 737 kb)

Supplementary Fig. 5

Fits to the time course of the fMRI responses in the examined ROIs. (PDF 377 kb)

Supplementary Fig. 6

Mean peak fMRI response across the visual areas when individual reference stimuli are presented alone. (PDF 191 kb)

Supplementary Table 1

Measurement of psychophysical behavior of each observer. (PDF 515 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Welchman, A., Deubelius, A., Conrad, V. et al. 3D shape perception from combined depth cues in human visual cortex. Nat Neurosci 8, 820–827 (2005). https://doi.org/10.1038/nn1461

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn1461

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing