nach oben

Brain Topography

Erschienen in:

01.05.2015 | Original Paper

Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms

verfasst von: Nicholas Altieri, Ryan A. Stevenson, Mark T. Wallace, Michael J. Wenger

Erschienen in: Brain Topography | Ausgabe 3/2015

Einloggen, um Zugang zu erhalten

Abstract

The ability to effectively combine sensory inputs across modalities is vital for acquiring a unified percept of events. For example, watching a hammer hit a nail while simultaneously identifying the sound as originating from the event requires the ability to identify spatio-temporal congruencies and statistical regularities. In this study, we applied a reaction time and hazard function measure known as capacity (e.g., Townsend and AshbyCognitive Theory 200–239, 1978) to quantify the extent to which observers learn paired associations between simple auditory and visual patterns in a model theoretic manner. As expected, results showed that learning was associated with an increase in accuracy, but more significantly, an increase in capacity. The aim of this study was to associate capacity measures of multisensory learning, with neural based measures, namely mean global field power (GFP). We observed a co-variation between an increase in capacity, and a decrease in GFP amplitude as learning occurred. This suggests that capacity constitutes a reliable behavioral index of efficient energy expenditure in the neural domain.

It is worth noting that these principles are not necessarily associated with stimulus features per se, but rather neural features including receptive field properties, effectiveness in eliciting action potentials, etc.

Contingencies, or probabilistic stimulus configurations that could facilitate/inhibit audiovisual target (i.e., matched) trial responses (Mordkoff and Yantis 1991), were reduced by appropriately balancing redundant, single, and target-absent trials. Although this yielded a difference in the number of matched and mismatched audiovisual trials, the proportion remained constant across training days. Hence, holding the proportion of audiovisual matched versus mismatched trials constant across days provides a valid way to examine the effects of training on changes in capacity and neural signals.

Allison PD (1996) Fixed-effects partial likelihood for repeated events. Sociol Methods Res 25:2207–2222CrossRef

Altieri N, Townsend JT (2011) An assessment of behavioral dynamic information processing measures in audiovisual speech perception. Front Psychol 2(238):1–15

Atkinson RC, Shiffrin RM (1968) Human memory: a proposed system and its control processes. In: Spence KW, Spence JT (eds) The psychology of learning and motivation, vol 2. Academic Press, New York, pp 89–195

Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823CrossRefPubMed

Belardinelli MO, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A et al (2004) Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn Process 5(3):167–174CrossRef

Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657CrossRefPubMed

Cappe C, Thut G, Romei V, Murray MM (2010) Auditory-visual multisensory interactions in humans: timing, topography, directionality, and sources. J Neurosci 30(38):12572–12580CrossRefPubMed

Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009CrossRefPubMed

Colonius H, Diederich A (2010) The optimal time window of visual-auditory integration: a reaction time analysis. Front Integr Neurosci 4:11PubMedCentralPubMed

Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073CrossRefPubMedCentralPubMed

Cowan N, Elliott EM, Saults JS, Morey CC, Mattox S, Hismjatullina A, Conway ARA (2005) On the capacity of attention: its estimation and its role in working memory and cognitive aptitudes. Cogn Psychol 51:42–100CrossRefPubMedCentralPubMed

Cox DR (1972) Regresion models and life tables (with discussion). J Roy Stat Soc B 34:187–220

Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404CrossRefPubMed

Diederich A, Colonius H (2009) Crossmodal interaction in speeded responses: time window of integration model. Prog Brain Res 174:119–135CrossRefPubMed

Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721CrossRefPubMed

Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150CrossRefPubMed

Dosher B, Lu ZL (1999) Mechanisms of perceptual learning. Vision Res 39:3197–3221CrossRefPubMed

Erber NP (2003) Use of hearing aids by older people: influence of non-auditory factors (vision, manual dexterity). Int J Audiol 42:2S21–2S25CrossRefPubMed

Fiebelkorn IC, Foxe JJ, Schwartz TH, Molholm S (2010) Staying within the lines: the formation of visuospatial boundaries influences multisensory feature integration. Eur J Neurosci 31(10):1737–1743CrossRefPubMed

Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys 57(6):802–816CrossRefPubMed

Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143CrossRefPubMed

Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27(30):7881–7887CrossRefPubMed

Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293CrossRefPubMed

Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467CrossRefPubMedCentralPubMed

James W (1890) The principles of psychology. Harvard University Press, BostonCrossRef

James TW, Stevenson RA (2012) The use of fMRI to assess multisensory integration. In: Wallace MH, Murray MM (eds) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London

James TW, VanDerKlok RM, Stevenson RA, James KH (2011) Multisensory perception of action in posterior temporal and parietal cortices. Neuropsychologia 49(1):108–114CrossRefPubMedCentralPubMed

James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge

Johnson SA, Blaha LM, Houpt JW, Townsend JT (2010) Systems Factorial Technology provides new insights on global-local information processing in autism spectrum disorders. J Math Psychol 54:53–72CrossRefPubMedCentralPubMed

Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640CrossRefPubMed

Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19(4):213–223CrossRefPubMed

Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453CrossRefPubMed

Luce RD (1986) Response times: their role in inferring elementary mental organization. Oxford University Press, New York

Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21(2):725–732CrossRefPubMed

McKee SP, Westheimer G (1978) Improvement in vernier acuity with practice. Percept Psychophys 24:258–262CrossRefPubMed

Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354CrossRefPubMed

Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662PubMed

Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75(5):1843–1857PubMed

Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I. Temporal factors. J Neurosci 7(10):3215–3229PubMed

Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893CrossRefPubMed

Molholm S, Ritter W, Murray M, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128CrossRef

Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538CrossRefPubMed

Munhall K, Vatikiotis-Bateson E (2004) Spatial and Temporal Constraints on Audiovisual Speech Perception. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 177–188

Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135CrossRefPubMed

Naue N, Rach S, Strüber D, Huster RJ, Zaehle T, Körner U, Herrmann CS (2011) Auditory event-related responses in visual cortex modulates subsequent visual responses in humans. J Neurosci 31(21):7729–7736CrossRefPubMed

Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley, RL (1998) Effects of localized auditory information on visual target detection performance using helmet-mounted display. Hum Factors 40(3):452–460CrossRefPubMed

Neufeld RWJ, Townsend JT, Jetté J (2007) Quantitative response time technology for measuring cognitive-processing capacity in clinical studies. In: Neufeld RWJ (ed) Advances in clinical cognitive science: formal modeling and assessment of processes and symptoms. American Psychological Association, Washington DCCrossRef

Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18(3):598–609CrossRefPubMed

Pilling M (2009) Auditory event-related potentials (ERPs) in audiovisual speech perception. J Speech Lang Hearing Res 52:1073–1081CrossRef

Powers AR, Hevey MA, Wallace MT (2012) Neural correlates of multisensory perceptual learning. J Neurosci 32(18):6263–6274CrossRefPubMedCentralPubMed

Powers AR 3rd, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274CrossRefPubMedCentralPubMed

Saffran JR, Johnson EK, Aslin RN, Newport EL (1999) Statistical learning of tone sequences by human infants and adults. Cognition 70:27–52CrossRefPubMed

Seitz A, Kim, Shams M (2006) Sound Facilitates Visual Learning. Curr Biol 16(14):1422–1427CrossRefPubMed

Seitz A, Kim, Van Wassenhove V, Shams M (2007) Simultaneous and Independent Acquisition of Multisensory and Unisensory Associations. Perception 36:1445–1453CrossRefPubMed

Shams M, Seitz A (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417CrossRefPubMed

Skrandies W (1990) Global field power and topographic similarity. Brain Topogr 3:147–541CrossRef

Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA (2009) Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198(2–3):113–126

Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44(3):1210–1223CrossRefPubMed

Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95CrossRefPubMed

Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194CrossRefPubMed

Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. Neuroimage 49(4):3308–3318CrossRefPubMedCentralPubMed

Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55(3):1339–1345CrossRefPubMedCentralPubMed

Stevenson RA, Krueger Fister J, Barnett ZP, Nidiffer AR, Wallace MT (2012) Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp Brain Res 219:121–137CrossRefPubMedCentralPubMed

Stevenson RA, Wilson MM, Powers AR, Wallace MT (2013) The effects of visual training on multisensory temporal processing. Exp Brain Res. doi:10.1007/s00221-012-3387-y

Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328CrossRefPubMedCentralPubMed

Tanabe S, Doi T, Umeda K, Fujita I (2005) Disparity-tuning characteristics of neuronal responses to dynamic random-dot stereograms in macaque visual area V4. J Neurophysiol 94(4):2683–2699CrossRefPubMed

Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci U S A 103(21):8239–8244CrossRefPubMedCentralPubMed

Thelen A, Murray MM (2013) The efficacy of single-trial multisensory memories. Multisens Res. doi:10.1163/22134808-00002426 PubMed

Thelen A, Cappe C, Murray MM (2012) Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62(3):1478–1488CrossRefPubMed

Townsend JT (1990) The truth and consequences of ordinal differences in statistical distributions: toward a theory of hierarchical inference. Psychol Bull 108:551–567CrossRefPubMed

Townsend JT, Altieri NA (2012) An accuracy-response time capacity assessment function that measures performance against standard parallel predictions. Psychol Rev. doi:10.1037/a0028448 PubMed

Townsend JT, Ashby FG (1978) Methods of modeling capacity in simple processing systems. In: Castellan J, Restle F (eds) Cognitive theory, vol III. Erlbaum Associates, Hillsdale, pp 200–239

Townsend JT, Nozawa G (1995) Spatio-temporal properties of elementary perception: an investigation of parallel, serial and coactive theories. J Math Psychol 39:321–360CrossRef

Townsend JT, Wenger MJ (2004) A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychol Rev 111(4):1003–1035CrossRefPubMed

van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974CrossRefPubMed

van Atteveldt N, Formisano E, Goebel R, Blomert, L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282CrossRefPubMed

van Wassenhove V, Grant K, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186CrossRefPubMedCentralPubMed

van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607CrossRefPubMed

Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22:32–35CrossRef

Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76(2):1246–1266PubMed

Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258CrossRefPubMed

Wallace MT, Carriere BN, Perrault TJ Jr, Vaughan JW, Stein BE (2006) The development of cortical multisensory integration. J Neurosci 26(46):11844–11849CrossRefPubMed

Wenger MJ, Gibson BS (2004) Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. J Exp Psychol Hum Percept Perform 30:708–719CrossRefPubMed

Wenger MJ, Negash S, Petersen RC, Petersen L (2010) Modeling and estimating recall processing capacity: sensitivity and diagnostic utility in application to mild cognitive impairment. J Math Psychol 54:73–89CrossRefPubMedCentralPubMed

Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544CrossRefPubMed

Titel: Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms
verfasst von: Nicholas Altieri
Ryan A. Stevenson
Mark T. Wallace
Michael J. Wenger
Publikationsdatum: 01.05.2015
Verlag: Springer US
Erschienen in: Brain Topography / Ausgabe 3/2015
Print ISSN: 0896-0267
Elektronische ISSN: 1573-6792
DOI: https://doi.org/10.1007/s10548-013-0333-7

Leitlinien kompakt für die Neurologie

Mit medbee Pocketcards sicher entscheiden.

^{Seit 2022 gehört die medbee GmbH zum Springer Medizin Verlag}

Kostenlos registrieren

Neu im Fachgebiet Neurologie

Akuter Schwindel: Wann lohnt sich eine MRT?

28.04.2024 Schwindel Nachrichten

Akuter Schwindel stellt oft eine diagnostische Herausforderung dar. Wie nützlich dabei eine MRT ist, hat eine Studie aus Finnland untersucht. Immerhin einer von sechs Patienten wurde mit akutem ischämischem Schlaganfall diagnostiziert.

Niedriger diastolischer Blutdruck erhöht Risiko für schwere kardiovaskuläre Komplikationen

25.04.2024 Hypotonie Nachrichten

Wenn unter einer medikamentösen Hochdrucktherapie der diastolische Blutdruck in den Keller geht, steigt das Risiko für schwere kardiovaskuläre Ereignisse: Darauf deutet eine Sekundäranalyse der SPRINT-Studie hin.

Frühe Alzheimertherapie lohnt sich

25.04.2024 AAN-Jahrestagung 2024 Nachrichten

Ist die Tau-Last noch gering, scheint der Vorteil von Lecanemab besonders groß zu sein. Und beginnen Erkrankte verzögert mit der Behandlung, erreichen sie nicht mehr die kognitive Leistung wie bei einem früheren Start. Darauf deuten neue Analysen der Phase-3-Studie Clarity AD.

Viel Bewegung in der Parkinsonforschung

25.04.2024 Parkinson-Krankheit Nachrichten

Neue arznei- und zellbasierte Ansätze, Frühdiagnose mit Bewegungssensoren, Rückenmarkstimulation gegen Gehblockaden – in der Parkinsonforschung tut sich einiges. Auf dem Deutschen Parkinsonkongress ging es auch viel um technische Innovationen.

Update Neurologie

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.

Newsletter bestellen

Springer Medizin

Abstract

Bitte loggen Sie sich ein, um Zugang zu diesem Inhalt zu erhalten

Weitere Artikel der Ausgabe 3/2015

Learning-Stage-Dependent Plasticity of Temporal Coherence in the Auditory Cortex of Rats

Temporal Evolution of Neural Activity Underlying Auditory Discrimination of Frequency Increase and Decrease

Source Analysis of Event-Related Potentials During Pitch Discrimination and Pitch Memory Tasks

Hemispheric Asymmetry of Auditory Mismatch Negativity Elicited by Spectral and Temporal Deviants: A Magnetoencephalographic Study

How Does Experience Modulate Auditory Spatial Processing in Individuals with Blindness?