Research reportDistinctions between manipulation and function knowledge of objects: evidence from functional magnetic resonance imaging
Introduction
A growing body of evidence indicates that representations of man-made objects (hereafter, artifacts) can be distinguished from representations of living things (see Refs. [11], [16], [48] for recent reviews). The reasons for these dissociations remain unclear. One prominent account suggests that animals and artifacts differ in the underlying format of their representations. Based on the observation that exemplars of a given artifact—different telephones, for example—may differ in their size, shape, and color, yet still share a common function, it has been suggested that functional properties play a central role in the distinctions among artifacts, whereas visual properties and other sensory information distinguish living things [52], [53]; see also [18], [20], [47]. This account, however, does not explain several findings, including preservation of functional feature knowledge in patients with artifact impairments (e.g., Ref. [28]), sparing of knowledge of large artifacts with clear functions (buildings, vehicles) in patients with deficits for smaller manipulable artifacts [52], or co-occurrence of deficits in body-part knowledge and artifact knowledge [41], [52] (and see Ref. [11]). An alternative account having the potential to explain these patterns is the “sensory-motor” (SM) account. The SM account is perhaps most clearly articulated in a distributed model of the semantic system proposed by Allport [3] (see also Refs. [52], [53]). This model proposes that living and non-living things alike are represented in terms of auto-associated auditory, olfactory, visual and tactile-motor attribute domains as a function of the degree to which these forms of information were activated when the knowledge was acquired. On some related accounts, the distributed semantic system includes verbal-propositional elements as well [7]. On this model, the observed dissociations between living things and artifacts (and between large and small artifacts) might reflect the degree to which the objects under consideration have been held, moved and manipulated.
Consistent with this proposal, there is growing evidence that function and manipulation information are dissociable forms of object knowledge associated with different lesion loci, and that manipulation knowledge may be closely tied to overall knowledge of artifacts. Deficits in tool function knowledge in the face of relatively intact manipulation knowledge have occurred in patients with temporal lobe lesions due to semantic dementia or herpes encephalitis [7], [30], [45]. Deficits in manipulation but not function knowledge have occurred in apraxic patients with left frontoparietal lesions affecting sensory-motor association cortex [8]. In a recent group study [6], we tested left hemisphere stroke patients with measures of limb apraxia (gesture pantomime), tests of tool, animal and body part knowledge and knowledge of the function and manipulation of manipulable artifacts. The function and manipulation knowledge test required a judgment of synonymy about a triplet of items. In the function condition, for example, subjects were shown a record player, radio and telephone; only the first two objects are alike in function and all are held and used differently. In the manipulation condition, subjects were shown, for instance, a typewriter, piano and stove; the first two are both ‘tapped’ with the fingertips, whereas a stove is not, and all serve a different function. There were picture and word versions of the test. Patients with relative deficits in tool knowledge tended to be apraxic, impaired in manipulation (but not function) knowledge with both picture and word stimuli, deficient in body part knowledge and to have left inferior parietal lesions. The coherence of this pattern supports the contention that manipulation knowledge subserved by sensory-motor association cortices may play a prominent role in the representations of tools (but see Ref. [31] for objections).
Consistent with this possibility, cortical activations reported in neuroimaging studies of artifact or tool representations include the left posterior parietal lobe and/or intraparietal sulcus (IPS), left ventral premotor cortex (PMv) of the frontal lobe and left posterior middle temporal gyrus (PMTG) [12], [21], [24], [32] (see also Refs. [16], [27] for review). All three regions (albeit not necessarily lateralized to the left) have been associated with action representations in prior neurophysiological studies of the monkey [4], [33], [39], [43].
Several recent imaging studies have directly tested the hypothesis that activations associated with manipulable artifacts reflect action-related processes. Gerlach et al. [22] demonstrated using fMRI that categorizing pictures as natural or man-made activated PMv for manipulable objects (fruit/vegetables and clothing) more than animals and non-manipulable objects. Phillips et al. [37] showed PET activations in the left PMTG with action decisions about manipulable objects. In keeping with the findings of Gerlach et al. [22], this pattern was observed even for small manipulable living things (i.e., fruit), suggesting that the activation was not attributable to an effect of semantic category per se. In a recent PET study by Kellenbach, Brett and Patterson [27], participants were shown pictures of manipulable and non-manipulable objects, and asked to make judgments about their function (e.g., is this used to attach objects together?) and, for manipulable objects, the manner in which they are manipulated (e.g., is this used with a swinging motion?). There were common activations in left PMv and left PMTG in all object conditions relative to a control condition, but these regions were also relatively more activated by manipulable objects (irrespective of task) than non-manipulable objects. This is consistent with the possibility that these regions are involved in processing action-related information about manipulable objects. Additionally, there was highly significant activation of left posterior parietal lobe in the region of the IPS when subjects responded to questions about actions associated with manipulable objects. The specificity of these activations led the investigators to suggest that this region may be selectively involved with explicit retrieval of action knowledge.
A question that is not addressed by the study of Kellenbach et al. [27] is whether the results obtained are specific to pictures of objects. This is critical given behavioral evidence that pictures may provide more rapid access to action knowledge than do words [40], [42], and functional imaging evidence that pictures and words may be processed differently, particularly in the domains of function and action knowledge. Using PET, Vandenberghe et al. [51] identified a large distributed system associated with processing the functional associates of both pictures and words, along with unique activations in left posterior inferior temporal sulcus for picture stimuli and left superior temporal sulcus, left anterior middle temporal gyrus and left inferior frontal gyrus for words. In a recent PET study, Phillips et al. [37] also identified a large left inferior frontal and anterior temporal region commonly activated for action decisions (e.g., “do you use this with a twisting motion?”) for both pictures and words, as well as left inferior frontal, left anterior fusiform, and left anterior temporal regions that were more activated for such decisions with words than pictures. They concluded that the reduction in semantic activation in the picture condition resulted from facilitation of direct and automatic access to action by the visual-motor affordances present in objects.
On the basis of these considerations, we assessed the neural correlates of function and manipulation knowledge using both picture and word formats with a variant of the synonymy judgment task we had developed previously [6]. The inclusion of word stimuli permits us to address the critical question of whether activations observed with pictures are automatically driven by object affordances (i.e., shape and size information compatible with, and linked to, particular motor actions, e.g., as the handle of a hammer is compatible with a prehensile hand posture [23]). If activations are entirely affordance-related, that is, automatically driven by visual shape and size information, then they should not be observed with word stimuli. Subjects also performed two baseline tasks designed to parcel out visual-perceptual and motor response processes. We aimed to address the following general hypotheses: (1) function and manipulation knowledge of objects are associated with partially overlapping, but partially distinct patterns of neural activation; (2) manipulation representations are not solely triggered automatically by object affordances, but in part reflect object-specific, abstract, modality-independent processes; and (3) the brain is more responsive to the actions associated with objects than to their functions. More specifically, the study was designed to test several predictions. First, consistent with the previous literature, we expected overlapping regions of activation in dorsolateral and ventrolateral prefrontal and premotor cortex, MPTG, and IPL when subjects answered questions about the manipulation (M) and function (F) of manipulable artifacts. Second, we expected that F and M conditions would also be associated with distinct patterns of activation. Based on previous observations, we expected that the manipulation (M) condition would disproportionately activate the left IPL; an expectation from the neuropsychological literature was that the function (F) condition would give rise to relative activations in the temporal lobe. Third, we expected to see many of the same brain regions activated with word and picture stimuli. Finally, given the results of Phillips et al. [37] and Vandenberghe et al. [51], we also expected that we might find specific differences between pictures and words in the function condition (left posterior inferior temporal sulcus, left superior temporal sulcus), manipulation condition (left anterior fusiform gyrus) or both (left inferior frontal, left anterior middle temporal gyri).
Section snippets
Participants
Twenty-two healthy subjects were recruited from the University of Pennsylvania community and scanned. Three subjects were not included in data analysis because of incomplete behavioral data, two because of incomplete fMRI data and two because their accuracy scores were >2 S.D. below the average, leaving 15 participants (8 females, 7 males). Subjects were aged 21–33 (mean, 23.1), spoke English as a first language, were right handed and had normal or corrected to normal vision. All gave informed
Reaction time
Summary data are presented in Table 1. As there was no principled reason to think that the baseline conditions for the function (F) and manipulation (M) conditions were different (i.e., both consisted of scrambled, unrecognizable pictures or words), baseline data from the F and M conditions were collapsed into two conditions: (B)aseline pictures and B words. An analysis of variance (ANOVA) was performed on the reaction time (RT) data. This analysis, which included the four primary conditions of
Discussion
In this study, we addressed four major predictions generated by the SM theory regarding the semantic representations of manipulable artifacts. First, consistent with the previous literature, we expected overlapping regions of activation in dorsolateral and ventrolateral prefrontal and premotor cortex, MPTG and IPL in F and M conditions. Second, we expected that F and M conditions would also be associated with distinct patterns of activation. Specifically, we expected that the manipulation (M)
Acknowledgments
Supported by NIH grants NS36387, NS37920, DC02754, NS045839 and HD39621.
References (55)
- et al.
Empirical analysis of BOLD fMRI statistics: II. Spatially smoothed data collected under the null-hypothesis and experimental conditions
Neuroimage
(1997) - et al.
The variability of human BOLD hemodynamic responses
Neuroimage
(1998) - et al.
Knowledge of object manipulation and object function: dissociations in apraxic and non-apraxic subjects
Brain and Language
(2002) - et al.
Cognitive representations of hand posture in ideomotor apraxia
Neuropsychologia
(2003) - et al.
Representation of manipulable man-made objects in the dorsal stream.
Neuroimage
(2000) - et al.
Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI
Neuropsychologia
(2002) - et al.
Visuomotor neurons: ambiguity of the discharge or ‘motor’ perception?
International Journal of Psychophysiology
(2000) - et al.
Categorization and category effects in normal object recognition: a PET study
Neuropsychologia
(2000) - et al.
Premotor cortex activation during observation and naming of familiar tools
Neuroimage
(1997) - et al.
Differences in word associations to pictures and words
Neuropsychologia
(2003)
Neuroimaging studies of semantic memory: inferring “how” from “where”
Neuropsychologia
Dissociable brain regions process object meaning and object structure during picture naming
Neuropsychologia
Distributed memory, modular subsystems and dysphagia
Coordinate Transformations and Motor Planning in Posterior Parietal Cortex
Gesturing and naming: the use of functional knowledge in object identification
Psychological Science
The role of semantic memory in object use
Cognitive Neuropsychology
Function and manipulation tool knowledge in apraxia: knowing ‘what for’ but not ‘how’
Neurocase
What are the facts of semantic category-specific deficits? A critical review of the clinical evidence
Cognitive Neuropsychology
3D Model-Based Segmentation of Individual Brain Structures From Magnetic Resonance Imaging Data
Control of goal-directed and stimulus-driven attention in the brain
Nature Reviews Neuroscience
Visuomotor priming
Visual Cognition
A computational model of semantic memory impairment: modality specificity and emergent category specificity
Journal of Experimental Psychology, General
Spatial registration and normalization of images
Human Brain Mapping
Category-specific semantic loss in dementia of Alzheimer's type
Brain
When action turns into words. Activation of motor-based knowledge during categorization of manipulable objects
Journal of Cognitive Neuroscience
The theory of affordances
Cited by (227)
Using 360-degree video to explore teachers' professional noticing
2022, Computers and EducationFluency test generation and errors in focal frontal and posterior lesions
2021, NeuropsychologiaVisuomotor control in the healthy and damaged brain
2021, Encyclopedia of Behavioral Neuroscience: Second Edition