Link between image and sound

Babies link the sound of a word with the image of an object in their early learning of language and this is an important ability. How do they come to have this mechanism? Are there predispositions to making links between sounds and images?

Research by Asano and others (citation below) shows one type of link. They show that sound symbolism can be used by infants about to learn language (about 11 months) to match certain pseudo-words to drawings - “moma” to rounded shapes and “kipi” to sharply angled shapes. Sound symbolism is interesting but it need not be the first or most important link between auditory and visual information. It seems to me that a 11 month old child would associate barks with dogs, twitters with bird, honks and engine noises with cars, and so on. They even mimic sounds to identify an object. It is clear that objects are recognized by their feel, smell, and sound as well as by sight. The ability to derive meaning from sound is completely natural, as is deriving it from sight. What is important is not the linking of sound and sight with the same meaning/object – mammals without language have this ability.

What is important about sound symbolism is that it is arbitrary and abstract. We appear to be born with certain connections of phonemes and meanings ready to be used. These sorts of connections would be a great help to a child grasping the nature of language as opposed to natural sounds.

Here is the abstract: “A fundamental question in language development is how infants start to assign meaning to words. Here, using three Electroencephalogram (EEG)-based measures of brain activity, we establish that preverbal 11-month-old infants are sensitive to the non-arbitrary correspondences between language sounds and concepts, that is, to sound symbolism. In each trial, infant participants were presented with a visual stimulus (e.g., a round shape) fol lowed by a novel spoken word that either sound-symbolically matched (“moma”) or mis matched (“kipi”) the shape. Amplitude increase in the gamma band showed perceptual integration of visual and auditory stimuli in the match condition within 300 msec of word onset. Furthermore, phase synchronization between electrodes at around 400 msec revealed intensified large-scale, left-hemispheric communication between brain regions in the mismatch condition as compared to the match condition, indicating heightened processing effort when integration was more demanding. Finally, event-related brain potentials showed an increased adult-like N400 response - an index of semantic integration difficulty - in the mismatch as compared to the match condition. Together, these findings suggest that 11-month-old infants spontaneously map auditory language onto visual experience by recruiting a cross-modal perceptual processing system and a nascent semantic network within the first year of life.
ResearchBlogging.org

Asano, M., Imai, M., Kita, S., Kitajo, K., Okada, H., & Thierry, G. (2015). Sound symbolism scaffolds language development in preverbal infants Cortex, 63, 196-205 DOI: 10.1016/j.cortex.2014.08.025

2 thoughts on “Link between image and sound

  1. Lyndon

    Hi Janet,

    I am thrown by this a bit:

    “Here, using three Electroencephalogram (EEG)-based measures of brain activity, we establish that preverbal 11-month-old infants are sensitive to the non-arbitrary correspondences between language sounds and concepts, that is, to sound symbolism. In each trial, infant participants were presented with a visual stimulus (e.g., a round shape) fol lowed by a novel spoken word that either sound-symbolically matched (“moma”) or mis matched (“kipi”) the shape.”

    Why or how does “moma” sound-symbolically match a round shape? Is that just an intuitive given that I somehow miss? It makes sense to me that the softness of “moma” matches the smooth edges of the circle, but that feels like abstract post hoc analysis, and not something that would be given. I guess the question is: Is there really an abstract quality of softness that babies intuit from rounded shapes? And one that we would then carry over into our sounds? Outside of sounds mimicking other sounds, the inference from spatial or visual quality to sound quality seems stretched. I often feel that where we do make such judgments, it is from a reflected, abstract standpoint, perhaps one that we then attach emotional or connotational structures to.

    I find things like “that object is long,” therefore we expect a long word or long sound to match that object to be rather arbitrary and difficult to analyze. There may be something there, but it also seems like our ability to judge how a naked brain (a predisposed brain) will judge the qualities of an object, and then to infer the appropriate sound, to be a tall task fraught with given assumptions from our own world, from how we see things.

    I may buy something like a word similar to ‘moma’ was a sound set for face early in the species, say genetically somehow, and therefore babies respond to round objects with expectation of similar sounds. (I assume monkeys, birds, other animals have some common, genetically set calls for other objects (?)). But that is different than a symbol-sound quality matching a visual quality.

    Reply
    1. JKwasniak Post author

      Hi Lyndon, thanks for the comment. I find the following interesting. Does it answer any of your questions?
      This is from the Wikipedia entry on sound symbolism:
      In the 2003 BBC Reith Lectures, Vilayanur S. Ramachandran outlined his research into the links between brain structure and function. In the fourth lecture of the series he describes the phenomena of synesthesia in which people experience, for example, sounds in terms of colors, or sounds in terms of tastes. In one type of synesthesia, people see numbers, letters of the alphabet, or even musical notes as having a distinct color. Ramachandran proposes a model for how language might have evolved. The theory may explain how humans create metaphors and how sounds can be metaphors for images – why for example sounds can be described as “bright” or “dull”. In explaining how language might have evolved from cross activation of adjacent areas in the brain, Ramachandran notes four crucial factors, not all related to language, but which combined might have resulted in the emergence of language. Two of these four processes are of particular interest here.
      Synesthetic cross modal abstraction: i.e. we recognize properties that sounds and images have in common and abstract them to store them independently. The sounds and shapes of the objects have characteristics in common that can be abstracted; for example, a “sharp”, “cutting” quality of a word, and the shape it describes. Ramachandran calls this the ‘Bouba/kiki effect’, based on the results of an experiment with two abstract shapes, one blob-like and the other spiky, that asked people to relate the nonsense words bouba and kiki to them. The effect is real and observable, repeatable across linguistic groups, and evident even in the description of the experiment (with the bouba shape usually described using similar-sounding words like bulbous or blobby while the kiki shape is prickly or spiky).
      Built in preexisting cross activation. Ramachandran points out that areas of the brain which appear to be involved in the mix-ups in synesthesia are adjacent to each other physically, and that cross-wiring, or cross activation, could explain synesthesia and our ability to make metaphors. He notes that the areas that control the muscles around the mouth are also adjacent to the visual centers, and suggests that certain words appear to make our mouth imitate the thing we are describing. Examples of this might be words like “teeny weeny”, “diminutive” to describe small things; “large” or “enormous” to describe big things.
      More recently, research on ideasthesia indicated that Kiki and Bouba have an entire semantic-like network of associated cross-modal experiences.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *