Tag Archives: sound

Cooperation of sight and sound

As a child you were probably taught to tell how far away lightening was. When there is a flash, you count with a particular rhythm until you hear the thunder and that is how many miles the lightening is away from you. Parents are not going to stop teaching this because it is something for a nervous child to do in a thunder storm and it convinces them that they are usually a safe distance from danger. But it only works for distant events.

Events that are close by are synchronized by the brain and consciously we collapse the vision and hearing clues both for time and space to make a single event. We are not conscious of a difference in the timing or of any slight difference in the placing of the event. A particular region of the brain does this aligning - “the superior colliculus, a midbrain region that functions imperatively for integrating auditory and visual signals for attending to and localizing audiovisual stimuli”. But if the difference is too large between the vision and hearing, the collapse into a single event does not happen.

However, we know that, even though it is not consciously experienced, the information about small differences in sound arrival can be used by blind humans to echo-locate by making continuous little clicking noises. Could it be that the discrepancy between sound and sight could be used in other ways? A recent paper (Jaekl P, Seidlitz J, Harris LR, Tadin D (2015) Audiovisual Delay as a Novel Cue to Visual Distance. PLoS ONE 10(10): e0141125. doi:10.1371/journal.pone.0141125) studies the effect of sound delays on the perception of distance. Like the lightening calculation, but it is done unconsciously.

Here is the abstract:

For audiovisual sensory events, sound arrives with a delay relative to light that increases with event distance. It is unknown, however, whether humans can use these ubiquitous sound delays as an information source for distance computation. Here, we tested the hypothesis that audiovisual delays can both bias and improve human perceptual distance discrimination, such that visual stimuli paired with auditory delays are perceived as more distant and are thereby an ordinal distance cue. In two experiments, participants judged the relative distance of two repetitively displayed three-dimensional dot clusters, both presented with sounds of varying delays. In the first experiment, dot clusters presented with a sound delay were judged to be more distant than dot clusters paired with equivalent sound leads. In the second experiment, we confirmed that the presence of a sound delay was sufficient to cause stimuli to appear as more distant. Additionally, we found that ecologically congruent pairing of more distant events with a sound delay resulted in an increase in the precision of distance judgments. A control experiment determined that the sound delay duration influencing these distance judgments was not detectable, thereby eliminating decision-level influence. In sum, we present evidence that audiovisual delays can be an ordinal cue to visual distance.

The power of sound

Using faked sounds, subjects experienced the illusion that their hand was becoming more like marble according to a recent paper (citation below). We should not be as surprised by this illusion as we are. We assume that our perception of the substance of our bodies is not going to change. But in this illusion it does. The subjects watched a small hammer hit their hand. The experimenters slowly changed the sound that the subject heard from the real sound to the sound of a hammer hitting stone. The subjects had the illusion, the marble-hand illusion, that their hand had become dense, hard, stiff and heavy, like stone. Our surprise at this is itself somewhat surprising. Illusions are like that though – in the famous example, we know, we may even have measured, that the lines are the same length but the illusion is still there and the one line still looks longer than the other. At the level perception, an illusion is not touched by our knowledge that it is wrong. This should not be a surprise.



Why do we even need to perceive the nature of our own flesh to begin with. It does not change. Oh, but it does, sort of. Think of someone with a cast, or wearing armor for some pageant, or thick winter clothing. We change our body-scheme in many ways when we use tools, wear cloths, are injured and so on. The brain has to examine the nature of our bodies and quickly change the body-scheme if necessary. Fiurther, the process of examining the material nature of the world would include our own bodies; to ourselves would require extra work. So for those two reasons alone, it should not surprise us that our body-scheme can be updated by perception.



But why should sound create so powerful an illusion? We understand the world from our senses. The senses are limited by what they are sensitive to. The eyes are sensitive to the little window of light that we call the visual spectrum and that is processed into colour and brightness and sheen of surfaces. From this we can identify many aspects of objects. Touch and smell give other pieces of information. But only sound really seems to tells us about the inside of an object and the material it is made from. Because matter vibrates when it is stroked, hit, stressed, broken, or bounced, each type of material has very unique sounds. It can tell us about the hidden inside of an object which is not easy to sense (weight, heat conductivity and other perceptions can give interior information but it is spotty). It is likely that hearing takes precedence over other senses when it come to the material of an object. If I see a slat of wood and when I drop it, I hear a metallic sound, I know that it is a metal strip painted to resemble wood. It is not seen as a piece of wood made to sound like metal. It is not easy to fake the sound of a material. That is exactly what is done in this experiment, the sound is faked. People say seeing is believing but when it comes to materials – hearing is believing.



So the illusion should not surprise us: illusions are not reversed by knowledge; the brain does update the body-scheme; and, sound is a powerful indicator of the nature of material. “When exposed to multisensory signals that correlate in time and space, but provide incongruent cues to body material, the brain can either keep those signals segregated, or else integrate them and resolve the incongruence by updating the perception of body material. The MHI (marble-hand illusion) demonstrates that the brain integrates correlated signals, and quickly updates the body schema, which consistently results in a vivid bodily illusion. ”



Here is the abstract:


Our body is made of flesh and bones. We know it, and in our daily lives all the senses constantly provide converging information about this simple, factual truth. But is this always the case? Here we report a surprising bodily illusion demonstrating that humans rapidly update their assumptions about the material qualities of their body, based on their recent multisensory perceptual experience. To induce a misperception of the material properties of the hand, we repeatedly gently hit participants’ hand with a small hammer, while progressively replacing the natural sound of the hammer against the skin with the sound of a hammer hitting a piece of marble. After five minutes, the hand started feeling stiffer, heavier, harder, less sensitive, unnatural, and showed enhanced Galvanic skin response (GSR) to threatening stimuli. Notably, such a change in skin conductivity positively correlated with changes in perceived hand stiffness. Conversely, when hammer hits and impact sounds were temporally uncorrelated, participants did not spontaneously report any changes in the perceived properties of the hand, nor did they show any modulation in GSR. In two further experiments, we ruled out that mere audio- tactile synchrony is the causal factor triggering the illusion, further demonstrating the key role of material information conveyed by impact sounds in modulating the perceived material properties of the hand. This novel bodily illusion, the ‘Marble-Hand Illusion’, demonstrates that the perceived material of our body, surely the most stable attribute of our bodily self, can be quickly updated through multisensory integration.


Senna, I., Maravita, A., Bolognini, N., & Parise, C. (2014). The Marble-Hand Illusion PLoS ONE, 9 (3) DOI: 10.1371/journal.pone.0091688

Editor’s Selection IconThis post was chosen as an Editor's Selection for ResearchBlogging.org