How Sound Shapes What We See

And all this time you thought you saw with your eyes. A mathematical physicist explains his research into how sound defines the world.

Question: Does sound influence what we see?
Marcelo Magnasco: Yes indeed so in fact we believe that we attribute internally to vision most of our understanding of the space around us but a lot of it happens because of an auditory construction is linked to the vision reconstruction and it this process begins as early as a newborn who will turn their head towards the sound of their mother's voice looking for her face. Now in experiments in which you present a subject in darkness with a flash and a click of sound coming from the same place and you ask them okay please point to the place where it happened, if you move away this auditory source of the click and the flash but they still happen simultaneously but in a different place the person still perceives them as being a single source as being a single object and they will point to a point in between that the flash and the sound of the click which is roughly one-third of the you know towards the light and two-thirds towards the click meaning that internally the brain is using both the evidence from hearing as well as the evidence from vision to try to reconstruct the location of that particular event and its roughly waiting vision twice as much as hearing but this is you know relatively surprising. We would think that actually we see the precise place where things happen and we sort of hear you know something very broad but it is not so. Our hearing acutely helps us to have a very detailed imagery of the world and again coming back to my previous point this is surprising because the amount of primary sensory receptors that are used for hearing is so much smaller than the ones that are used for vision that there is a substantial amount of information processing that the nervous system divulge to coerce this information out of the auditory stream.

Question: Does sound influence what we see?
Marcelo Magnasco: Yes indeed so in fact we believe that we attribute internally to vision most of our understanding of the space around us but a lot of it happens because of an auditory construction is linked to the vision reconstruction and it this process begins as early as a newborn who will turn their head towards the sound of their mother's voice looking for her face. Now in experiments in which you present a subject in darkness with a flash and a click of sound coming from the same place and you ask them okay please point to the place where it happened, if you move away this auditory source of the click and the flash but they still happen simultaneously but in a different place the person still perceives them as being a single source as being a single object and they will point to a point in between that the flash and the sound of the click which is roughly one-third of the you know towards the light and two-thirds towards the click meaning that internally the brain is using both the evidence from hearing as well as the evidence from vision to try to reconstruct the location of that particular event and its roughly waiting vision twice as much as hearing but this is you know relatively surprising. We would think that actually we see the precise place where things happen and we sort of hear you know something very broad but it is not so. Our hearing acutely helps us to have a very detailed imagery of the world and again coming back to my previous point this is surprising because the amount of primary sensory receptors that are used for hearing is so much smaller than the ones that are used for vision that there is a substantial amount of information processing that the nervous system divulge to coerce this information out of the auditory stream.