How what we see and hear feed together to help us understand emotions

Scientific American · Tyler Santora · 2026-10-19 17:00 UTC

The emotions themselves may be more important than the senses we use to perceive them

For decades, scientists thought that in order to read other people’s emotions, the brain processed sensory information from faces and voices in separate areas. But new research shows otherwise—seeing an emotive face lights up the brain’s voice-processing centers, and vice versa.

In everyday life, “we don’t only see a face that is sad,” explains Stefania Mattioni , the study’s lead author and a cognitive neuroscience postdoctoral researcher at Ghent University in Belgium. “We normally also hear the sound of the person crying, for instance. So, the brain is actually structured to interact with this kind of multisensory stimulation.”

For the study, Mattioni’s team played clips of actors portraying different emotions, such as happiness, disgust, and fear, to 24 participants who were hooked up to a magnetic resonance imaging (MRI) machine. Some of the clips showed both the actors’ faces and voices; others only included video or audio.

Read the full story at Scientific American · All coverage of this story