Are Two Senses Better Than One? | Psychology Today Canada

We tend to think of our five senses (vision, audition, smell, taste, and touch) as separate processes occurring independently within the brain and body. But, ever since the landmark studies on multisensory integration conducted in the 1990s, researchers have begun to appreciate the complex interactions between our different sensory modalities.

Effects of Sounds on Visual Perception

In a famous 1997 study by Robert Sekuler, participants judged whether two dots approaching each other on a computer screen appeared to cross paths or bounce off each other. Although the visual stimulus was always ambiguous and could be interpreted either way, presenting a brief auditory click at the moment the two dots’ paths intersected led most observers to judge the dots as bouncing off each other. In another classic study from 2000, Ladan Shams and colleagues found that presenting multiple auditory beeps in quick succession led most observers to misjudge the number of visual flashes that were shown; a single flash presented along with two beeps was often misperceived as two flashes.

In 2015, Hannah Goldberg and colleagues examined whether sounds can affect performance in a visual modulation rate task. In this type of task, participants are asked to judge how quickly a visual stimulus (in this case, a fish) changes colors between bright and dark. The color modulation rates could be 5 Hz (five times per second; the “slow rate”) or 6 Hz (the “fast rate”). Simultaneously, participants heard a task-irrelevant sound that modulated in volume between loud and soft, at a rate of either 5 Hz or 6 Hz. Importantly, in some trials, the visual and auditory modulation rates were matched (e.g., both sounds and visuals modulated at 5 Hz) while in other trials they were mismatched (e.g., the sound modulated at 5 Hz while the visuals modulated at 6 Hz).

Goldberg and colleagues found that participants were much better at judging the visual modulation rates when the auditory rate matched the visual rate. This result has been replicated several times and regarded as evidence of a processing advantage for multisensory stimuli. However, without a proper control condition, it is difficult to say whether these results truly reflect an advantage in multisensory processing when the senses are matched, or a disadvantage in multisensory processing when the senses are mismatched.

Improvement or Impairment?

In a recent study published in the September 2022 issue of Perception, Jiayue Tai, Jack Forrester, and Robert Sekuler addressed this question by presenting participants not only with matched and mismatched audio-visual conditions but also with a silent control condition in which the visually modulating stimulus was presented with no sound. Their results confirmed that matching the auditory and visual modulation rates improved performance by about 7 percent compared to the silent condition.

However, their results also showed that mismatched audio-visual conditions had an even larger impact on performance. Accuracy dropped by about 12 percent when the auditory and visual modulation rates were mismatched, compared to trials in which visuals were presented in silence. Therefore, the impairment due to an audio-visual mismatch was substantially greater than the improvement due to an audio-visual match.

Multitasking in Everyday Life

Although it is certainly true that congruent sounds can improve performance in some visual tasks, it is important to highlight that incongruent sounds can impair visual performance, sometimes even more so. Thus, two senses are not always better than one.

We may be inclined to believe that we can perform visual tasks (e.g., driving) and auditory tasks (e.g., listening to a podcast) independently of each other. This research shows that certain types of cross-sensory conflict can actually impair performance. More research is still needed to understand how visual–auditory conflicts might influence performance in everyday tasks.