Scientists at NYU Tandon have discovered how the human brain manages and balances visual and auditory signals in real time, thanks to a unique experiment.
A study published in the journal Nature Communications demonstrated that the frontal cortex acts as a kind of “traffic controller,” constantly deciding which sensory stream to prioritize at any given moment.
While most scientific experiments typically rely on simplified laboratory tests, this time neuroscientists took a completely different approach. They created an environment as close to real life as possible and used a movie screening to observe human subjects.
What Did the Study Reveal?
The study involved 19 patients with epilepsy who had temporary electrodes implanted in their brains for clinical monitoring purposes. During their hospital stay, the patients watched a 12-minute multilingual short film containing scenes in English, Greek, German, and French (with subtitles provided for foreign-language scenes).
Since the electrodes were placed directly on or within the cerebral cortex, researchers were able to record neuronal responses with millisecond precision, far exceeding the capabilities of standard functional magnetic resonance imaging (fMRI).
Data analysis revealed that the frontal cortex does not process sensory information uniformly. Instead, there is a clear internal structural division: ventral (lower) regions react much more strongly to auditory (audio) signals, while dorsal (upper) regions are more focused on visual information. This indicates that the frontal cortex possesses a highly organized map for managing different types of sensory streams, rather than serving as just a generic control center.
The Most Intriguing Part of the Study
The most fascinating part of the research involves the introduction of a language barrier. When the characters in the film spoke English (a language understood by the patients), the frontal cortex focused on processing auditory signals. However, the moment the characters switched to a foreign language, neural activity instantly shifted to the visual regions—the viewers’ brains automatically switched to reading subtitles and actively analyzing facial expressions and gestures.
The scientists also confirmed this dynamic brain response through a survey of online volunteers. They evaluated each scene of the movie, identifying moments where auditory or visual cues were critical to understanding the plot. The subjective ratings of the volunteers perfectly matched the neural data recorded from the patients’ brains: the brain shifts and reallocates sensory priorities “on the fly,” depending on which source is more informative.
So, What Do These Findings Give Us?
This discovery sheds light on a long-standing question in neuroscience: how the organism integrates multisensory information in a complex, changing environment. The study shows that the frontal cortex does not simply combine information post-factum; it actively determines which sensory stream to prioritize before the individual reaches conscious perception.
The research team notes that a deeper understanding of this mechanism will help develop new therapeutic approaches in the future for patients with speech disorders, autism, attention deficit disorders, or hearing loss. Furthermore, this adaptive capacity of the brain could serve as an inspiration for creating more flexible Artificial Intelligence (AI) systems that can correctly prioritize between audio and visual parameters based on context.
The scientists acknowledge a certain limitation of the study: the participants were patients with epilepsy, and the placement of the electrodes was determined by purely medical necessity rather than experimental design. Nevertheless, the precision of direct neural recording still provides a unique and rare opportunity to study the inner workings of the living human brain.
