Breakthrough device reads brainwaves and turns thoughts into speech

A new device that ‘reads’ a person’s mind can turn their thoughts into speech.

A team of engineers from the University of California invented a breakthrough brain-computer interface (BCI) system with electrodes that get adhered to a person’s scalp to measure brain activity and brainwaves.

The brainwaves are analyzed and then converted into audible speech by a computer, which then translates them to spoken words read aloud by AI. 

The researchers believe the new technology could restore paralyzed people’s ability to communicate by converting the brain activity from the motor cortex into audible speech.

The motor cortex is instrumental in controlling speech, and signals are generated even when a person has lost the ability to speak.

Researchers employed advanced AI models to capture brain signals in that area and convert them to sound in about one second, allowing continuous speech output without delays.

They tested the BCI on a woman named Ann, who has severe paralysis and cannot speak. She had participated in a previous study by the same team, but their system at that time resulted in a lag-time of eight seconds.

Kaylo Littlejohn, a Ph.D. student at UC Berkeley’s Department of Electrical Engineering and co-leader of the study, said: ‘We wanted to see if we could generalize to the unseen words and really decode Ann’s patterns of speaking.

‘We found that our model does this well, which shows that it is indeed learning the building blocks of sound or voice.’

Ann, who suffered a stroke in 2005 that cut off blood flow to her brainstem and paralyzed her, told researchers the device helped her feel more in control of her communication and instantly made her feel more connected to her body. 

Technology that decodes brain waves into spoken sentences is still in its infancy. 

In previous studies, it has been shown to work in a limited capacity, decoding just a handful of words but not phrases or full sentences.

But the team from California believes the new proof-of-concept study, published in the journal Nature Neuroscience, will help them ‘make advances at every level.’

Several different areas of the brain are responsible for speech, including the motor cortex, where words have unique ‘fingerprints.’

In this region, specific brainwave signatures are produced for each sound that helps identify different words, such as ‘hello’ and ‘goodbye’. 

When a person first attempts to speak, the brain sends signals to the lips, the tongue, and the vocal chords, and changes breathing patterns in order to speak. 

The researchers trained their ‘naturalistic speech synthesizer’ AI using Ann’s brainwaves to analyze and interpret them and turn those into spoken word. 

They placed electrodes on Ann’s skull and collected brain wave activity in her motor cortex as she tried to say simple phrases, like ‘Hey, how you?’

As Ann formed sentences in her head, and, even though she did not have the ability to speak, her brain fired commands for speech that were picked up by the electrodes. 

Researchers divided different signals into small time segments representing different parts of the sentence.

They incorporated audio of Ann’s voice from before she became paralyzed, and AI used a text-to-speech model to generate simulated audio in her natural voice.

Ann was shown text prompts, such as, ‘Hello, how are you?’ and mentally rehearsed speaking them—activating her motor cortex as if articulating the words, even though no sound was produced. 

AI slowly learned her speech patterns. She could soon speak words she had not been trained to visualize, such as Alpha and Bravo. 

The program also began recognizing words she didn’t see in her mind’s eye, filling in any gaps to form complete sentences.

Dr Gopala Anumanchipalli, an electrical engineer at UC Berkeley and co-leader of the study, said: ‘We can see relative to that intent signal, within 1 second, we are getting the first sound out.

‘And the device can continuously decode speech, so Ann can keep speaking without interruption.’

The program was also highly accurate, constituting a significant breakthrough in and of itself, according to Dr Littlejohn: ‘Previously, it was not known if intelligible speech could be streamed from the brain in real time.’

BCI technology has seen a burst of interest among scientists and tech giants.

In 2023, a team of engineers at Brown University’s BrainGate consortium successfully implanted sensors in the cerebral cortex of Pat Bennett, who has ALS.

Over the course of 25 training sessions, an AI algorithm decoded electrical signals from Ms. Bennett’s brain, learning to identify phonemes — the essential speech sounds like ‘sh’ and ‘th’ — based on neural activity patterns. 

The decoded brain waves were then fed into a language model, which assembled them into words and displayed her intended speech on a screen.

When vocabulary was limited to 50 words, the error rate was about nine percent, but it rose to 23 percent when the vocabulary expanded to 125,000 words, encompassing roughly every word a person would want to say, the researchers concluded.

The results of the study did not give an exact word count when training concluded, but researchers now know that machine-learning tools are capable of recognizing thousands of words.  

The results are far from perfect, but they believe their findings marked a significant stepping stone toward perfecting brain wave-to-speech systems.

Meanwhile, Elon Musk’s Neuralink was implanted in 29-year-old Noland Arbaugh’s head in January 2024, making him the first human participant in Neuralink’s clinical trial.

Arbaugh suffered severe brain trauma in 2016 that left him unable to move his body from body from the shoulders down. 

He was chosen years later to participate in the clinical trial of the device, allows for direct communication between the brain and external devices, like a computer or smartphone.

The Neuralink chip, implanted in Arbaugh’s brain, is connected to over 1,000 electrodes placed in the motor cortex.

When neurons fire, signaling intentions like hand movement, the electrodes capture these signals. The data is wirelessly transmitted to an application, allowing Arbaugh to control devices with his thoughts.

Arbaugh compares using Neuralink to calibrating a computer cursor—moving it left or right based on cues, and the system learns his intentions over time.

After five months with the device, he finds life has improved, particularly with texting, where he can now send messages in seconds. 

He uses a virtual keyboard and custom dictation tool, and also plays chess and Mario Kart using the same cursor technology.

The results from the research team at the University of California marks a breakthrough that it expects will move researchers nearer to creating natural speech with BCI devices and paves the way for further advancements.

Dr Littlejohn said: ‘That’s ongoing work, to try to see how well we can actually decode these paralinguistic features from brain activity.

‘This is a longstanding problem even in classical audio synthesis fields and would bridge the gap to full and complete naturalism.’