Artificial Intelligence Turns Brain Activity into Speech

Researchers at UC San Francisco have developed a speech neuroprosthesis. It has enabled a man with severe paralysis to communicate in sentences. It does this by translating signals from his brain to the vocal tract directly into words that appear as text on a screen.

"To our knowledge, this is the first successful demonstration of direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak," says neurosurgeon Edward Chang in a press release issued by UC San Francisco. "It shows strong promise to restore communication by tapping into the brain's natural speech machinery."

Previous speech neuroprosthetics work has demonstrated the possibility to capture the brain signals of paralyzed patients. And it has been possible to translate the signals to allow the patients to slowly type out letters one-by-one in text.

But now Chang and his team have found ways to capture and translate the brain signals intended to control muscles of the vocal system for speaking words, rather than the signals to move the arm or hand to enable typing. This new approach promises more rapid and organic communication.

"With speech, we normally communicate information at a very high rate, up to 150 or 200 words per minute," explains Chang. "Going straight to words, as we're doing here, has great advantages because it's closer to how we normally speak."

A study is published in New England Journal of Medicine. It reports that the researchers implanted a high-density electrode array over a paralyzed patient’s speech motor cortex. And they used artificial intelligence, specifically custom neural networks, to translate the patterns of recorded neural activity into specific intended words. The system was able to decode words from brain activity and generate synthesized speech at a rate of up to 18 words per minute, with up to 93% accuracy (75% median).

The patient was a man in his late 30s who suffered a devastating brainstem stroke more than 15 years ago. He worked with the researchers to create a 50-word vocabulary that Chang's team could recognize from brain activity using advanced computer algorithms. The vocabulary includes words such as "water," "family," and "good.” It was sufficient to create hundreds of sentences relevant to the patient’s daily life.

"This trial is just the beginning,” said Chang as reported by CNN. “This is the very first participant that's been in the trial, and the first set of experiments that were part of this trial to show that this is possible."

The next steps are to see “if this is better, worse or the same in more people,” said Chang as reported by Scientific American. The researchers will also use a larger vocabulary to train the machine that decodes the brain’s output. The vocabulary has already expanded beyond the 50 words reported in this study, he says, and “it’s exciting to see things grow in that kind of way.”