Neuroscientists Translate Brain Activity into Birdsong

Researchers at UC San Diego have re-created a bird's song by reading the song from the bird’s brain. This provides proof of concept that complex natural behaviors can be directly synthesized from ongoing brain activity. The research is applicable to the development of general purpose brain-machine interfaces.

A study is published in Current Biology. It describes how the researchers were able to reproduce the songbird's complex vocalizations down to the pitch, volume, and timbre of the original.

The study is a first step towards developing vocal prostheses for humans who have lost the ability to speak.

"The current state of the art in communication prosthetics is implantable devices that allow you to generate textual output, writing up to 20 words per minute," says lead researcher Timothy Gentner in a press release issued by UC San Diego. "Now imagine a vocal prosthesis that enables you to communicate naturally with speech, saying out loud what you're thinking nearly as you're thinking it. That is our ultimate goal, and it is the next frontier in functional recovery."

The researchers implanted silicon electrodes in male adult zebra finches. And they monitored the birds' neural activity while singing. Specifically, they recorded the electrical activity of multiple populations of neurons in the part of the birds’ brain that ultimately controls the muscles responsible for singing.

Then the researchers used a machine learning system to create computer-generated copies of actual zebra finch songs based on the birds' neural activity. Instead of directly translating patterns of neural activity into patterns of sounds, the researchers translated the neural patterns into the physical changes in pressure and tension that happen in the vocal organs of singing birds.

The researchers now plan to demonstrate that their system can reconstruct birdsong from neural activity in real time. When singing, birds make adjustments based on what they just heard themselves singing.

This is also the case of humans when they speak. And, therefore, speech prostheses must be able to process human brain feedback in real time. A successful vocal prosthesis, explains Gentner, will ultimately need to work on a timescale that is similarly fast and also intricate enough to accommodate the entire feedback loop, including making adjustments for errors.

The research is a cross-collaborative effort between engineers and neuroscientists at UC San Diego. "With our collaboration," concludes Gentner, "we are leveraging 40 years of research in birds to build a speech prosthesis for humans - a device that would not simply convert a person's brain signals into a rudimentary set of whole words but give them the ability to make any sound, and so any word, they can imagine, freeing them to communicate whatever they wish."