Granted Patent B2

US 11,961,533 · App. 17/826,474 · Granted Apr 16, 2024

Inventors: Nima Mesgarani (New York, NY); Yi Luo (New York, NY); James O'Sullivan (New York, NY); Zhuo Chen (New York, NY)

Assignee: The Trustees of Columbia University in the City of New York

G10L25/30G10L17/26A61B5/12G10L21/0272G10L25/66

Abstract

Claims (62)

1. A method comprising:

obtaining, by a device, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

generating a combined sound spectrogram from the combined sound signal;

applying, by the device, speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources;

obtaining, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

selecting one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including:

generating an attended speaker spectrogram based on the neural signals for the person;

comparing the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms; and

transforming the selected one of the multiple resultant speaker spectrograms into an acoustic signal;

wherein applying the speech-separation processing to the combined sound spectrogram comprises applying neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including:

separating the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.

2. The method of claim 1 , wherein obtaining the neural signals for the person comprises:

obtaining one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp, or in-ear EEG recordings.

3. The method of claim 1 , further comprising:

processing the transformed selected one of the multiple resultant speaker spectrograms, including performing one or more of: amplifying the transformed selected one of the multiple resultant speaker spectrograms, or attenuating a transformed at least one non-selected speaker spectrogram from the multiple resultant speaker spectrograms.

4. The method of claim 1 , wherein obtaining the combined sound signal for the multiple sound sources comprises:

receiving the combined sound signal for the multiple sound sources at a single microphone coupled to the device.

5. The method of claim 1 , wherein applying the neural-network-based speech-separation processing to the combined sound spectrogram further comprises:

providing the combined sound spectrogram to a deep neural network (DNN) configured to identify individual sound sources from the combined sound spectrogram.

6. The method of claim 1 , wherein comparing the attended-speaker spectrogram to the derived multiple resultant speaker spectrograms comprises:

comparing the attended speaker spectrogram to the derived multiple resultant speaker spectrograms using normalized correlation analysis.

7. A system comprising:

at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:

generate a combined sound spectrogram from the combined sound signal;

apply speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources; and

select one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including to:

generate an attended speaker spectrogram based on the neural signals for the person;

compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms; and

transform the selected one of the multiple resultant speaker spectrograms into an acoustic signal;

wherein the controller configured to apply the speech-separation processing to the combined sound spectrogram is configured to apply neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including to:

separate the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.

8. A non-transitory computer readable media programmed with instructions, executable on a processor, to:

obtain, by a device comprising the processor, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

generate a combined sound spectrogram from the combined sound signal;

apply, by the device, speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources;

obtain, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

select one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including to:

generate an attended speaker spectrogram based on the neural signals for the person;

compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms; and

transform the selected one of the multiple resultant speaker spectrograms into an acoustic signal;

wherein the instructions to apply the speech-separation processing to the combined sound spectrogram comprise one or more instructions to apply neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including to:

separate the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.

9. The system of claim 7 , wherein the one or more neural sensors configured to obtain the neural signals for the person are configured to:

obtain one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp sensor, or in-ear EEG recordings.

10. The system of claim 7 , wherein the controller is further configured to:

process the transformed selected one of the multiple resultant speaker spectrograms, including to perform one or more of: amplify the transformed selected one of the multiple resultant speaker spectrograms, or attenuate a transformed at least one non-selected speaker spectrogram from the multiple resultant speaker spectrograms.

11. The system of claim 7 , wherein the at least one microphone configured to obtain the combined sound signal for the multiple sound sources includes a single microphone to receive the combined sound signal for the multiple sound sources.

12. The system of claim 7 , wherein the controller configured to apply the neural-network-based speech-separation processing to the combined sound spectrogram is further configured to:

provide the combined sound spectrogram to a deep neural network (DNN) configured to identify individual sound sources from the combined sound spectrogram.

13. The system of claim 7 , wherein the controller configured to compare the attended-speaker spectrogram to the derived multiple resultant speaker spectrograms is configured to:

compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms using normalized correlation analysis.

14. The computer-readable media of claim 8 , wherein the instructions to obtain the neural signals for the person include one or more instructions to:

obtain one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp sensor, or in-ear EEG recordings.

15. The computer-readable media of claim 8 , wherein the instructions include one or more additional instructions to:

process the transformed selected one of the multiple resultant speaker spectrograms, including to perform one or more of: amplify the transformed selected one of the multiple resultant speaker spectrograms, or attenuate a transformed at least one non-selected speaker spectrogram from the multiple resultant speaker spectrograms.

16. The computer-readable media of claim 8 , wherein the instructions to obtain the combined sound signal for the multiple sound sources include one or more instructions to obtain the combined sound signal via a single microphone.

17. The computer-readable media of claim 8 , wherein the instructions to compare the attended-speaker spectrogram to the derived multiple resultant speaker spectrograms comprise one or more instructions to:

compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms using normalized correlation analysis.

18. The computer-readable media of claim 8 , wherein the instructions to apply the neural-network-based speech-separation processing to the combined sound spectrogram includes one or more instructions to further:

provide the combined sound spectrogram to a deep neural network (DNN) configured to identify individual sound sources from the combined sound spectrogram.

Assignments (1)

Continuity (9)