Granted Patent B2

US 11,373,672 · App. 16/169,194 · Granted Jun 28, 2022

Inventors: Nima Mesgarani (New York, NY); Yi Luo (New York, NY); James O'Sullivan (New York, NY); Zhuo Chen (New York, NY)

Assignee: The Trustees of Columbia University in the City of New York

G10L25/30A61B5/318G10L17/26A61B5/12G10L21/0272G10L25/66

Abstract

Claims (144)

1. A method comprising:

obtaining, by a device, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

applying, by the device, a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein applying the speech-separation processing comprises:

transforming the combined signal into an embedding space;

determining respective reference points in the embedding space for each of the multiple sound sources, with the reference points representing locations of the sound sources in the embedding space for the combined sound signal;

deriving masks for the determined reference points; and

extracting at least one of the multiple sound sources using at least one of the derived masks;

obtaining, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

selecting one of the plurality of separated signals based on the obtained neural signals for the person.

2. The method of claim 1 , wherein obtaining the neural signals for the person comprises:

obtaining one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp, or in-ear EEG recordings.

3. The method of claim 1 , further comprising:

processing the selected one of the plurality of separated sound signals, including performing one or more of: amplifying the selected one of the plurality of separated signals, or attenuating at least one non-selected signal from the plurality of separated signals.

4. The method of claim 1 , wherein obtaining the combined sound signal for the multiple sound sources comprises:

receiving the combined sound signal for the multiple sound sources at a single microphone coupled to the device.

5. The method of claim 1 , wherein applying the neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources comprises:

providing the combined sound signal from the multiple sound sources to a deep neural network (DNN) configured to identify individual sound sources from the combined sound signal.

6. The method of claim 1 , wherein determining the respective reference points comprises:

determining the respective reference points using a deep neural network.

7. The method of claim 1 , wherein deriving the masks comprises:

computing similarity between embedded points within the embedding space and the determined respective reference points.

8. The method of claim 1 , wherein the reference points comprise attractor points in the embedding space.

9. A method comprising:

obtaining, by a device, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

applying, by the device, a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein applying the neural-network-based speech-separation processing comprises:

dividing the combined sound signal into non-overlapping segments;

transforming the non-overlapping segments into respective weighted sums of a learnable overcomplete basis of signals, wherein weight coefficients for the respective weighted sums are non-negative;

performing neural-network-based processing on the respective weighted sums of the learnable overcomplete basis of signals to derive a plurality of mask matrices corresponding to different groups of the multiple sound sources; and

estimating a plurality of reconstructed sounds signals from the derived plurality of mask matrices corresponding to the different groups of the multiple sound sources;

obtaining, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

selecting one of the plurality of separated signals based on the obtained neural signals for the person.

10. The method of claim 9 , wherein transforming the non-overlapping segments into the respective weighted sums comprises:

estimating the respective weighted sums of the learnable overcomplete basis of signals using a gated 1-D convolution layer according to:

w k =ReLU( x k *U )⊙σ( x k *V ), k= 1,2, . . . , K

where U∈R N×L and V∈R N×L are N vectors with length L, w k ∈R 1×N is a mixture weight vector for segment k, σ denotes a Sigmoid activation function, and * denotes a convolution operator.

11. The method of claim 9 , wherein performing neural-network-based processing on the respective weighted sums to derive the plurality of mask matrices comprises:

inputting the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function for mask generation.

12. The method of claim 9 , wherein estimating the plurality of reconstructed sounds signals comprises:

computing a source weight matrix, D i , according to D i =W⊙M i , where D i =[d i,1 , . . . , d i,K ]∈R K×N is the weight matrix for source i; and

synthesizing a time-domain synthesis of the sources by multiplying the weight matrix D i with basis signals B∈R N×L .

13. A method comprising:

obtaining, by a device, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

applying, by the device, a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein applying the speech-separation processing comprises:

representing the combined sound signal as a time-frequency mixture signal in a time-frequency space;

projecting the time-frequency mixture signal into an embedding space comprising multiple embedded time-frequency bins;

tracking respective reference points for each of the multiple sound sources, with the reference points representing locations of the multiple sound sources in the embedding space, based at least in part on previous locations of the respective reference points at one or more earlier time instances;

deriving masks for the tracked respective reference points; and

extracting at least one of the multiple sound sources using at least one of the derived masks;

obtaining, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

selecting one of the plurality of separated signals based on the obtained neural signals for the person.

14. The method of claim 13 , wherein projecting the time-frequency mixture signal into the embedding space comprises:

processing the time-frequency mixture signal with a neural network comprising a plurality of stacked long short-term memory (LSTM) layers, coupled to a fully connected network.

15. The method of claim 13 , wherein the reference points comprise attractor points in the embedding space.

16. The method of claim 13 , wherein tracking the respective reference points comprises:

computing distances of current embedded time-frequency bins to previous locations of the respective reference points at an earlier time instance; and

assigning the each of the current embedded time-frequency bins to respective ones of the multiple sources based on the computed distances.

17. The method of claim 16 , further comprising updating current locations of the reference points based, at least in part, on assignments of the each of the current embedded time-frequency bins to the respective ones of the multiple sources;

and wherein deriving masks for the tracked reference points comprises generating the masks based, at least in part, on the updated current locations of the references points.

18. A method comprising:

obtaining, by a device, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

applying, by the device, a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein applying the speech-separation processing comprises:

dividing the combined sound signal into a plurality of segments;

transforming the plurality of segments into a plurality of corresponding encoded segments represented in an intermediate feature space;

estimating, for each of the plurality of corresponding encoded segments, multiple mask functions for respective ones of the multiple sound sources by passing the each of the plurality of corresponding encoded segments through a stacked dilated convolutional network;

multiplying the estimated mask functions with the respective one of the plurality of corresponding encoded segments to produce respective resultant multiple masked segments; and

generating separated estimates of the signals from the multiple sound sources based on the respective resultant multiple masked segments;

obtaining, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

selecting one of the plurality of separated signals based on the obtained neural signals for the person.

19. A system comprising:

at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:

apply a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein the controller configured to apply the speech-separation processing is configured to:

transform the combined signal into an embedding space;

determine respective reference points in the embedding space for each of the multiple sound sources, with the reference points representing locations of the sound sources in the embedding space for the combined sound signal;

derive masks for the determined reference points; and

extract at least one of the multiple sound sources using at least one of the derived masks; and

select one of the plurality of separated signals based on the obtained neural signals for the person.

20. A system comprising:

at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:

apply a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein the controller configured to apply the speech-separation processing is configured to:

divide the combined sound signal into non-overlapping segments;

transform the non-overlapping segments into respective weighted sums of a learnable overcomplete basis of signals, wherein weight coefficients for the respective weighted sums are non-negative;

perform neural-network-based processing on the respective weighted sums of the learnable overcomplete basis of signals to derive a plurality of mask matrices corresponding to different groups of the multiple sound sources; and

estimate a plurality of reconstructed sounds signals from the derived plurality of mask matrices corresponding to the different groups of the multiple sound sources; and

select one of the plurality of separated signals based on the obtained neural signals for the person.

21. The system of claim 20 , wherein the controller configured to perform the neural-network-based processing on the respective weighted sums to derive the plurality of mask matrices is configured to:

input the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function for mask generation.

22. A system comprising:

at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:

apply a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein the controller configured to apply the speech-separation processing is configured to:

represent the combined sound signal as a time-frequency mixture signal in a time-frequency space;

project the time-frequency mixture signal into an embedding space comprising multiple embedded time-frequency bins using a neural network comprising a plurality of stacked long short-term memory (LSTM) layers, coupled to a fully connected network;

track respective reference points for each of the multiple sound sources, with the reference points representing locations of the multiple sound sources in the embedding space, based at least in part on previous locations of the respective reference points at one or more earlier time instances;

derive masks for the tracked respective reference points; and

extract at least one of the multiple sound sources using at least one of the derived masks; and

select one of the plurality of separated signals based on the obtained neural signals for the person.

23. A non-transitory computer readable media programmed with instructions, executable on a processor, to:

obtain, by a device comprising the processor, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;

apply, by the device, a neural-network-based speech-separation processing to the combined sound signal from the multiple sound sources to derive a plurality of separated signals that each contains signals corresponding to different groups of the multiple sound sources, wherein the instructions to apply the speech-separation processing include one or more instructions to:

transform the combined signal into an embedding space;

determine respective reference points in the embedding space for each of the multiple sound sources, with the reference points representing locations of the sound sources in the embedding space for the combined sound signal;

derive masks for the determined reference points; and

extract at least one of the multiple sound sources using at least one of the derived masks;

obtain, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and

select one of the plurality of separated signals based on the obtained neural signals for the person.

24. The system of claim 19 , wherein the one or more neural sensors configured to obtain the neural signals for the person are configured to:

obtain one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp sensor, or in-ear EEG recordings.

25. The system of claim 19 , wherein the controller is further configured to:

process the selected one of the plurality of separated sound signals, including to perform one or more of: amplify the selected one of the plurality of separated signals, or attenuate at least one non-selected signal from the plurality of separated signals.

26. The system of claim 19 , wherein the at least one microphone configured to obtain the combined sound signal for the multiple sound sources includes a single microphone to receive the combined sound signal for the multiple sound sources.

27. The system of claim 19 , wherein the controller configured to determine the respective reference points is configured to:

determine the respective reference points using a deep neural network.

28. The system of claim 19 , wherein the controller configured to derive the masks is configured to:

compute similarity between embedded points within the embedding space and the determined respective reference points.

29. The system of claim 19 , wherein the reference points comprise attractor points in the embedding space.

30. The system of claim 20 , wherein the controller configured to transform the non-overlapping segments into the respective weighted sums is configured to:

estimate the respective weighted sums of the learnable overcomplete basis of signals using a gated 1-D convolution layer according to:

w k =ReLU( x k *U )⊙σ( x k *V ), k= 1,2, . . . , K,

where U∈R N×L and V∈R N×L are N vectors with length L, w k ∈R 1×N is a mixture weight vector for segment k, σ denotes a Sigmoid activation function, and * denotes a convolution operator.

31. The system of claim 20 , wherein the controller configured to perform neural-network-based processing on the respective weighted sums to derive the plurality of mask matrices is configured to:

input the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function for mask generation.

32. The system of claim 20 , wherein the controller configured to estimate the plurality of reconstructed sounds signals is configured:

compute a source weight matrix, D i , according to D i =W⊙M i , where D i =[d i,1 , . . . , d i,K ]∈R K×N is the weight matrix for source i; and

synthesize a time-domain synthesis of the sources by multiplying the weight matrix D i with basis signals B∈R N×L .

33. The system of claim 20 , wherein the one or more neural sensors configured to obtain the neural signals for the person are configured to:

obtain one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp sensor, or in-ear EEG recordings.

34. The system of claim 22 , wherein the controller configured to track the respective reference points is configured to:

compute distances of current embedded time-frequency bins to previous locations of the respective reference points at an earlier time instance; and

assign the each of the current embedded time-frequency bins to respective ones of the multiple sources based on the computed distances.

35. The system of claim 22 , wherein the controller is further configured to:

update current locations of the reference points based, at least in part, on assignments of the each of the current embedded time-frequency bins to the respective ones of the multiple sources;

and wherein the controller configured to derive masks for the tracked reference points is configured to generate the masks based, at least in part, on the updated current locations of the references points.

36. The system of claim 22 , wherein the one or more neural sensors configured to obtain the neural signals for the person are configured to:

obtain one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp sensor, or in-ear EEG recordings.

37. The system of claim 22 , wherein the reference points comprise attractor points in the embedding space.

38. The method of claim 18 , wherein obtaining the neural signals for the person comprises:

obtaining one or more of: electrocorticography (ECoG) signals for the person, neural measurements via a non-invasive scalp, or in-ear EEG recordings.

39. The method of claim 18 , further comprising:

processing the selected one of the plurality of separated sound signals, including performing one or more of: amplifying the selected one of the plurality of separated signals, or attenuating at least one non-selected signal from the plurality of separated signals.

Assignments (1)

Continuity (8)

Cited By (10)

US 12,223,105 US 12,231,851 US 12,266,378 US 12,315,515 US 12,361,956 US 12,367,871 US 12,395,783 US 12,424,232 US 12,620,044 US 12,620,399