Speaker specific speech enhancement
Embodiments described herein provide for a machine-learning architecture system that enhances the speech audio of a user-defined target speaker by suppressing interfering speakers, as well as background noise and reverberations. The machine-learning architecture includes a speech separation engine for separating the speech signal of a target speaker from a mixture of multiple speakers' speech, and a noise suppression engine for suppressing various types of noise in the input audio signal. The speaker-specific speech enhancement architecture performs speaker mixture separation and background noise suppression to enhance the perceptual quality of the speech audio. The output of the machine-learning architecture is an enhanced audio signal improving the voice quality of a target speaker on a single-channel audio input containing a mixture of speaker speech signals and various types of noise.
1 . A computer-implemented method comprising:
receiving, by a computer, an inbound audio signal including a signal mixture of a plurality of inbound speakers;
extracting, by the computer, a first set of one or more features from the inbound audio signal;
extracting, by the computer, a speaker mask for the inbound audio signal by applying a first machine-learning architecture on a target enrolled voiceprint and the first set of one or more features, wherein the target enrolled voiceprint comprises an enrollment feature vector;
generating, by the computer, a second set of one or more features of a target audio signal for a target speaker of the plurality of inbound speakers by applying the speaker mask on the first set of one or more features;
extracting, by the computer, a noise mask for the target audio signal having one or more types of noise by applying a second machine-learning architecture on the second set of one or more features;
generating, by the computer, an enhanced audio signal by applying the noise mask on the second set of one or more features of the target audio signal, the enhanced audio signal includes the target audio signal having suppressed noise of the one or more types of noise in the target audio signal according to the noise mask; and
generating, by the computer, an updated target enrolled voiceprint as an updated version of the target enrolled voiceprint corresponding to the target speaker based upon a feature vector extracted from the enhanced audio signal.
2 . The method according to claim 1 , wherein the computer generates a third set of one or more features of the enhanced audio signal by applying the noise mask, and wherein generating the enhanced audio signal includes:
applying, by the computer, a waveform synthesis engine on the third set of one or more features of the enhanced audio signal.
3 . The method according to claim 1 , wherein extracting the speaker mask for inbound audio signal includes:
determining, by the computer, a target speaker of the plurality of inbound speakers based upon one or more characteristics of the inbound audio signal; and
retrieving, by the computer, the enrolled voiceprint corresponding to the target speaker as the target enrolled voiceprint, the enrolled voiceprint stored in a non-transitory memory.
4 . The method according to claim 3 , wherein the computer generates a third set of one or more features of the enhanced audio signal by applying the noise mask, the method further comprising:
extracting, by the computer, the feature vector for the enhanced audio signal by applying a third machine-learning architecture on the third set of one or more features; and
updating, by the computer, the target enrolled voiceprint based upon the feature vector extracted from the third set of one or more features, thereby generating the updated target enrolled voiceprint.
5 . The method according to claim 4 , further comprising updating, by the computer, the target enrolled voiceprint by applying the third machine-learning architecture on one or more subsequent portions of the inbound audio signal.
6 . The method according to claim 1 , wherein the target enrolled voiceprint is a default voiceprint stored in a non-transitory memory.
7 . The method according to claim 1 , wherein the computer extracts the noise mask based upon a ratio of a magnitude of a first frequency spectrum of a clean audio signal to the magnitude of a second frequency spectrum of a noisy audio signal, the noisy audio signal containing at least one of a multi-speaker signal mixture and a type of noise.
8 . The method according to claim 1 , further comprising transmitting, by the computer executing a voice messaging software application, the enhanced audio signal to a second computing device executing the voice messaging software application.
9 . The method according to claim 1 , further comprising:
generating, by the computer, a similarity score based upon the enhanced audio signal and an enrolled voiceprint stored in a non-transitory memory; and
identifying, by the computer, the target speaker as an enrolled speaker in response to determining that the similarity score satisfies a speaker recognition threshold.
10 . The method according to claim 9 , further comprising:
extracting, by the computer, one or more enrollment speaker embeddings from one or more enrollment audio signals for the enrolled speaker; and
extracting, by the computer, the enrolled voiceprint for the enrolled speaker based upon the one or more enrollment speaker embeddings.
11 . A system comprising:
a non-transitory machine readable storage configured to store machine-executable instructions for one or more machine-learning architectures; and
a computer comprising a processor and configured to:
receive an inbound audio signal including a signal mixture of a plurality of inbound speakers;
extract a first set of one or more features from the inbound audio signal;
extract a speaker mask for the inbound audio signal by applying a first machine-learning architecture on a target enrolled voiceprint and the first set of one or more features, wherein the target enrolled voiceprint comprises an enrollment feature vector;
generate a second set of one or more features of a target audio signal for a target speaker of the plurality of inbound speakers by applying the speaker mask on the first set of one or more features;
extract a noise mask for the target audio signal having one or more types of noise by applying a second machine-learning architecture on the second set of one or more features;
generate an enhanced audio signal by applying the noise mask on the second set of one or more features of the target audio signal, the enhanced audio signal includes the target audio signal having suppressed noise of the one or more types of noise in the target audio signal according to the noise mask; and
generate an updated target enrolled voiceprint as an updated version of the target enrolled voiceprint corresponding to the target speaker based upon a feature vector extracted from the enhanced audio signal.
12 . The system according to claim 11 , wherein the computer generates a third set of one or more features of the enhanced audio signal using the noise mask, and wherein when generating the enhanced audio signal the computer is configured to:
apply a waveform synthesis engine on the third set of one or more features of the enhanced audio signal.
13 . The system according to claim 11 , wherein when extracting the speaker mask for inbound audio signal the computer is configured to:
determining, by the computer, a target speaker of the plurality of inbound speakers based upon one or more characteristics of the inbound audio signal; and
retrieving, by the computer, the enrolled voiceprint corresponding to the target speaker as the target enrolled voiceprint, the enrolled voiceprint stored in a non-transitory memory.
14 . The system according to claim 13 , wherein the computer generates a third set of one or more features of the enhanced audio signal by applying the noise mask, and wherein the computer is further configured to:
extract the feature vector for the enhanced audio signal by applying a third machine-learning architecture on the third set of one or more features; and
update the target enrolled voiceprint based upon the feature vector extracted from the third set of one or more features, thereby generating the updated target enrolled voiceprint.
15 . The system according to claim 14 , wherein the computer is configured to update the target enrolled voiceprint by applying the third machine-learning architecture on one or more subsequent portions of the inbound audio signal.
16 . The system according to claim 11 , wherein the target enrolled voiceprint is a default voiceprint stored in a non-transitory memory.
17 . The system according to claim 11 , wherein the computer extracts the noise mask based upon a ratio of a magnitude of a first frequency spectrum of a clean target audio signal to the magnitude of a second frequency spectrum of a noisy audio signal, the noisy audio signal containing at least one of a multi-speaker signal mixture and a type of noise.
18 . The system according to claim 11 , wherein the computer is further configured to:
execute a voice messaging software application; and
transmit the enhanced audio signal to a second computing device according to the voice messaging software application.
19 . The system according to claim 11 , wherein the computer is further configured to:
generate a similarity score based upon the enhanced audio signal and an enrolled voiceprint stored in a non-transitory memory; and
identify the target speaker as an enrolled speaker in response to determining that the similarity score satisfies a speaker recognition threshold.
20 . The system according to claim 19 , wherein the computer is further configured to:
extract one or more enrollment speaker embeddings from one or more enrollment audio signals for the enrolled speaker; and
extract the enrolled voiceprint for the enrolled speaker based upon the one or more enrollment speaker embeddings.
21 . A computer-implemented method comprising:
receiving, by a computer, an inbound audio signal including a signal mixture of a plurality of inbound speakers;
applying, by the computer, a speaker suppression machine-learning architecture to the inbound audio signal and a target enrolled voiceprint to generate a target audio signal for a target speaker of the plurality of inbound speakers, wherein the target enrolled voiceprint comprises an enrollment feature vector;
applying, by the computer, a noise suppression machine-learning architecture to the target audio signal having one or more types of noise;
generating, by the computer, an enhanced audio signal based on an output of the noise suppression machine-learning architecture, the enhanced audio signal includes the target audio signal having suppressed noise of the one or more types of noise in the target audio signal according to the noise mask;
extracting, by the computer, a feature vector for the enhanced audio signal by applying a speaker embedding extraction machine-learning architecture on the enhanced audio signal; and
updating, by the computer, the target enrolled voiceprint with an updated target enrolled voiceprint corresponding to the target speaker generated based upon the feature vector extracted from the enhanced audio signal.