Systems and methods of speaker-independent embedding for identification and verification from audio
Embodiments described herein provide for audio processing operations that evaluate characteristics of audio signals that are independent of the speaker's voice. A neural network architecture trains and applies discriminatory neural networks tasked with modeling and classifying speaker-independent characteristics. The task-specific models generate or extract feature vectors from input audio data based on the trained embedding extraction models. The embeddings from the task-specific models are concatenated to form a deep-phoneprint vector for the input audio signal. The DP vector is a low dimensional representation of the each of the speaker-independent characteristics of the audio signal and applied in various downstream operations.
1. A computer-implemented method for authenticating audio signals using deep phoneprint (DP) embedding vectors, the method comprising:
executing, by the computer, a plurality of task-specific machine learning models using a plurality of features of speech and non-speech portions of an enrollment audio signal having one or more enrollment speaker-independent characteristics as an input to extract a plurality of enrollment speaker-independent embeddings for the enrollment audio signal using one or more embedding extraction layers of each of the plurality of task-specific machine learning models, the plurality of features of the enrollment audio signal including at least one of a spectro-temporal feature of the enrollment audio signal and metadata associated with the enrollment audio signal;
extracting, by the computer, an enrollment DP vector for the enrollment audio signal based upon the plurality of enrollment speaker-independent embeddings extracted for the enrollment audio signal;
executing, by the computer, the plurality of task-specific machine learning models using a plurality of features of speech and non-speech portions of an inbound audio signal having one or more inbound speaker-independent characteristics as the input to extract a plurality of inbound speaker-independent embeddings for the inbound audio signal using one or more embedding extraction layers of each of the plurality of task-specific machine learning models, the plurality of features of the inbound audio signal including at least one of a spectro-temporal feature of the inbound audio signal and metadata associated with the inbound audio signal;
extracting, by the computer, an inbound DP vector for the inbound audio signal based upon the plurality of inbound speaker-independent embeddings extracted for the inbound audio signal; and
generating, by the computer, one or more similarity scores for the inbound audio signal using the inbound DP vector and the enrollment DP vector for the enrolled audio signal.
2. The method according to claim 1 , further comprising determining, by the computer, an authentication classification for the inbound audio signal based upon the one or more similarity scores for the inbound audio signal.
3. The method according to claim 1 , further comprising:
extracting, by the computer, a plurality of features from a training audio signal, the plurality of features including at least one of: a spectro-temporal feature and metadata associated with the training audio signal; and
extracting, by the computer, the plurality of features from the inbound audio signal.
4. The method according to claim 3 , wherein the metadata includes at least one of: a microphone-type used to capture the training audio signal, a device type from which the training audio signal originated, a codec type applied for compression and decompression of the training audio signal for transmission, a carrier from which the training audio signal was originated, a spoofing service used to change a source identifier associated with the training audio signal, a geography associated with the training audio signal, a network type from which the training audio signal is originated, or an audio event indicator associated with the training audio signal; and
wherein the inbound DP vector includes at least one of: the microphone-type, the device type, the codec type, the carrier, the spoofing service, the geography, the network type, or the audio event indicator.
5. The method according to claim 1 , further comprising training, by the computer, the plurality of task-specific machine learning models by executing each of the plurality of task-specific machine learning models using as input a plurality of training audio signals having the one or more speaker-independent characteristics.
6. The method according to claim 5 , wherein training a task-specific machine learning model includes:
generating, by the computer, predicted output data for a training audio signal by executing one or more post-modeling layers of the task-specific machine learning model using as input a first speaker-independent embedding and a second speaker-independent embedding of the training audio signal; and
adjusting, by the computer, one or more hyper-parameters of one or more modeling layers of the task-specific machine learning model by executing a loss function using the predicted output data and expected output data indicated by one or more labels associated with the training audio signal.
7. The method according to claim 5 , wherein training a task-specific machine learning model includes:
generating, by the computer, predicted output data for a training audio signal by executing one or more post-modeling layers of the task-specific machine learning model using as input a first speaker-independent embedding and a second speaker-independent embedding of the training audio signal; and
adjusting, by the computer, one or more hyper-parameters of the task-specific machine learning model by executing a shared loss function of the one or more task-specific machine learning models using the predicted output data and expected output data indicated by one or more labels associated with the training audio signal.
8. The method according to claim 5 , wherein training the plurality of task-specific machine learning models includes:
generating, by the computer, predicted output data for a training audio signal by executing one or more post-modeling layers of the task-specific machine learning model using as input a speaker-independent embedding for a substantial portion of the training audio signal; and
adjusting, by the computer, one or more hyper-parameters of the task-specific machine learning model by executing a loss function using the predicted output data and expected output data indicated by one or more labels associated with the training audio signal.
9. The method according to claim 1 , wherein a task-specific machine learning model comprises at least one of a convolutional neural network, recurrent neural network, and a fully connected neural network.
10. The method according to claim 1 , further comprising:
executing, by the computer, a voice activity detection (VAD) operation using as input a training audio signal, thereby generating one or more speech portions for the training audio signal and one or more non-speech portions for the training audio signal; and
executing, by the computer, the VAD operation using as input the inbound audio signal, thereby generating one or more speech portions for the inbound audio signal and one or more non-speech portions for the inbound audio signal.
11. A computer-implemented method for authenticating audio signals using deep phoneprint (DP) embedding vectors, the method comprising:
executing, by a computer, a plurality of task-specific machine learning models using as input a plurality of features of speech and non-speech portions of an inbound audio signal having one or more speaker-independent characteristics to extract a plurality of speaker-independent embeddings for the inbound audio signal using one or more embedding extraction layers of each of the plurality of task-specific machine learning models, the plurality of features of the inbound audio signal including at least one of a spectro-temporal feature of the inbound audio signal and metadata associated with the inbound audio signal;
extracting, by the computer, a DP vector for the inbound audio signal based upon the plurality of speaker-independent embeddings extracted for the inbound audio signal; and
generating, by the computer, an exclusion list similarity score for the inbound audio signal based upon comparing the DP vector of the inbound audio signal against an exclusion list containing one or more blocked DP vectors to determine a similarity between the inbound audio signal and each blocked DP vector of the exclusion list.
12. The method according to claim 11 , further comprising authorizing, by the computer, the inbound audio signal in response to determining that the exclusion list similarity score of the inbound audio signal satisfies an authorization threshold distance from each blocked DP vector of the exclusion list.
13. The method of claim 11 , wherein generating, by the computer, the exclusion list similarity score includes executing, by the computer, one or more exclusion list modeling layers using as input the DP vector of the inbound audio signal.
14. The method according to claim 13 , further comprising training the one or more exclusion list modeling layers by:
generating, by the computer, predicted classification scores for a plurality of training audio signals by executing one or more exclusion list modeling layers using as input DP vectors extracted for the plurality of training audio signals; and
adjusting, by the computer, one or more hyper-parameters of the exclusion list modeling layers by executing a loss function using the predicted classification scores and expected classification scores indicated by one or more labels associated with the plurality of training audio signals.
15. The method according to claim 14 , wherein the one or more labels associated with the plurality of training audio signals indicate whether a training audio signal of the plurality of training audio signals is fraudulent or genuine.
16. The method according to claim 11 , further comprising:
extracting, by the computer, a plurality of features from a training audio signal, the plurality of features including at least one of: a spectro-temporal feature and metadata associated with the training audio signal; and
extracting, by the computer, the plurality of features from the inbound audio signal.
17. The method according to claim 16 , wherein the metadata includes at least one of: a microphone-type used to capture the training audio signal, a device type from which the training audio signal originated, a codec type applied for compression and decompression of the training audio signal for transmission, a carrier from which the training audio signal was originated, a spoofing service used to change a source identifier associated with the training audio signal, a geography associated with the training audio signal, a network type from which the training audio signal is originated, or an audio event indicator associated with the training audio signal; and
wherein the DP vector of the inbound audio signal includes at least one of: the microphone-type, the device type, the codec type, the carrier, the spoofing service, the geography, the network type and the audio event indicator.
18. The method according to claim 11 , wherein a task-specific machine learning model comprises at least one of a neural network and a Gaussian mixture model.
19. The method according to claim 11 , further comprising training, by the computer, the plurality of task-specific machine learning models by executing each of the plurality of task-specific machine learning models using as input a plurality of training audio signals having the one or more speaker-independent characteristics.
20. The method according to claim 11 , further comprising:
executing, by the computer, a voice activity detection (VAD) operation using as input a training audio signal, thereby generating one or more speech portions for the training audio signal and one or more non-speech portions for the training audio signal; and
executing, by the computer, the VAD operation using as input the inbound audio signal, thereby generating one or more speech portions for the inbound audio signal and one or more non-speech portions for the inbound audio signal.