Robust spoofing detection system using deep residual neural networks
Embodiments described herein provide for systems and methods for implementing a neural network architecture for spoof detection in audio signals. The neural network architecture contains a layers defining embedding extractors that extract embeddings from input audio signals. Spoofprint embeddings are generated for particular system enrollees to detect attempts to spoof the enrollee's voice. Optionally, voiceprint embeddings are generated for the system enrollees to recognize the enrollee's voice. The voiceprints are extracted using features related to the enrollee's voice. The spoofprints are extracted using features related to features of how the enrollee speaks and other artifacts. The spoofprints facilitate detection of efforts to fool voice biometrics using synthesized speech (e.g., deepfakes) that spoof and emulate the enrollee's voice.
1. A computer-implemented method for spoofing countermeasures, the method comprising:
obtaining, by a computer, a plurality of training audio signals, including one or more clean training signals having one or more speaker biometric features and one or more spoof training signals having one or more spoofing features;
training, by the computer, a voiceprint embedding extractor for extracting a voiceprint based upon the one or more speaker biometric features, by applying the voiceprint embedding extractor on the one or more clean training signals having the one or more speaker biometric features;
training, by the computer, a spoofprint embedding extractor for extracting a spoofprint distinct from the voiceprint based upon the one or more spoofing features, by applying the spoofprint embedding extractor on the one or more spoof training signals having the one or more spoofing features; and
for an inbound audio signal at a deployment time, extracting, by the computer, an inbound spoofprint based upon the one or more spoofing features of the inbound audio signal by applying the spoofprint embedding extractor on the inbound audio signal.
2. The method according to claim 1 , wherein the computer trains the spoofprint embedding extractor using at least one clean audio signal.
3. The method according to claim 1 , wherein obtaining the plurality of training audio signals includes generating, by the computer, a spoof training signal as a simulated audio signal of a corresponding clean audio signal by executing one or more data augmentation operations on the corresponding clean audio signal.
4. The method according to claim 1 , wherein training the spoofprint embedding extractor includes:
applying, by the computer, the spoofprint embedding extractor on a spoof training signal to extract a training spoofprint based on the one or more spoofing features of the spoof training signal; and
executing, by the computer, a loss function using the training spoofprint extracted by the spoofprint embedding extractor for the spoof training signal.
5. The method according to claim 1 , further comprising updating, by the computer, one or more parameters of the spoofprint embedding extractor based upon a difference between the training spoofprint and an expected training spoofprint.
6. The method according to claim 1 , further comprising training, by the computer, a classifier of a neural network architecture for speaker recognition by applying the neural network architecture on each training voiceprint.
7. The method according to claim 1 , further comprising training, by the computer, a classifier of a neural network architecture for spoof detection by applying the neural network architecture on each training spoofprint.
8. The method according to claim 1 , further comprising:
obtaining, by the computer, a plurality of enrollment audio signals associated with an enrolled user, including one or more clean enrollment signals having the one or more speaker biometric features for the enrolled user and one or more enrollment spoof signals having the one or more spoofing features;
applying, by the computer, the spoofprint embedding extractor on the one or more enrollment spoof signals to extract an enrolled spoofprint for the enrolled user based upon the one or more spoofing features of the one or more enrollment spoof signals.
9. The method according to claim 8 , further comprising for the inbound audio signal at the deployment time, generating, by the computer, a spoof score for the inbound audio signal based upon a distance between the inbound spoofprint and the enrolled spoofprint.
10. The method according to claim 8 , further comprising:
applying, by the computer, the voiceprint embedding extractor on the one or more clean enrollment signals to extract an enrolled voiceprint for the enrolled user based upon the one or more speaker biometric features for the enrolled user of the one or more clean enrollment signals; and
for the inbound audio signal at the deployment time, generating, by the computer, a speaker recognition score for the inbound audio signal based upon a second distance between an inbound voiceprint and the enrolled voiceprint.
11. A computer-implemented method for spoofing countermeasures, the method comprising:
extracting, by a computer, an input voiceprint for an input audio signal based upon one or more speaker biometric features of the input audio signal by applying a voiceprint embedding extractor for extracting a voiceprint based upon the one or more speaker biometric features;
extracting, by the computer, an input spoofprint for the input audio signal based upon one or more spoofing features of the input audio signal by applying a spoofprint embedding extractor for extracting a spoofprint distinct from the voiceprint based upon the one or more spoofing features;
generating, by the computer, an input combined embedding based upon the input voiceprint and the input spoofprint; and
generating, by the computer, a combined similarity score between the input combined embedding and a second combined embedding.
12. The method according to claim 11 , further comprising identifying, by the computer, the input audio signal as being at least one of genuine or fraudulent based upon the combined similarity score and one or more corresponding preconfigured threshold scores.
13. The method according to claim 11 , further comprising:
extracting, by the computer, an enrolled voiceprint for an enrolled user based upon the one or more speaker biometric features in an enrollment audio signal;
extracting, by the computer, an enrolled spoofprint for the enrolled user based upon the one or more spoofing features in at least one enrollment audio signal; and
generating, by the computer, the second combined embedding as an combined enrollment embedding for the enrolled user based upon the enrolled voiceprint and the enrolled spoofprint.
14. The method according to claim 13 , further comprising generating, by the computer, one or more simulated enrollment spoofing signals by executing one or more data augmentation operations on the enrollment audio signal.
15. The method according to claim 11 , further comprising generating, by the computer, a spoofing score for the input audio signal based upon a distance between the input spoofprint and a second spoofprint used for the second combined embedding.
16. The method according to claim 11 , further comprising generating, by the computer, a voice similarity score for the input audio signal based upon a distance between the input voiceprint and a second voiceprint used for the second combined embedding.
17. The method according to claim 11 , further comprising executing, by the computer, a loss function using the combined similarity score as a distance loss between the input combined embedding as a predicted combined embedding and the second combined embedding as an expected combined embedding indicated by a training label associated with the input audio signal.
18. The method according to claim 17 , further comprising training, by the computer, the voiceprint embedding extractor by applying the voiceprint embedding extractor on a training audio signal having the one or more speaker biometric features, wherein the loss function updates one or more parameters of the voiceprint embedding extractor based upon the distance loss.
19. The method according to claim 17 , further comprising training, by the computer, the spoofprint embedding extractor by applying the spoofprint embedding extractor on at least one training audio signal having the one or more spoofing features, wherein the loss function updates one or more parameters of the spoofprint embedding extractor based upon the distance loss.
20. The method according to claim 17 , further comprising generating, by the computer, one or more simulated spoofed training signals by executing one or more data augmentation operations on the training audio signal.