Synthetic speech detection
Systems and methods for synthetic speech detection includes receiving an input sample comprising audio and extracting acoustic features corresponding to speech in the audio. The extracted acoustic features are processed using a plurality of neural networks to output abstracted features and generating a feature vector corresponding to the abstracted features using pooling. Training of an SSD task, a speaker classification task, and a channel classification task are performed at a same time, using the feature vector. Synthetic speech is detected using at least the trained SSD task.
1 . A computerized method for synthetic speech detection (SSD), the computerized method comprising:
receiving an input sample comprising audio;
extracting a plurality of acoustic features corresponding to speech in the audio;
processing the plurality of extracted acoustic features in parallel using a plurality of neural networks to output abstracted feature vectors, wherein the plurality of neural networks includes at least deep neural networks (DNNs) having a shared output used to perform training of an SSD model, a speaker classification model, and a channel classification model;
generating, via the plurality of neural networks, a pooled feature vector from the output abstracted feature vectors using a pooling operation, wherein the pooling operation comprises multi-head attentive pooling that assigns to each of the output abstracted feature vectors, a respective attention weight, used to generate the pooled feature vector, and wherein the pooled feature vector is a single vector;
performing the training of an SSD task of the SSD model, a speaker classification task of the speaker classification model, and a channel classification task of the channel classification model, at substantially a same time, using the pooled feature vector; and
detecting synthetic speech using at least the SSD task of the SSD model.
2 . The computerized method of claim 1 , wherein the training is performed using a feed-forward layer comprising the SSD model, the speaker classification model and the channel classification model having shared information.
3 . The computerized method of claim 1 , further comprising identifying at least one of a physical attack (PA) and a logical attack (LA) using the detected synthetic speech.
4 . The computerized method of claim 1 , wherein the pooling operation further comprises an averaging operation using a plurality of weights corresponding to the plurality of extracted acoustic features.
5 . The computerized method of claim 1 , further comprising using a gradient reversal layer in combination with the pooling operation to generate the pooled feature vector.
6 . A system for synthetic speech detection (SSD), the system comprising: at least one processor; and
at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the at least one processor to:
receive an input sample comprising audio;
extract a plurality of acoustic features corresponding to speech in the audio;
process the plurality extracted acoustic features in parallel using a plurality of neural networks to output abstracted feature vectors wherein the plurality of neural networks includes at least deep neural networks (DNNs) having a shared output used to perform training of an SSD model, a speaker classification model, and a channel classification model;
generate, via the plurality of neural networks, a pooled feature vector from to the output abstracted feature vectors using a pooling operation, wherein the pooling operation comprises multi-head attentive pooling that assigns to each of the output abstracted feature vectors, a respective attention weight, used to generate the pooled feature vector, and wherein the pooled feature vector is a single vector;
perform the training of an SSD task of the SSD model, a speaker classification task of the speaker classification model, and a channel classification task of the channel classification model, at substantially a same time, using the pooled feature vector; and
detect synthetic speech using at least the SSD task of the SSD model.
7 . The system of claim 6 , wherein the training is performed using a feed-forward layer comprising the SSD model, the speaker classification model, and the channel classification model having shared information.
8 . The system of claim 6 , further comprising identifying at least one of a physical attack (PA) and a logical attack (LA) using the detected synthetic speech.
9 . The system of claim 6 , wherein the pooling operation further comprises an averaging operation using a plurality of weights corresponding to the plurality of extracted acoustic features.
10 . The system of claim 6 , further comprising using a gradient reversal layer in combination with the pooling operation to generate the feature vector.
11 . One or more computer storage media having computer-executable instructions for synthetic speech detection (SSD) that upon execution by a processor, cause the processor to:
receive an input sample comprising audio;
filter the audio to extract a plurality of acoustic features corresponding to speech in the audio of the input sample;
process the plurality of extracted acoustic features in parallel using a plurality of neural networks to output abstracted feature vectors, wherein the plurality of neural networks includes at least deep neural networks (DNNs) having a shared output used to perform training of an SSD model, a speaker classification model, and a channel classification model;
generate, via the plurality of neural networks, a pooled feature vector from the output abstracted feature vectors using a pooling operation, wherein the pooling operation comprises multi-head attentive pooling that assigns to each of the output abstracted feature vectors, a respective attention weight, used to generate the pooled feature vector, and wherein the pooled feature vector is a single vector;
perform the training of an SSD task of the SSD model, a speaker classification task of the speaker classification model, and a channel classification task of the channel classification model, at substantially a same time, using the pooled feature vector; and
detect synthetic speech using at least the SSD task of the SSD model.
12 . The one or more computer storage media of claim 11 , having further computer-executable instructions that cause the processor to:
decode the processed audio to generate an output, the output including an indication of a word or word sequence received as part of the input sample that includes the detected synthetic speech.
13 . The one or more computer storage media of claim 11 , having further computer-executable instructions that cause the processor to:
generate a log probability that one or more input segments of the audio in the input sample are the synthetic speech.
14 . The one or more computer storage media of claim 13 , wherein the log probability defines a score corresponding to a likelihood that the one or more input segments are the synthetic speech.
15 . The one or more computer storage media of claim 14 , having further computer-executable instructions that cause the processor to:
convert the score to user displayable information showing SSD results and speaker information corresponding to the score.
16 . The one or more computer storage media of claim 11 , having further computer-executable instructions that cause the processor to:
identify at least one of a physical attack (PA) or a logical attack (LA) using the detected synthetic speech.