Active voice liveness detection system
Disclosed are systems and methods including software processes executed by a server that detect audio-based synthetic speech (“deepfakes”) in a call conversation. Embodiments include systems and methods for detecting fraudulent presentation attacks using multiple functional engines that implement various fraud-detection techniques, to produce calibrated scores and/or fused scores. A computer may, for example, evaluate the audio quality of speech signals within audio signals, where speech signals contain the speech portions having speaker utterances.
1 . A computer-implemented method for detecting machine-based speech in calls, comprising:
obtaining, by a computer, a verification prompt comprising challenge content for display at a user interface of a user device of a speaker;
obtaining, by the computer, an input audio signal comprising a speech signal containing response content as an utterance of the speaker, wherein the response content in the speech signal purportedly matches to the challenge content of the verification prompt;
extracting, by the computer, a text embedding using a first set of features extracted for the text of the challenge content of the verification prompt, and a spoken content embedding using a second set of features extracted using the speech signal of the input audio signal; and
executing, by the computer, a content verification engine to generate a content verification score indicating a probability that the response content matches to the challenge content, the content verification engine having one or more layers of machine-learning architecture trained to determine a distance between the text embedding and a spoken content embedding and output the content verification score according to the distance.
2 . The method according to claim 1 , further comprising:
executing, by the computer, the content verification engine taking the training text embedding and the training response content embedding to generate a predicted content verification score according to a predicted distance between a training text embedding and a training spoken content embedding; and
determining, by the computer, a level of error for the content verification engine based upon an expected content verification score and the predicted content verification score; and
training, by the computer, the content verification engine by updating a set of one or more hyperparameters of the content verification engine according to the level of error.
3 . The method according to claim 2 , further comprising:
obtaining, by the computer, a plurality of training verification prompts and a plurality of training speech signals, wherein each particular training speech signal includes training spoken content that matches to training challenge content of a training verification prompt corresponding to the particular training speech signal.
4 . The method according to claim 2 , further comprising:
executing, by the computer, a text embedding extractor comprising one or more layers of the machine-learning architecture taking the training challenge content of the training verification prompt as input, to extract the training text embedding for the training challenge content; and
training, by the computer, the text embedding extractor by updating a second set of one or more hyperparameters of the text embedding extractor according to one or more levels of error,
wherein the computer executes the text embedding extractor as trained to extract the text embedding for the text of the challenge content of the verification prompt.
5 . The method according to claim 2 , further comprising:
executing, by the computer, a content embedding extractor comprising one or more layers of the machine-learning architecture taking the training spoken content of the training response content of the training speech signal, to extract the training content embedding for the training response content; and
training, by the computer, the content embedding extractor by updating a second set of one or more hyperparameters of the content embedding extractor according to one or more levels of error,
wherein the computer executes the content embedding extractor as trained to extract the spoken content embedding for the text of the spoken response content of the speech signal.
6 . The method according to claim 1 , wherein obtaining the verification prompt comprising the challenge content for display at the user interface of the user device includes:
randomly generating, by the computer, the text of the challenge content of the verification prompt; and
transmitting, by the computer, the verification prompt to the user device.
7 . The method according to claim 1 , wherein obtaining the verification prompt comprising the challenge content for display at the user interface of the user device includes:
selecting, by the computer, from a database the text of the challenge content associated with the speaker associated with the user device for the verification prompt.
8 . The method according to claim 1 , wherein obtaining the input audio signal comprising the speech signal containing the response content as the utterance of a speaker includes:
identifying, by the computer, the utterance occurring in a speech portion of the input audio signal;
generating, by the computer, the speech signal comprising the speech portion of the input audio signal having the utterance, speech portion is filtered away a plurality of non-speech portions of the input audio signal; and
extracting, by the computer, the second set of one or more features for the spoken content embedding from the speech signal of the input audio signal.
9 . The method according to claim 1 , wherein the computer calibrates the content verification score according to one or more quality parameter values associated with the quality parameters.
10 . The method according to claim 1 , further comprising identifying, by the computer, the inbound audio signal as genuine or fraudulent based upon comparing the content verification score against a probability threshold.
11 . A system for detecting machine-based speech in calls, comprising:
a computer comprising at least one processor, configured to:
obtain a verification prompt comprising challenge content for display at a user interface of a user device of a speaker;
obtain an input audio signal comprising a speech signal containing response content as an utterance of the speaker, wherein the response content in the speech signal purportedly matches to the challenge content of the verification prompt;
extract a text embedding using a first set of features extracted for the text of the challenge content of the verification prompt, and a spoken content embedding using a second set of features extracted using the speech signal of the input audio signal; and
execute the content verification engine taking the training text embedding and the training response content embedding to generate a predicted content verification score according to a predicted distance between a training text embedding and a training spoken content embedding.
12 . The system according to claim 11 , wherein the computer is further configured to:
execute the content verification engine taking the training text embedding and the training response content embedding to generate a predicted content verification score according to a predicted distance between a training text embedding and a training spoken content embedding; and
determine a level of error for the content verification engine based upon an expected content verification score and the predicted content verification score; and
train the content verification engine by updating a set of one or more hyperparameters of the content verification engine according to the level of error.
13 . The system according to claim 12 , wherein the computer is further configured to:
obtain a plurality of training verification prompts and a plurality of training speech signals, wherein each particular training speech signal includes training spoken content that matches to training challenge content of a training verification prompt corresponding to the particular training speech signal.
14 . The system according to claim 12 , wherein the computer is further configured to:
execute a text embedding extractor comprising one or more layers of the machine-learning architecture taking the training challenge content of the training verification prompt as input, to extract the training text embedding for the training challenge content; and
train the text embedding extractor by updating a second set of one or more hyperparameters of the text embedding extractor according to one or more levels of error, and
wherein the computer executes the text embedding extractor as trained to extract the text embedding for the text of the challenge content of the verification prompt.
15 . The system according to claim 12 , wherein the computer is further configured to:
execute a content embedding extractor comprising one or more layers of the machine-learning architecture taking the training spoken content of the training response content of the training speech signal, to extract the training content embedding for the training response content; and
train the content embedding extractor by updating a second set of one or more hyperparameters of the content embedding extractor according to one or more levels of error, and
wherein the computer executes the content embedding extractor as trained to extract the spoken content embedding for the text of the spoken response content of the speech signal.
16 . The system according to claim 11 , wherein when obtaining the verification prompt comprising the challenge content for display at the user interface of the user device, the computer is further configured to:
randomly generate the text of the challenge content of the verification prompt; and
transmit the verification prompt to the user device.
17 . The system according to claim 11 , wherein when obtaining the verification prompt comprising the challenge content for display at the user interface of the user device, the computer is further configured to:
select from a database the text of the challenge content associated with the speaker associated with the user device for the verification prompt.
18 . The system according to claim 11 , wherein when obtaining the input audio signal comprising the speech signal containing the response content as the utterance of a speaker the computer is figure configured to:
identify the utterance occurring in a speech portion of the input audio signal;
generate the speech signal comprising the speech portion of the input audio signal having the utterance, speech portion is filtered away a plurality of non-speech portions of the input audio signal; and
extract the second set of one or more features for the spoken content embedding from the speech signal of the input audio signal.
19 . The system according to claim 11 , wherein the computer calibrates the content verification score according to one or more quality parameter values associated with the quality parameters.
20 . The system according to claim 11 , wherein the computer is further configured to identify the inbound audio signal as genuine or fraudulent based upon comparing the content verification score against a probability threshold.