Deepfake detection
Disclosed are systems and methods including software processes executed by a server that detect audio-based synthetic speech (“deepfakes”) in a call conversation. The server applies an NLP engine to transcribe call audio and analyze the text for anomalous patterns to detect synthetic speech. Additionally or alternatively, the server executes a voice “liveness” detection system for detecting machine speech, such as synthetic speech or replayed speech. The system performs phrase repetition detection, background change detection, and passive voice liveness detection in call audio signals to detect liveness of a speech utterance. An automated model update module allows the liveness detection model to adapt to new types of presentation attacks, based on the human provided feedback.
1 . A computer-implemented method for authenticating users based on speech of audio signals, comprising:
receiving, by a computer via a first communication channel for a resource, a user authentication request for accessing the resource including authentication credentials for a first authentication;
obtaining, by the computer via a second communication channel, responsive to successfully authenticating a user according to the first authentication using the authentication credentials, an audio speech signal of the user for a second authentication;
extracting, by the computer, a plurality of features from the audio speech signal, including one or more audio background features;
generating, by the computer, a plurality of initial liveness scores using the plurality of features by applying a plurality of machine-learning models of a machine-learning architecture to the plurality of features extracted from the audio speech signal, wherein each machine-learning model of the machine-learning architecture is trained to generate a corresponding initial liveness score of the plurality of initial liveness scores using a corresponding subset of features of the plurality of features extracted from the audio speech signal, the initial liveness scores including a first initial liveness score indicating an audio background change generated using at least one subset of features having the one or more audio background features from the plurality of features;
generating, by the computer, based upon the plurality of initial liveness scores generated using the plurality of features extracted from the audio speech signal via the second communication channel, a liveness score indicating a likelihood that the user is a human speaker; and
executing, by the computer, the second authentication to determine whether to permit the user access to the resource of the first communication channel based upon comparing the liveness score against a threshold.
2 . The method of claim 1 , wherein obtaining the audio speech signal includes presenting, by the computer, a prompt to direct the user to provide the audio speech signal, responsive to authenticating the user in the first authentication using the authentication credentials.
3 . The method of claim 1 , wherein generating the liveness score includes applying, by the computer, the machine-learning architecture to the plurality of features to generate the liveness score,
wherein the machine-learning architecture is trained on a plurality of examples, each example identifying a second plurality of features and a label indicating one of human speech or machine-generated speech.
4 . The method of claim 1 , wherein the plurality of initial liveness scores further comprises at least one of: a second score identifying passive liveness of the speech signal of the speaker, or a third score identifying repetition of speech within the speech signal.
5 . The method of claim 1 wherein extracting the plurality of features includes applying, by the computer, the machine-learning architecture to generate the plurality of features including a set of embeddings representing spoofing artifacts in the audio speech signal.
6 . The method of claim 5 , wherein generating the liveness score includes applying, by the computer, the machine-learning architecture to the set of embedding representing the spoofing artifacts to determine the liveness score.
7 . The method of claim 1 , wherein performing the second authentication includes performing, by the computer, the second authentication to restrict the user access to a resource, responsive to the liveness score not satisfying the threshold.
8 . The method of claim 1 , wherein performing the second authentication includes performing, by the computer, the second authentication to permit the user access to a resource, responsive to the liveness score satisfying the threshold.
9 . The method of claim 1 , further comprising generating, by the computer, an indication of a result of the second authentication indicating whether to permit the user access based on the liveness score.
10 . A system for authenticating callers using speech of audio signals in calls, comprising:
a computer comprising one or more processors configured to:
receive, via a first communication channel for a resource, a user authentication request for accessing the resource including authentication credentials for a first authentication;
obtain, via a second communication channel, responsive to successfully authenticating a user according to the first authentication using the authentication credentials, an audio speech signal of the user for a second authentication;
extract a plurality of features from the audio speech signal, including one or more audio background features;
generate a plurality of initial liveness scores using the plurality of features by applying a plurality of machine-learning models of a machine-learning architecture to the plurality of features, wherein each machine-learning model of the plurality of machine-learning models is trained to generate a corresponding initial liveness score of the plurality of initial liveness scores using a corresponding subset of features of the plurality of features extracted from the audio speech signal, the plurality of initial liveness scores including a first initial liveness score indicating an audio background change generated using at least one subset of features having the one or more audio background features from the plurality of features;
generate based upon the plurality of initial liveness scores generated using the plurality of features extracted from the audio speech signal via the second communication channel, a liveness score indicating a likelihood that the user is a human speaker; and
execute the second authentication to determine whether to permit the user access to the resource of the first communication channel based upon comparing the liveness score against a threshold.
11 . The system of claim 10 , wherein, when obtaining the audio speech signal, the computer is further configured to generate a prompt for a user interface, instructing the user to provide the audio speech signal, responsive the computer authenticating the user based on the first authentication of the user using the authentication credentials.
12 . The system of claim 10 , wherein, when generating the liveness score the computer is further configured to apply the machine-learning architecture to the plurality of features to generate the liveness score; and
wherein the machine-learning architecture is trained on a plurality of examples, each example includes a second plurality of features and a label indicating one of human speech or machine-generated speech.
13 . The system of claim 10 , wherein, when generating the plurality of initial liveness scores, the computer is further configured to include at least one of: a second score identifying passive liveness of the speech signal of the speaker, or a third score identifying repetition of speech within the speech signal.
14 . The system of claim 10 wherein, when extracting the plurality of features, the computer is further configured to apply the machine-learning architecture to generate the plurality of features including a set of embeddings representing spoofing artifacts in the audio speech signal.
15 . The system of claim 14 , wherein, when generating the liveness score, the computer is further configured to apply the machine-learning architecture to the set of embedding representing the spoofing artifacts to determine the liveness score.
16 . The system of claim 10 , wherein, when performing the second authentication, the computer is further configured to perform the second authentication to restrict the user access to a resource, responsive to the liveness score not satisfying the threshold.
17 . The system of claim 10 , wherein, when performing the second authentication, the computer is further configured to perform the second authentication to permit the user access to a resource, responsive to the liveness score satisfying the threshold.
18 . The system of claim 10 , wherein the computer is further configured to generate an indication of a result of the second authentication indicating whether to permit the user access based on the liveness score.