DEEPFAKE DETECTION
Disclosed are systems and methods including software processes executed by a server that detect audio-based synthetic speech (“deepfakes”) in a call conversation. The server applies an NLP engine to transcribe call audio and analyze the text for anomalous patterns to detect synthetic speech. Additionally or alternatively, the server executes a voice “liveness” detection system for detecting machine speech, such as synthetic speech or replayed speech. The system performs phrase repetition detection, background change detection, and passive voice liveness detection in call audio signals to detect liveness of a speech utterance. An automated model update module allows the liveness detection model to adapt to new types of presentation attacks, based on the human provided feedback.
1 . A computer-implemented method for training models of a machine-learning architecture to detect machine-based speech in calls, comprising:
obtaining, by a computer, training data including training textual content records and corresponding training labels, each training label indicates that training textual content a corresponding training textual content record is machine-generated or human-generated;
generating, by the computer, a plurality of natural language processing (NLP) features extracted from the textual content of the training data using a feature extractor, each of the plurality of NLP features indicating a degree of likelihood that the textual content is human-generate or machine-generated;
training, by the computer, one or more classification models for classifying a caller as machine or human according to the NLP features derived from call audio of the caller, each classification model is trained based upon the plurality of NLP features extracted for the training textual content and the corresponding training label of the training data;
generating, by the computer, the plurality of natural language processing (NLP) features extracted from the textual content of an inbound audio signal for an inbound caller using the feature extractor; and
classifying, by the computer, the caller as machine or human based upon the plurality of NLP features of the inbound audio signal using the one or more classification models.
2 . The method of claim 1 , wherein generating the plurality of NLP features includes generating, by the computer, an NLP feature indicating authorship verification using the feature extractor trained on second textual content.
3 . The method of claim 1 , wherein generating the plurality of NLP features includes generating, by the computer, an NLP feature indicating a degree of contextual similarity between the textual content of the caller and textual content of an agent, using the feature extractor trained on a textual corpus.
4 . The method of claim 1 , wherein generating the plurality of NLP features includes generating, by the computer, an NLP feature indicating a degree of likelihood that the textual content is generated by a large language model (LLM), using the feature extractor trained on a corpus containing human text and machine-generated text.
5 . The method of claim 1 , wherein generating the plurality of NLP features includes generating, by the computer, one or more NLP features indicating a sentiment of the caller from a plurality of candidate sentiments and at least one of an intensity or a duration of the sentiment within the textual content.
6 . The method of claim 1 , wherein generating the plurality of NLP features includes generating, by the computer, an NLP feature identifying a distribution of unique words within the textual content.
7 . The method of claim 1 , wherein training the one or more classification models comprises further training, by the computer, the classification models to classify the caller based on a plurality of acoustic features extracted from the audio speech signal.
8 . The method of claim 1 , further comprising comparing, by the computer, the plurality of NLP features against a rule-configuration indicating a combination of NLP features correlated with one of machine-generated speech or human-generated speech.
9 . The method of claim 1 , further comprising classifying, by the computer, the caller as machine, responsive to determining that the plurality of NLP features satisfy a combination of NLP features correlated with machine-generated speech.
10 . The method of claim 1 , further comprising providing, by the computer, via an interface, an indication of the caller classified as one of machine or human.
11 . A computer-implemented method for detecting machine-generated speech in audio signals using natural language processing (NLP), comprising:
obtaining, by a computer, an audio speech signal for a first speaker and a second speaker;
identifying, by the computer, first textual content for the first speaker and second textual content for the second speaker in the audio speech signal using an automated speech recognition engine;
applying, by the computer, a feature extractor to the first textual content of the first speaker and the second textual content of the second speaker in the audio speech signal to generate an NLP feature indicating of a degree of contextual similarity between the first textual content and the second textual content;
generating, by the computer, a machine-likelihood score indicating a likelihood that the first textual content represents machine-generated speech from the first speaker based on the NLP feature indicating of the degree of contextual similarity; and
classifying, by the computer, the first speaker as one of a machine or a human based upon the machine-likelihood score.
12 . The method of claim 11 , wherein applying the feature extractor includes generating, by the computer, a second NLP feature indicating authorship verification by applying the feature extractor trained on third textual content from the first speaker.
13 . The method of claim 11 , wherein applying the feature extractor includes generating, by the computer, a second NLP feature for lexical-diversity indicating a distribution of unique words within the first textual content from the first speaker.
14 . The method of claim 11 , wherein applying the feature extractor includes generating, by the computer, a second NLP feature indicating a sentiment of the first speaker and at least one of an intensity or a duration of the sentiment within the first textual content.
15 . The method of claim 11 , further comprising comparing, by the computer, the NLP feature indicating the degree of contextual similarity against a rule-configuration indicating a combination of features correlated with machine-generated speech.
16 . The method of claim 15 , wherein classifying the first speaker as one of machine or human includes determining, by the computer, that the NLP feature satisfies a threshold distance from the combination of features indicated by the rule-configuration correlated with the machine-generated speech.
17 . The method of claim 11 , wherein the second textual content represents a prompt from the second speaker in the audio speech signal, and wherein the first textual content represents a response to the prompt from the first speaker in the audio speech signal.
18 . The method of claim 17 , wherein the computer identifies a mismatch between a context topic in the first textual content of the response and an expected context topic in the second textual content of the prompt.
19 . The method of claim 11 , wherein applying the feature extractor includes generating, by the computer, a second NLP feature indicating a degree of likelihood that the first textual content is generated by a large language model (LLM) by applying the feature extractor trained on a corpus comprising human-generated text and machine-generated text.
20 . The method of claim 11 , further comprising providing, by the computer, via an interface, an indication of the first speaker as one of machine or human.