Deepfake detection
Disclosed are systems and methods including software processes executed by a server that detect audio-based synthetic speech (“deepfakes”) in a call conversation. The server applies an NLP engine to transcribe call audio and analyze the text for anomalous patterns to detect synthetic speech. Additionally or alternatively, the server executes a voice “liveness” detection system for detecting machine speech, such as synthetic speech or replayed speech. The system performs phrase repetition detection, background change detection, and passive voice liveness detection in call audio signals to detect liveness of a speech utterance. An automated model update module allows the liveness detection model to adapt to new types of presentation attacks, based on the human provided feedback.
1 . A computer-implemented method for detecting machine-based speech in calls, comprising:
obtaining, by a computer, an audio speech signal for a caller from a caller device;
identifying, by the computer, textual content from the audio speech signal for the caller;
applying, by the computer, a feature extractor to the textual content to generate a plurality of natural language processing (NLP) features, each of the plurality of NLP features indicating a degree of a likelihood that the textual content as machine generated; and
classifying, by the computing, based on the plurality of NLP features, the caller as one of a machine or a human.
2 . The method of claim 1 , wherein applying the feature extractor includes generating, by the computer, an NLP feature indicating authorship verification by applying the feature extractor trained on second textual content from the caller.
3 . The method of claim 1 , wherein applying the feature extractor includes generating, by the computer, an NLP feature indicating a degree of contextual similarity between the textual content of the caller and a second textual content of an agent in the audio speech signal, by applying the feature extractor trained on a textual corpus.
4 . The method of claim 1 , wherein applying the feature extractor includes generating, by the computer, an NLP feature indicating a degree of likelihood that the textual content is generated by a large language model (LLM), by applying the feature extractor trained on a corpus of human text and machine text.
5 . The method of claim 1 , wherein applying the feature extractor includes generating, by the computer, one or more NLP features indicating (i) a sentiment of the caller from a plurality of candidate sentiments and (ii) at least one of an intensity or a duration of the sentiment within the textual content.
6 . The method of claim 1 , wherein applying the feature extractor includes generating, by the computer, an NLP feature identifying a distribution of unique words within the textual content.
7 . The method of claim 1 , wherein the computer classifies the caller based on a plurality of acoustic features extracted from the audio speech signal.
8 . The method of claim 1 , further comprising comparing, by the computer, the plurality of NLP features against a rule defining a combination of NLP features correlated with one of machine-generate speech or human-generated speech.
9 . The method of claim 1 , further comprising classifying, by the computer, the caller as a machine, responsive to determining that the plurality of NLP features satisfying a combination of NLP features correlated with machine generated speech.
10 . The method of claim 1 , further comprising providing, by the computer, via an interface, an indication of the caller classified as one of the machine or human.
11 . A system for detecting machine-based speech in calls, comprising:
a computer having one or more processors and configured to:
obtain an audio speech signal for a caller from a caller device;
identify textual content from the audio speech signal for the caller;
apply a feature extractor to the textual content to generate a plurality of natural language processing (NLP) features, each of the plurality of NLP features indicating a degree of a likelihood that the textual content as machine generated; and
classify based on the plurality of NLP features, the caller as one of a machine or a human.
12 . The system of claim 11 , wherein, when applying the feature extractor, the computer is configured to generate an NLP feature indicating authorship verification by applying the feature extractor trained on second textual content from the caller.
13 . The system of claim 11 , wherein, when applying the feature extractor, the computer is further configured to generate an NLP feature indicating a degree of contextual similarity between the textual content of the caller and a second textual content of an agent in the audio speech signal, by applying the feature extractor trained on a textual corpus.
14 . The system of claim 11 , wherein, when applying the feature extractor, the computer is further configured to generate an NLP feature indicating a degree of likelihood that the textual content is generated by a large language model (LLM), by applying the feature extractor trained on a corpus containing human text and machine text.
15 . The system of claim 11 , wherein, when applying the feature extractor, the computer is further configured to generate one or more NLP features including (i) a sentiment of the caller from a plurality of candidate sentiments and (ii) at least one of an intensity or a duration of the sentiment within the textual content.
16 . The system of claim 11 , wherein, when applying the feature extractor, the computer is further configured to generate an NLP feature indicating a distribution of unique words within the textual content.
17 . The system of claim 11 , wherein the computer is configured to classify the caller based on a plurality of acoustic features extracted from the audio speech signal.
18 . The system of claim 11 , wherein the computer is further configured to compare the plurality of NLP features with a rule defining a combination of NLP features correlated with one of machine-generated speech or human-generated speech.
19 . The system of claim 11 , wherein the computer is further configured to classify the caller as machine, responsive to determining that the plurality of NLP features satisfying a combination of NLP features correlated with machine generated speech.
20 . The system of claim 11 , wherein the computer is further configured to provide, via an interface, an indication of the caller classified as one of the machine or human.