Video transformer for deepfake detection with incremental learning
A method, apparatus, and system for detecting DeepFake videos, includes an input device for inputting a potential DeepFake video, the input device inputs a sequence of video frames of the video, and processing circuitry. The processing circuitry detects faces frame by frame in the video to obtain consecutive face images, creates UV texture maps from the face images, inputs both face images and corresponding UV texture maps, extracts image feature maps, by a convolution neural network (CNN) backbone, from the input face images and corresponding UV texture maps and forms an input data structure, receives the input data structure, by a video transformer model that includes multiple encoders, and computes, by the video transformer model, a classification of the video as being Real or Fake. A display device plays back the potential DeepFake video and an indication that the video is Real or Fake.
1. A system for detecting DeepFake videos, comprising:
an input device for inputting a potential DeepFake video, wherein the input device is configured to input a sequence of video frames of the potential DeepFake video;
processing circuitry that
detects faces frame by frame in the potential DeepFake video to obtain consecutive face images,
creates UV texture maps from the face images,
inputs both face images and corresponding UV texture maps,
extracts image feature maps, by a convolution neural network (CNN) backbone, from the input face images and corresponding UV texture maps and forms an input data structure,
receives the input data structure, by a video transformer model that includes multiple encoders,
computes, by the video transformer model, a classification of the video as being Real or Fake; and
a display device that plays back the potential DeepFake video and an indication that the potential DeepFake video is Real or Fake,
the system further comprising:
learnable segment embeddings, wherein the learnable segment embeddings are a fixed token for the face image and a fixed token for the UV texture map, and
wherein the processing circuitry forms the input data structure including the extracted image feature maps and the learnable segment embeddings,
wherein all tokens belonging to the face image are assigned to a first vector (index 0), and all tokens belonging to the UV texture map are assigned to a second vector (index 1), and the first vector and the second vector are concatenated into a single feature vector.
2. The system of claim 1 , further comprising:
a learnable positional embedding, wherein the learnable positional embedding is a position of a raster image within a video frame,
wherein the processing circuitry forms the input data structure including the extracted image feature maps and the learnable positional embedding.
3. The system of claim 1 , the processing circuitry further comprising:
a training module that trains a combination of the CNN backbone and the transformer model using an incremental learning strategy, the incremental learning including
training the combination using a first dataset that is generated using a faceswap technique;
fine-tuning by further training the combination using a second dataset that is different from the first data set, in which the second dataset is generated using a facial re-enactment technique;
fine-tuning by further training the combination using a third image dataset that is different from the first and second datasets;
fine-tuning by further training the combination using a subset of the first dataset; and
fine-tuning by further training the combination using a subset of the third dataset.
4. The system of claim 3 , wherein the training module trains the combination using a loss function, including:
measuring similarity between weights from a current dataset and a previous dataset, and
measuring accuracy of the training of the combination on the current training set.
5. A DeepFake detection system, comprising:
an artificial intelligence (AI) workstation;
a plurality of client computers connected to the AI workstation,
wherein the AI workstation includes program instructions which when executed by processing circuitry having multiple processing cores, each processing core performs a method comprising:
inputting a sequence of video frames of a video;
detecting faces frame by frame in the video to obtain consecutive face images;
creating UV texture maps from the face images;
inputting both the face images and corresponding said UV texture maps;
extracting image feature maps, by a convolution neural network (CNN) backbone, from the face images and corresponding said UV texture maps and forming an input data structure;
receiving the input data structure, by a video transformer model that includes multiple encoders;
computing, by the video transformer model, a classification of the video as being Real or Fake; and
the plurality of client computers each interacting with the AI workstation in parallel to detect whether a video being played in the respective client computer is Real or Fake and display an indication of the detection,
the system further comprising:
learnable segment embeddings, wherein the learnable segment embeddings are a fixed token for the face image and a fixed token for the UV texture map, and
wherein the processing circuitry forms the input data structure including the extracted image feature maps and the learnable segment embeddings,
wherein all tokens belonging to the face image are assigned to a first vector (index 0), and all tokens belonging to the UV texture map are assigned to a second vector (index 1), and the first vector and the second vector are concatenated into a single feature vector.
6. The DeepFake detection system of claim 5 , wherein the plurality of client computers each inserts a label into a video indicating that the video has been detected as being Real or Fake.
7. The DeepFake detection system of claim 5 , wherein the plurality of client computers each stores those videos that have been detected as being Fake into a database of Fake videos.
8. The DeepFake detection system of claim 5 , wherein the plurality of client computers each send a command to an external system to block a video that has been detected as being Fake from being streamed.
9. A non-transitory computer-readable medium that stores program instructions for implementing an embedded DeepFake detector which when executed by processing circuitry having multiple processing cores, each processing core, in parallel with others of the multiple processing cores, performs a method comprising:
receiving a sequence of video frames of a video captured from a streaming service;
detecting faces frame by frame in the video to obtain consecutive face images;
creating UV texture maps from the face images;
inputting both the face images and corresponding created UV texture maps;
extracting image feature maps, by a convolution neural network (CNN) backbone, from the face images and corresponding created UV texture maps and forming an input data structure;
receiving the input data structure, by a video transformer model consisting of multiple encoders;
computing, by the video transformer model, a classification of the video as being Real or Fake; and
performing an action on the video depending on whether the video is detected as being Real or Fake,
the method further comprising:
providing learnable segment embeddings, wherein the learnable segment embeddings are a fixed token for the face image and a fixed token for the UV texture map, and
forming the input data structure including the extracted image feature maps and the learnable segment embeddings,
wherein all tokens belonging to the face image are assigned to a first vector (index 0), and all tokens belonging to the UV texture map are assigned to a second vector (index 1), and the first vector and the second vector are concatenated into a single feature vector.
10. The non-transitory computer-readable medium of claim 9 , wherein the method includes the action of inserting a label into a video indicating that the video has been detected as being Real or Fake.
11. The non-transitory computer-readable medium of claim 9 , wherein the method includes the action of redirecting those videos that have been detected as being Fake into a database of Fake videos.
12. The non-transitory computer-readable medium of claim 9 , wherein the method includes an action of blocking a video that has been detected as being Fake from being streamed by the video streaming service.
13. The non-transitory computer-readable medium of claim 9 , wherein the detector is a mobile application that is installed in a mobile display device.
14. The non-transitory computer-readable medium of claim 9 , wherein the detector is an application that is performed in a server computer.
15. The non-transitory computer-readable medium of claim 9 , wherein the detector is a middleware device connected between the video streaming service and end user display devices.
16. The non-transitory computer-readable medium claim 9 , further comprising an artificial intelligence (AI) workstation that trains the embedded DeepFake detector using an incremental learning strategy, the incremental learning including
training the detector using a first dataset that is generated using a faceswap technique;
fine-tuning by further training the detector using a second dataset that is different from the first data set, in which the second dataset is generated using a facial re-enactment technique;
fine-tuning by further training the detector using a third image dataset that is different from the first and second datasets;
fine-tuning by further training the detector using a subset of the first dataset; and
fine-tuning by further training the detector using a subset of the third dataset.