Methods and systems for detecting deepfakes
A system for detecting synthetic videos may include a server, a plurality of weak classifiers, and a strong classifier. The server may be configured to receive a prediction result from each of a plurality of weak classifiers; and send the prediction results from each of the plurality of weak classifiers to a strong classifier. The weak classifiers may be trained on real videos and known synthetic videos to analyze a distinct characteristic of a video file; detect irregularities of the distinct characteristic; generate a prediction result associated with the distinct characteristic, the prediction result being a prediction on whether the video file is synthetic; and output the prediction result to the server. The strong classifier may be trained to receive the prediction results of the plurality of weak classifiers from the server; analyze the prediction results; and determine if the video file is synthetic based on the prediction results.
1. A method of training a system to detect fake videos comprising:
creating, by one or more processors, a training set, wherein the training set comprises a first plurality of videos known to be real and a second plurality of videos known to be fake;
identifying, by the one or more processors, a distinct characteristic in each video of the training set;
extracting, by the one or more processors, the distinct characteristic from each video of the training set;
down-sampling, by the one or more processors, the extracted distinct characteristic in at least one frame; and
training, by the one or more processors, a weak classifier to, detect irregularities of down-sampled extracted distinct characteristic and, in response to detecting the irregularities, generate a prediction result on whether a test video is fake, the prediction result being dependent on the distinct characteristic and the detected irregularities.
2. The method of claim 1 further comprising, prior to training the weak classifier, removing, by the one or more processors, blurred frames from each video in the training set known to be fake.
3. The method of claim 1 further comprising, prior to training the weak classifier, removing up-sampled frames from the training set.
4. The method of claim 1 , wherein the prediction result is a numerical confidence level.
5. The method of claim 1 comprising:
extracting a pre-defined number of frames from a video file in which the distinct characteristic has been detected;
extracting the down-sampled distinct characteristic from each extracted frame;
training the weak classifier to generate preliminary prediction results based on the distinct characteristic in each extracted frame; and
training the weak classifier to generate an average prediction from the preliminary prediction results.
6. The method of claim 1 , wherein the distinct characteristic comprises movement of a head, wherein training the weak classifier comprises training the weak classifier to:
detect the movement of the head in a video file;
calculate a pulse based on the detected movement;
detect irregularities of the pulse; and
generate the prediction result associated with the irregularities of the pulse.
7. The method of claim 6 , wherein calculating the pulse based on the detected movement comprises:
identifying a plurality of features of the head;
decomposing trajectories of each feature into a set of component motions;
determining a component that best corresponds to a heartbeat based on its temporal frequency;
identifying peaks associated with the determined component; and
calculating the pulse based on the peaks.
8. The method of claim 1 , wherein the distinct characteristic comprises a mouth, wherein training the weak classifier comprises training the weak classifier to:
detect the mouth in a video file;
extract the mouth from the video file;
detect irregularities of the mouth; and
generate the prediction result based on the irregularities.
9. The method of claim 8 , wherein the irregularities are associated with at least one of teeth or facial hair.
10. The method of claim 1 , wherein the distinct characteristic comprises audio gain, wherein training the weak classifier comprises training the weak classifier to detect irregularities in the audio gain.
11. The method of claim 10 , wherein training the weak classifier to detect irregularities in the audio gain comprises:
stripping background audio from a video file;
isolating a voice in the video file; and
training the weak classifier to detect irregularities based on the audio gain.
12. The method of claim 1 , wherein the weak classifier is a first weak classifier trained to, based on a first distinct characteristic and a down-sampled first distinct characteristic, generate a first prediction result on whether the test video is fake, further comprising training, by the one or more processors, a second weak classifier to, based on a second distinct characteristic and a down-sampled second distinct characteristic, generate a second prediction result on whether the test video is fake.
13. The method of claim 12 comprising training, by the one or more processors, a strong classifier to determine if the test video is fake based on the first and second prediction results.
14. The method of claim 13 , wherein training the strong classifier comprises training the strong classifier to learn weights for each prediction result.
15. A system for training classifiers to detect fake videos comprising:
a database comprising a training set, wherein the training set comprises a first plurality of videos known to be real and a second plurality of videos known to be fake; and
a server communicably coupled to the database and configured to, via one or more processors: identify, by the one or more processors, a distinct characteristic in each video of the training set;
extract, by the one or more processors, the distinct characteristic from each video of the training set;
down-sample, by the one or more processors, the extracted distinct characteristic in at least one frame; and
train, by the one or more processors, a weak classifier to, detect irregularities of the extracted distinct characteristic and, in response to detecting the irregularities of down-sampled extracted, generate a prediction result on whether a test video is fake, the prediction result being dependent on the distinct characteristic and the detected irregularities.
16. The system of claim 15 , wherein the weak classifier is a first weak classifier trained to, based on a first distinct characteristic and a down-sampled first distinct characteristic, generate a first prediction result on whether the test video is fake, wherein the server is further configured to train, by the one or more processors, a second weak classifier to, based on a second distinct characteristic and a down-sampled second distinct characteristic, generate a second prediction result on whether the test video is fake.
17. The system of claim 16 , wherein the server is further configured to train, by the one or more processors, a strong classifier to determine if the test video is fake based on the first and second prediction results.
18. The system of claim 17 , wherein training the strong classifier comprises training the strong classifier to learn weights for each prediction result.
19. The system of claim 15 , wherein the server is configured to, prior to training the weak classifier, remove, by the one or more processors, blurred frames from each video in the training set known to be fake.
20. The system of claim 15 , wherein the server is configured to, prior to training the weak classifier, remove up-sampled frames from the training set.