Video compression system
Systems and methods herein describe a video compression system. The described systems and methods accesses a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame, detects a first set of keypoints for the first image frame, transmits the first image frame and the first set of keypoints to a second computing device, detects a second set of keypoints for the second image frame, transmits the second set of keypoints to the second computing device, causes an animated image to be displayed on the second computing device.
1. A method comprising:
accessing a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame;
detecting, using a motion estimation machine learning model, a first set of keypoints for the first image frame;
transmitting, via a network, the first image frame and the first set of keypoints to a second computing device;
detecting, using the motion estimation machine learning model, a second set of keypoints for the second image frame;
transmitting, via the network, the second set of keypoints to the second computing device;
based on the first set of keypoints and the second set of keypoints, predicting a dense motion field from the sequence of image frames to the first image frame and generating an occlusion mask for the sequence of image frames; and
causing an animated image to be displayed on the second computing device, the animated image generated using an image generation neural network trained to warp the first image frame according to the dense motion field and the occlusion mask to generate the animated image.
2. The method of claim 1 , wherein the motion estimation machine learning model comprises a keypoint detector neural network and a dense motion neural network.
3. The method of claim 2 , wherein the keypoint detector neural network is trained to identify keypoints for centers of a pair of eyes and keypoints for mouth corners.
4. The method of claim 2 , wherein the keypoint detector neural network and the dense motion neural network are configured to generate motion estimation differences between the first set of keypoints and the second set of keypoints.
5. The method of claim 4 , wherein the animated image comprises an animated first image frame and wherein the image generation neural network is configured to generate the animated first image frame by applying the motion estimation differences between the first set of keypoints and the second set of keypoints.
6. The method of claim 1 , wherein the network is a low-bandwidth network.
7. The method of claim 1 , wherein the first image frame, the first set of keypoints and the second set of keypoints are transmitted in real-time.
8. The method of claim 1 , wherein the motion estimation machine learning model operates on the first computing device.
9. The method of claim 1 , wherein the image generation neural network operates on the second computing device, and wherein the image generation neural network is trained to analyze the first image frame, the first set of keypoints and the second set of keypoints.
10. A computing system, the computing system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
accessing a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame;
detecting, using a motion estimation machine learning model, a first set of keypoints for the first image frame;
transmitting, via a network, the first image frame and the first set of keypoints to a second computing device;
detecting, using the motion estimation machine learning model, a second set of keypoints for the second image frame;
transmitting, via the network, the second set of keypoints to the second computing device;
based on the first set of keypoints and the second set of keypoints, predicting a dense motion field from the sequence of image frames to the first image frame and generating an occlusion mask for the sequence of image frames; and
causing an animated image to be displayed on the second computing device, the animated image generated using an image generation neural network trained to warp the first image frame according to the dense motion field and the occlusion mask to generate the animated image.
11. The computing system of claim 10 , wherein the motion estimation machine learning model comprises a keypoint detector neural network and a dense motion neural network.
12. The computing system of claim 11 , wherein the keypoint detector neural network is trained to identify keypoints for centers of a pair of eyes and keypoints for mouth corners.
13. The computing system of claim 11 , wherein the keypoint detector neural network and the dense motion neural network are configured to generate motion estimations differences between the first set of keypoints and the second set of keypoints.
14. The computing system of claim 13 , wherein the animated image comprises an animated first image frame and wherein the image generation neural network is configured to generate the animated first image frame by applying the motion estimation differences between the first set of keypoints and the second set of keypoints.
15. The computing system of claim 10 , wherein the network is a low-bandwidth network.
16. The computing system of claim 10 , wherein the first image frame, the first set of keypoints and the second set of keypoints are transmitted in real-time.
17. The computing system of claim 10 , wherein the motion estimation machine learning model operates on the first computing device.
18. The computing system of claim 10 , wherein the image generation neural network operates on the second computing device, and wherein the image generation neural network is trained to analyze the first image frame, the first set of keypoints and the second set of keypoints.
19. A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors of a machine, cause the computer-readable storage medium to perform operations comprising:
accessing a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame;
detecting, using a motion estimation machine learning model, a first set of keypoints for the first image frame;
transmitting, via a network, the first image frame and the first set of keypoints to a second computing device;
detecting, using the motion estimation machine learning model, a second set of keypoints for the second image frame;
transmitting, via the network, the second set of keypoints to the second computing device;
based on the first set of keypoints and the second set of keypoints, predicting a dense motion field from the sequence of image frames to the first image frame and generating an occlusion mask for the sequence of image frames; and
causing an animated image to be displayed on the second computing device, the animated image generated using an image generation neural network trained to warp the first image frame according to the dense motion field and the occlusion mask to generate the animated image.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the network is a low-bandwidth network.