Video synthesis within a messaging system
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN, by performing operations comprising identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthesized video based on the primary GAN and the input data.
1 . A video synthesis method comprising:
accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator;
generating an updated GAN based on the primary GAN, by performing operations comprising
identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, and
training the motion generator based on the input data using a contrastive image discriminator with contrastive loss to constrain generated images to possess similar quality and content, the contrastive loss comprising positive pairs of augmented frames from a same video that share same content and negative pairs of augmented images from different videos; and
generating a synthesized video based on the primary GAN and the input data,
wherein the pre-trained image generator is trained on a first dataset of a first domain while learning motion generator parameters using an second dataset of a second domain, the first domain and the second domain corresponding to different types of captured subjects, and the first dataset and the second dataset corresponding to different types of data.
2 . The video synthesis method of claim 1 , wherein generating the updated GAN based on the primary GAN further comprises:
adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator.
3 . The video synthesis method of claim 1 , wherein the motion domain dataset corresponds to a motion trajectory vector that is used to synthesize each individual data frame corresponding to the generated synthesized video.
4 . The video synthesis method of claim 1 , wherein the pre-trained image generator is configured to receive the initial latent code and output from the motion generator, to generate the synthesized video.
5 . The video synthesis method of claim 1 , wherein the pre-trained image generator is pre-trained with a primary dataset comprising at least one of real images or a content dataset.
6 . The video synthesis method of claim 5 , wherein a generator corresponding to the motion generator and the pre-trained image generator is trained with a secondary dataset that is different than the primary dataset.
7 . The video synthesis method of claim 1 , wherein the motion generator is configured to receive the initial latent code to predict consecutive latent codes.
8 . The video synthesis method of claim 1 , wherein the motion generator is implemented with two long short-term memory neural networks.
9 . The method of claim 1 , wherein the motion generator learns motion patterns from the second domain to synthesize temporally consistent video frames in the first domain, thereby enabling generation of realistic videos of the first domain using motion extracted from the second domain.
10 . The method of claim 9 , wherein the first domain corresponds to captured subjects of animal faces,
wherein the second domain corresponds to captured subjects of human facial expressions, and
wherein that the pre-trained image generator is trained on the video dataset of the animal faces while learning motion generator parameters using the image dataset of the human facial expressions.
11 . A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising:
accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator;
generating an updated GAN based on the primary GAN, by performing operations comprising
identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, and
training the motion generator based on the input data using a contrastive image discriminator with contrastive loss to constrain generated images to possess similar quality and content, the contrastive loss comprising positive pairs of augmented frames from a same video that share same content and negative pairs of augmented images from different videos; and
generating a synthesized video based on the primary GAN and the input data,
wherein the pre-trained image generator is trained on a first dataset of a first domain while learning motion generator parameters using an second dataset of a second domain, the first domain and the second domain corresponding to different types of captured subjects, and the first dataset and the second dataset corresponding to different types of data.
12 . The system of claim 11 , wherein generating the updated GAN based on the primary GAN further comprises:
adjust weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator.
13 . The system of claim 11 , wherein the motion domain dataset corresponds to a motion trajectory vector that is used to synthesize each individual data frame corresponding to the generated synthesized video.
14 . The system of claim 11 , wherein the pre-trained image generator is configured to receive the initial latent code and output from the motion generator, to generate the synthesized video.
15 . The system of claim 11 , wherein the pre-trained image generator is pre-trained with a primary dataset comprising at least one of real images or a content dataset.
16 . The system of claim 15 , wherein a generator corresponding to the motion generator and the pre-trained image generator is trained with a secondary dataset that is different than the primary dataset.
17 . The system of claim 11 , wherein the motion generator is configured to receive the initial latent code to predict consecutive latent codes.
18 . The system of claim 11 , wherein the motion generator is implemented with two long short-term memory neural networks.
19 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator;
generating an updated GAN based on the primary GAN, by performing operations comprising
identifying input data of the updated GAN, the input data comprising an initial latent code and a motion domain dataset, and
training the motion generator based on the input data using a contrastive image discriminator with contrastive loss to constrain generated images to possess similar quality and content, the contrastive loss comprising positive pairs of augmented frames from a same video that share same content and negative pairs of augmented images from different videos; and
generating a synthesized video based on the primary GAN and the input data,
wherein the pre-trained image generator is trained on a first dataset of a first domain while learning motion generator parameters using an second dataset of a second domain, the first domain and the second domain corresponding to different types of captured subjects, and the first dataset and the second dataset corresponding to different types of data.
20 . The computer-readable storage medium of claim 19 , wherein generating the updated GAN based on the primary GAN further comprises:
adjust weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator.