Systems and methods for controllable video generation
Embodiments described herein provide a video generation framework built on a decoupled multimodal cross-attention module to simultaneously condition the generation on both an input image and a text input. The video generation may thus be conditioned on the visual appearance of a target object reflected in the input image. In this way, zero-shot video generation may be achieved with little fine-tuning efforts.
1. A method of video generation conditioned on an image and a text description, the method comprising:
encoding an image input depicting a visual appearance of a target object into an image feature vector;
encoding a text input describing a target video containing the target object into a text feature vector;
attending, at an image cross-attention layer within a video diffusion model (VDM), the image feature vector and a latent feature vector of the VDM into an image cross-attention feature;
attending, at a text cross-attention layer within the VDM, the text feature vector and the latent feature vector into a text cross-attention feature; and
generating, by the VDM, a video output based on a combined cross-attention of the image cross-attention feature and the text cross-attention feature.
2. The method of claim 1 , wherein the VDM comprises one or more multimodal video blocks, and wherein each multimodal video block comprises one or more of:
a spatial convolution layer;
a self-attention layer;
the image cross-attention layer;
the text cross-attention layer operated in parallel; and
a temporal-attention layer that receives the combined cross-attention from the image cross-attention layer and the text cross-attention layer.
3. The method of claim 2 , wherein the latent feature vector of the VDM is generated by the spatial convolution layer and the self-attention layer from an input to a respective multimodal video blocks.
4. The method of claim 2 , further comprises:
generating, by the temporal-attention layer, a respective multimodal video block output that captures temporal features from the combined cross-attention; and
feeding the respective multimodal video block output to a next multimodal video block.
5. The method of claim 1 , wherein the video output is generated by the VDM iteratively removing noises from an initialized vector conditioned on the image input and the text input over one or more iterations.
6. The method of claim 5 , wherein the iteratively removing noises further comprises:
at each iteration, the VDM generates an updated vector output from a previous vector output from a previous iteration based on the combined cross-attention from the image cross-attention layer and the text cross-attention layer.
7. The method of claim 1 , further comprising training the VDM using a training video and a training text, wherein the training comprises:
sampling a video frame from the training video as a conditioning image;
iteratively adding a noise term to the training video into a noised video;
iteratively, by the VDM, denoising the noised video by predicting an added noise term conditioned on the conditioning image and the training text; and
updating the VDM based on a loss objective computed based on a difference between the noised term and the predicted added noise term.
8. The method of claim 7 , wherein the predicting the added noise term conditioned on the conditioning image and the training text is performed based on a combined cross-attention of the image cross-attention feature corresponding to the conditioning image and the text cross-attention feature corresponding to the training text.
9. The method of claim 1 , wherein the text input comprises a video editing instruction, and the image input comprises a visual guidance for video editing, and wherein the video output is generated by the VDM to edit a source video by iteratively denoising the source video based on a combined cross-attention of the image cross-attention feature and the text cross-attention feature, wherein the video output is an edited version of the source video conditioned on the image input and the text input.
10. The method of claim 1 , wherein the video output comprises an image animation of the target object.
11. A system of video generation conditioned on an image and a text description, the system comprising:
a memory storing a video diffusion model (VDM) and a plurality of processor-executed instructions; and
one or more processors that execute the plurality of processor-executed instructions to perform operations comprising:
encoding an image input depicting a visual appearance of a target object into an image feature vector;
encoding a text input describing a target video containing the target object into a text feature vector;
attending, at an image cross-attention layer within the video diffusion model (VDM), the image feature vector and a latent feature vector of the VDM into an image cross-attention feature;
attending, at a text cross-attention layer within the VDM, the text feature vector and the latent feature vector into a text cross-attention feature; and
generating, by the VDM, a video output based on a combined cross-attention of the image cross-attention feature and the text cross-attention feature.
12. The system of claim 11 , wherein the VDM comprises one or more multimodal video blocks, and wherein each multimodal video block comprises one or more of:
a spatial convolution layer;
a self-attention layer;
the image cross-attention layer;
the text cross-attention layer operated in parallel; and
a temporal-attention layer that receives the combined cross-attention from the image cross-attention layer and the text cross-attention layer.
13. The system of claim 12 , wherein the latent feature vector of the VDM is generated by the spatial convolution layer and the self-attention layer from an input to a respective multimodal video blocks.
14. The system of claim 12 , wherein the operations further comprise:
generating, by the temporal-attention layer, a respective multimodal video block output that captures temporal features from the combined cross-attention; and
feeding the respective multimodal video block output to a next multimodal video block.
15. The system of claim 11 , wherein the video output is generated by the VDM iteratively removing noises from an initialized vector conditioned on the image input and the text input over one or more iterations.
16. The system of claim 15 , wherein the operation of iteratively removing noises further comprises:
at each iteration, the VDM generates an updated vector output from a previous vector output from a previous iteration based on the combined cross-attention from the image cross-attention layer and the text cross-attention layer.
17. The system of claim 11 , wherein the operations further comprise training the VDM using a training video and a training text, wherein the training comprises:
sampling a video frame from the training video as a conditioning image;
iteratively adding a noise term to the training video into a noised video;
iteratively, by the VDM, denoising the noised video by predicting an added noise term conditioned on the conditioning image and the training text; and
updating the VDM based on a loss objective computed based on a difference between the noised term and the predicted added noise term.
18. The system of claim 17 , wherein the operation of predicting the added noise term conditioned on the conditioning image and the training text is performed based on a combined cross-attention of the image cross-attention feature corresponding to the conditioning image and the text cross-attention feature corresponding to the training text.
19. The system of claim 11 , wherein the text input comprises a video editing instruction, and the image input comprises a visual guidance for video editing, and wherein the video output is generated by the VDM to edit a source video by iteratively denoising the source video based on a combined cross-attention of the image cross-attention feature and the text cross-attention feature, wherein the video output is an edited version of the source video conditioned on the image input and the text input.
20. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for video generation conditioned on an image and a text description, the instructions being executed by one or more processors to perform operations comprising:
encoding an image input depicting a visual appearance of a target object into an image feature vector;
encoding a text input describing a target video containing the target object into a text feature vector;
attending, at an image cross-attention layer within a video diffusion model (VDM), the image feature vector and a latent feature vector of the VDM into an image cross-attention feature;
attending, at a text cross-attention layer within the VDM, the text feature vector and the latent feature vector into a text cross-attention feature; and
generating, by the VDM, a video output based on a combined cross-attention of the image cross-attention feature and the text cross-attention feature.