High-resolution video generation using image diffusion models
In various examples, systems and methods are disclosed relating to aligning images into frames of a first video using at least one first temporal attention layer of a neural network model. The first video has a first spatial resolution. A second video having a second spatial resolution is generated by up-sampling the first video using at least one second temporal attention layer of an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.
1. A processor, comprising:
one or more circuits to:
align a plurality of images into frames of a first video using a neural network model comprising a latent diffusion model (LDM), wherein the first video has a first spatial resolution, the LDM comprises:
an encoder to map an input from an image space to a latent space; and
a decoder to map latent encoding from the latent space to the image space; and
generate a second video having a second spatial resolution by up-sampling the first video using an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution, wherein the decoder is updated according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space.
2. The processor of claim 1 , wherein the neural network model is modified from an image diffusion model by adding at least one first temporal attention layer into the image diffusion model.
3. The processor of claim 1 , wherein the up-sampler neural network model comprises a first diffusion model for video generation, and the first diffusion model is modified from a second diffusion model for image generation by adding at least one second temporal attention layer into the first diffusion model for image generation.
4. The processor of claim 1 , wherein the plurality of images are consecutive frames of the first video.
5. The processor of claim 1 , wherein:
the neural network model comprises a first diffusion model and a second diffusion model;
the first diffusion model is to generate a third video; and
the second diffusion model is to generate the first video by generating at least one frame between two consecutive frames of the third video.
6. The processor of claim 1 , wherein the neural network model is to:
generate a third video; and
generate the first video by generating at least one frame between two consecutive frames of the third video according to relative time step embedding.
7. The processor of claim 1 , wherein the first video is generated according to at least one of:
one or more text prompts;
one or more bounding boxes; or
one or more conditioning signals.
8. A processor, comprising:
one or more circuits to:
update a neural network model to align a plurality of images into frames of a first video by updating at least one layer of the neural network model, wherein the first video has a first spatial resolution, wherein the neural network model comprises a Latent Diffusion Model (LDM) comprising:
an encoder to map an input from an image space to a latent space; and
a decoder to map latent encoding from the latent space to the image space,
wherein to update the neural network model, the one or more processing units are to update the decoder according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space; and
update an up-sampler neural network model to generate a second video having a second spatial resolution via up-sampling the first video by updating at least one layer of the up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.
9. The processor of claim 8 , wherein the one or more circuits are to:
insert the at least one neural network layer into a denoising neural network of an image diffusion model; and
update the at least one neural network layer to align the plurality of images into the frames of the first video.
10. The processor of claim 8 , wherein the one or more circuits are to:
update at least one other layer of the neural network model using a plurality of sample images; and
update the at least one layer using at least one sample sequence of images, while maintaining parameters of the at least one other layer unchanged.
11. The processor of claim 8 , wherein the plurality of images are consecutive frames of the first video.
12. The processor of claim 8 , wherein:
the neural network model comprises a first diffusion model and a second diffusion model;
the first diffusion model is updated to generate a third video; and
the second diffusion model is updated to generate the first video by generating at least one frame between two consecutive frames of the third video.
13. The processor of claim 8 , wherein:
the neural network model is updated to generate a third video; and
the neural network model is updated to generate the first video by generating at least one frame between two consecutive frames of the third video according to relative time step embedding.
14. The processor of claim 8 , wherein the one or more circuits are to update the neural network model to generate the first video according to at least one of:
one or more text prompts;
one or more bounding boxes; or
one or more conditioning signals.
15. The processor of claim 8 , wherein the one or more circuits are to update the neural network model to generate the first video according to modified conditioning signals for classifier-free guidance.
16. A method, comprising:
updating a neural network model to align a plurality of images into frames of a first video by updating at least one first temporal attention layer of the neural network model, wherein the first video has a first spatial resolution, wherein the neural network model comprises a Latent Diffusion Model (LDM) comprising:
an encoder to map an input from an image space to a latent space; and
a decoder to map latent encoding from the latent space to the image space,
wherein updating the neural network model comprises updating the decoder according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space; and
updating an up-sampler neural network model to generate a second video having a second spatial resolution via up-sampling the first video, by updating at least one second temporal attention layer of the up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.