Systems and techniques to perform 4D-guided video generation with diffusion models

1 . A computer-implemented method, comprising:

receiving a prompt and an input mesh to generate a four-dimensional (4D) video;

generating, using a diffusion model, keyframes from a depth map and a UV coordinate map based on the input mesh;

extracting, using the diffusion model, features from the keyframes; and

generating frames of the 4D video based on the prompt, UV-guided noise initialization of each object, and injecting the features extracted from each of the keyframes into the diffusion model during a regeneration process.

2 . The computer-implemented method of claim 1 , wherein extracting the features comprises:

performing a plurality of diffusion steps; and

performing extended attention on a subset of the keyframes to extract the features during each of the plurality of diffusion steps.

3 . The computer-implemented method of claim 2 , wherein the features extracted comprise pre-attention features, the pre-attention features comprise a key, a query, and a value, and the method comprises:

concatenating the pre-attention features of the subset of the keyframes with the pre-attention features of a current frame; and

injecting a result of the concatenating into the diffusion model for the current frame.

4 . The computer-implemented method of claim 2 , wherein the features extracted comprise post-attention features, and the post-attention features comprising outputs from a previous attention feature module, and wherein injecting the features includes reprojecting the outputs from the previous attention feature module to a current frame.

5 . The computer-implemented method of claim 2 , wherein the features comprise pre-attention features and post-attention features and the method comprises generating a UV-space feature map by blending the pre-attention features and the post-attention features into correspondence.

6 . The computer-implemented method of claim 5 , wherein blending the pre-attention features and the post-attention features comprises:

blending, for each of the frames, the pre-attention features and the post-attention features from the frames sequentially and filling a texel with features of its corresponding pixel in a particular frame if the corresponding pixel has not been filled before, and

determining, for each of the frames, a unified texture by taking a mean of inpainted texture and an average texture.

7 . The computer-implemented method of claim 1 , comprising generating the depth map and the UV coordinate map with a renderer processing the input mesh.

8 . The computer-implemented method of claim 1 , wherein the input mesh is a scene-level proxy mesh, and the prompt is a text prompt.

9 . The computer-implemented method of claim 1 , comprising performing UV-guided noise initialization for each object in the input mesh by:

generating a Gaussian noise texture for each object in the input mesh, and

projecting the noise textures to each of the frames utilizing a frame-UV correspondence to the regeneration process.

10 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor, cause the processor to perform the operations comprising:

determining, using a renderer, a depth map and a UV coordinate map from an input mesh;

determining, using a diffusion model, a plurality of keyframes from the depth map and the UV map;

identifying, using the diffusion model, pre-attention features and post-attention features;

performing UV-guided noise initialization for each object in the input mesh; and

generating frames for a 4D video based on a prompt, the UV-guided noise initialization, and injecting the pre-attention features and the post-attention features into the diffusion model during a regeneration process performed on noise images.

11 . The computer-readable storage medium of claim 10 , wherein identifying the pre-attention features and the post-attention features comprises the processor performing, for each of a plurality of diffusion steps, extended attention on at least a subset of keyframes to extract the pre-attention features and the post-attention features.

12 . The computer-readable storage medium of claim 10 , wherein the pre-attention features comprise a key, a query, and a value, and the processor to perform:

concatenating the pre-attention features of the subset of the keyframes with the pre-attention features of a current frame; and

injecting a result of the concatenating into the diffusion model during the regeneration process for the current frame.

13 . The computer-readable storage medium of claim 10 , wherein the post-attention features comprise outputs from a previous attention feature module, and the processor injecting the post-attention features by reprojecting the outputs from the previous attention feature module to a current frame.

14 . The computer-readable storage medium of claim 10 , comprises the processor performing the operations comprising:

obtaining the prompt and the input mesh to generate the four-dimensional (4D) video; and

generating a UV-space feature map by blending the pre-attention features and the post-attention features into correspondence.

15 . The computer-readable storage medium of claim 14 , wherein blending the pre-attention features and the post-attention features comprises:

blending, for each of the frames, the pre-attention features and the post-attention features from the frames sequentially and filling a texel with features of its corresponding pixel in a particular frame if the corresponding pixel has not been filled, and

determining, for each of the frames, a unified texture by taking a mean of inpainted texture and an average texture.

16 . The computer-readable storage medium of claim 10 , comprising the processor performing the operations comprising generating the depth map and the UV coordinate map with the renderer processing the input mesh.

17 . The computer-readable storage medium of claim 10 , wherein the input mesh is a scene-level proxy mesh, and the prompt is a text prompt.

18 . The computer-readable storage medium of claim 10 , wherein performing the UV-guided noise initialization comprises:

generating a Gaussian noise texture for each object in the input mesh; and

projecting the noise textures to each of the frames utilizing a frame-UV correspondence.

19 . A computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to perform the operations of:

processing a request to generate a four-dimensional (4D) video;

generating keyframes from a depth map and a UV coordinate map, wherein the depth map and the UV coordinate map are generated from an input mesh;

extracting, using a diffusion model, features from the keyframes; and

generating frames of the 4D video based on a prompt, UV-guided noise initialization of each object, or a combination thereof; and

injecting the features extracted from each of the keyframes into the diffusion model during a regeneration process.

20 . The computing apparatus of claim 19 , wherein the features comprise pre-attention features and post-attention features, the processor to perform blending of the pre-attention features and the post-attention features for injecting into the diffusion model:

blending, for each of the frames, the pre-attention features and the post-attention features from the frames sequentially and filling a texel with features of its corresponding pixel in a particular frame if the corresponding pixel has not been filled before, and

determining, for each of the frames, a unified texture by taking a mean of inpainted texture and an average texture.