Graphic rendering using diffusion models

1 . A method comprising:

receiving, from a client device, a sequence of one or more three-dimensional scenes;

rasterizing the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;

extracting features from the first set of one or more two-dimensional images;

generating one or more text prompts based on the extracted features;

for each of the one or more text prompts,

generating a set of prompt embeddings based on the text prompt; and

generating a set of latent variables based on the features associated with the text prompt;

sampling the first set of one or more two-dimensional images at different times;

applying a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and

causing the second set of one or more images to be rendered at a client device.

2 . The method of claim 1 , wherein the features include two-dimensional feature maps extracted by a convolutional neural network.

3 . The method of claim 1 , further comprising:

identifying an object based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the object.

4 . The method of claim 1 , further comprising:

identifying an environment based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the environment.

5 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:

receive, from a client device, a sequence of one or more three-dimensional scenes;

rasterize the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;

extract features from the first set of one or more two-dimensional images;

generate one or more text prompts based on the extracted features;

for each of the one or more text prompts,

generate a set of prompt embeddings based on the text prompt; and

generate a set of latent variables based on the features associated with the text prompt;

sample the first set of one or more two-dimensional images at different times;

apply a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and

cause the second set of one or more images to be rendered at a client device.

6 . The computer program product of claim 5 , wherein the features include two-dimensional feature maps extracted by a convolutional neural network.

7 . The computer program product of claim 5 , wherein the one or more processors are further caused to:

identify an object based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the object.

8 . The computer program product of claim 5 , wherein the one or more processors are further caused to:

identify an environment based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the environment.

9 . A computer system comprising:

one or more processors; and

a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:

receiving, from a client device, a sequence of one or more three-dimensional scenes;

rasterize the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;

extract features from the first set of one or more two-dimensional images;

generate one or more text prompts based on the extracted features;

for each of the one or more text prompts,

generate a set of prompt embeddings based on the text prompt; and

generate a set of latent variables based on the features associated with the text prompt;

sample the first set of one or more two-dimensional images at different times;

apply a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and

cause the second set of one or more images to be rendered at a client device.