Graphic rendering using diffusion models
A system or method for generating computer graphics. One or more three-dimensional (3D) scenes are obtained and rasterized into a first set of two-dimensional (2D) images having a first resolution. Features are extracted from the first set of 2D images, and text prompts are generated based on the features. A diffusion model is applied to the features, and the text prompts to generate a second set of 2D images having a second resolution greater than the first resolution. The diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt. The second set of 2D images having the second resolution are caused to be rendered at a client device.
1 . A method comprising:
receiving, from a client device, a sequence of one or more three-dimensional scenes;
rasterizing the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;
extracting features from the first set of one or more two-dimensional images;
generating one or more text prompts based on the extracted features;
for each of the one or more text prompts,
generating a set of prompt embeddings based on the text prompt; and
generating a set of latent variables based on the features associated with the text prompt;
sampling the first set of one or more two-dimensional images at different times;
applying a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and
causing the second set of one or more images to be rendered at a client device.
2 . The method of claim 1 , wherein the features include two-dimensional feature maps extracted by a convolutional neural network.
3 . The method of claim 1 , further comprising:
identifying an object based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the object.
4 . The method of claim 1 , further comprising:
identifying an environment based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the environment.
5 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
receive, from a client device, a sequence of one or more three-dimensional scenes;
rasterize the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;
extract features from the first set of one or more two-dimensional images;
generate one or more text prompts based on the extracted features;
for each of the one or more text prompts,
generate a set of prompt embeddings based on the text prompt; and
generate a set of latent variables based on the features associated with the text prompt;
sample the first set of one or more two-dimensional images at different times;
apply a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and
cause the second set of one or more images to be rendered at a client device.
6 . The computer program product of claim 5 , wherein the features include two-dimensional feature maps extracted by a convolutional neural network.
7 . The computer program product of claim 5 , wherein the one or more processors are further caused to:
identify an object based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the object.
8 . The computer program product of claim 5 , wherein the one or more processors are further caused to:
identify an environment based on the features extracted from the one or more two-dimensional images, wherein the one or more text prompts include a text prompt corresponding to the environment.
9 . A computer system comprising:
one or more processors; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
receiving, from a client device, a sequence of one or more three-dimensional scenes;
rasterize the sequence of one or more three-dimensional scenes into a first set of one or more two-dimensional images having a first resolution;
extract features from the first set of one or more two-dimensional images;
generate one or more text prompts based on the extracted features;
for each of the one or more text prompts,
generate a set of prompt embeddings based on the text prompt; and
generate a set of latent variables based on the features associated with the text prompt;
sample the first set of one or more two-dimensional images at different times;
apply a diffusion model to 1) the features associated with a first subset of the first set of two-dimensional images and associated with a second subset of the first set of two-dimensional images selected to overlap the first subset and 2) the one or more text prompts to generate a second set of one or more two-dimensional images having a second resolution greater than the first resolution, wherein the diffusion model is trained over a dataset comprising images and corresponding text descriptions to generate an image consistent with a text prompt, wherein the diffusion model includes a two-dimensional Unet configured to receive the extracted features from the first set of one or more two-dimensional images, the sampled first set of one or more two-dimensional images, the set of prompt embeddings, and the set of latent variables to generate the second set of one or more images; and
cause the second set of one or more images to be rendered at a client device.