Methods and systems for generating a multi-dimensional image using cross-view correspondences

1 . A method for generating a three-dimensional (3D) image using one or more diffusion models, the method comprising:

obtaining input data into at least one diffusion model, wherein the input data includes a prompt for generating the 3D image;

generating at least two sets of two-dimensional (2D) images from a 3D representational model that is configured to render the 3D image;

determining a correspondence loss between reprojected points from the 3D representational model and cross-view correspondences from the at least one diffusion model, wherein the cross-view correspondences are determined between image pairs from the at least two sets of 2D images based on feature maps from the at least one diffusion model, and the reprojected points are reprojections from at least one image of the image pairs to a corresponding point in the other image of the image pairs based on the 3D representational model; and

updating the 3D representational model based on the correspondence loss.

2 . The method of claim 1 , wherein

the input data is a text prompt.

3 . The method of claim 1 , further comprising

determining a set of features from the at least two sets of 2D images by:

adding noise to the at least two sets of 2D images from the 3D representational model modulated by a timestep; and

predicting at least one noise component using the at least one diffusion model.

4 . The method of claim 3 , further comprising

up-sampling the at least two sets of 2D images into one or more up-sampled layers; and

extracting the set of features from the at least two sets of 2D images across the one or more up-sampled layers.

5 . The method of claim 4 , wherein

the cross-view correspondences between the image pairs are determined by:

extracting the feature maps from the at least one diffusion model, the feature maps having varying spatial dimensions across one or more layers;

interpolating the feature maps across the one or more layers to a common resolution; and

determining pairwise similarity across the spatial dimensions for the image pairs.

6 . The method of claim 5 , further comprising

normalizing the feature maps.

7 . The method of claim 5 , further comprising

computing a correlation map at each feature level using the determined pairwise similarity across the spatial dimensions;

and aggregating the correlation map from all feature levels for one or more views of the 3D image.

8 . The method of claim 7 , wherein

for each spatial location, the cross-view correspondences are determined as positions with a highest value in the aggregated correlation map.

9 . The method of claim 1 , wherein

the reprojected points are based on camera parameters and rendered depths from the 3D representational model.

10 . The method of claim 1 , further comprising

updating the 3D representational model based on a multi-view diffusion loss.

11 . The method of claim 10 , wherein

the updating of the 3D representational model includes performing the updating based on the correspondence loss and the multi-view diffusion loss in an alternating and iterative manner.

12 . A three-dimensional (3D) image generator system, the system comprising:

one or more processors and one or more memories storing executable models, the executable models comprising:

at least one diffusion model for generating one or more images based on received input data; and

a 3D representational model for rendering a 3D image from the one or more images from the at least one diffusion model,

wherein the 3D representational model is configured to generate at least two sets of two-dimensional (2D) images of the 3D image, and

wherein the 3D representational model is updated based on a correspondence loss, wherein the correspondence loss is determined between reprojected points from the 3D representational model and cross-view correspondences from the at least one diffusion model, wherein the cross-view correspondences are determined between image pairs from the at least two sets of 2D images based on feature maps of the at least one diffusion model, and the reprojected points are reprojections from at least one image of the image pairs to a corresponding point in the other image of the image pairs based on the 3D representational model.

13 . The 3D image generator system of claim 12 , wherein

the at least one diffusion model includes one or more up-sampling layers and wherein the at least one diffusion model is configured to extract a set of features from the at least two sets of 2D images across the one or more up-sampling layers.

14 . The 3D image generator system of claim 12 , further comprising

at least one second diffusion model, wherein the 3D representational model is updated based on a multi-view diffusion loss, wherein the multi-view diffusion loss is determined from a difference between images from one of the at least two sets of 2D images and images generated by the at least one second diffusion model.

15 . The 3D image generator system of claim 14 , wherein

the 3D representational model is updated iteratively using the correspondence loss and the multi-view diffusion loss.

16 . A method for increasing image fidelity in a multi-dimensional (MD) image using one or more diffusion models, the method comprising:

generating at least two sets of images based on a MD representational model that is configured to render the MD image;

determining a correspondence loss between reprojected points from a 3D representational model and cross-view correspondences from at least one diffusion model, wherein the cross-view correspondences are determined between image pairs from the at least two sets of 2D images based on feature maps from the at least one diffusion model, and the reprojected points are reprojections from at least one image of the image pairs to a corresponding point in the other image of the image pairs based on the 3D representational model; and

updating parameters in the MD representational model based on the correspondence loss.

17 . The method of claim 16 , further comprising

determining a set of features from the at least two sets of images by:

adding noise to the at least two sets of images from the MD representational model modulated by a timestep; and

predicting at least one noise component using the at least one diffusion model.