Consistent latent diffusion for mesh texturing
A method of generating textures for a 3D mesh is provided. In the method, the 3D mesh is received. The 3D mesh includes a plurality of vertices and a plurality of faces. The plurality of faces is formed based on the plurality of vertices. A latent texture map is generated based on a plurality of latent images of the 3D mesh in a latent space from a plurality of view angles. The latent texture map is denoised to remove noise based on a diffusion process. The textures are generated for the 3D mesh in a pixel space based on the denoised latent texture map.
1 . A method of generating textures for a three-dimensional (3D) mesh, the method comprising:
receiving the 3D mesh that includes a plurality of vertices and a plurality of faces formed based on the plurality of vertices;
generating a latent texture map based on a plurality of latent images of the 3D mesh in a latent space from a plurality of view angles;
denoising the latent texture map to remove noise based on a diffusion process; and
generating the textures for the 3D mesh in a pixel space based on the denoised latent texture map.
2 . The method of claim 1 , wherein the generating the latent texture map further comprises:
generating the latent texture map based on rendered textures of the 3D mesh in the pixel space for the plurality of view angles, the latent texture map including latent textures of the 3D mesh and a plurality of independent identically distributed (I. I. D.) Gaussian noises, each of a plurality of latent pixels in the latent texture map including a respective latent value; and
determining a spherical harmonic coefficient for each of the latent values of the plurality of latent pixels in the latent texture map based on the view angle associated with the respective latent value.
3 . The method of claim 2 , wherein a selected region of each of the plurality of latent images of the latent texture map includes a same set of the I. I. D. Gaussian noises.
4 . The method of claim 1 , wherein the denoising further comprises:
rendering latent textures into a first latent image of the plurality of latent images of the latent texture map, the first latent image being generated for a first view angle;
denoising the latent textures of the first latent image in a selected region of the first latent image, the selected region being defined by a mask; and
generating updated textures of the first latent image based on the denoised latent textures.
5 . The method of claim 4 , wherein the generating the updated textures of the first latent image further comprises:
determining differences between the denoised latent textures and rendered latent textures by projecting the denoised latent textures onto the rendered latent textures in the first latent image; and
generating the updated textures of the first latent image based on the determined differences.
6 . The method of claim 5 , wherein the denoising further comprises:
determining a spherical harmonic coefficient for each of latent values of a plurality of latent pixels in the updated textures of the first latent image with weighted least squares; and
denoising the latent texture map based on a least square of weighted updated textures of the first latent image and a weighted view angle of the first view angle.
7 . The method of claim 4 , wherein the generating the textures for the 3D mesh further comprises:
determining a weighted average texture of the updated textures of the plurality of latent images in the denoised latent texture map;
determining a gradient that indicates a minimum difference between the weighted average texture and the updated textures of each of the plurality of latent images; and
optimizing the updated textures of the plurality of latent images based on the determined gradient to generate optimized textures of the plurality of latent images.
8 . The method of claim 7 , wherein the determining the weighted average texture further comprises:
decoding the updated textures of the plurality of latent images into the pixel space to generate a plurality of pixel images;
determining a difference between each pair of shared regions in the plurality of pixel images; and
determining the weighted average texture as an average of weighted differences between the pairs of shared regions in the plurality of the pixel images.
9 . The method of claim 7 , wherein the generating the textures for the 3D mesh further comprises:
decoding the optimized textures of the plurality of latent images into the pixel spaces to generate a plurality of RGB images in the pixel space, the plurality of RGB images including textures, a number of samples in the plurality of RGB images being greater than a number of samples in the plurality of latent images.
10 . The method of claim 9 , wherein the generating the textures for the 3D mesh further comprises:
projecting the plurality of RGB images onto the 3D mesh to generate an RGB texture map of the 3D mesh;
determining differences between the textures of the plurality of RGB images and rendered textures of the 3D mesh that are rendered for the plurality of view angles; and
updating the textures of the plurality of RGB images in the RGB texture map to generate the textures of the 3D mesh based on the differences between the textures of the plurality of RGB images and the rendered textures of the 3D mesh.
11 . An apparatus for generating textures for a three-dimensional (3D) mesh, the apparatus comprising:
processing circuitry configured to:
receive the 3D mesh that includes a plurality of vertices and a plurality of faces formed based on the plurality of vertices;
generate a latent texture map based on a plurality of latent images of the 3D mesh in a latent space from a plurality of view angles;
denoise the latent texture map to remove noise based on a diffusion process; and
generate the textures for the 3D mesh in a pixel space based on the denoised latent texture map.
12 . The apparatus of claim 11 , wherein the processing circuitry is configured to:
generate the latent texture map based on rendered textures of the 3D mesh in the pixel space for the plurality of view angles, the latent texture map including latent textures of the 3D mesh and a plurality of independent identically distributed (I. I. D.) Gaussian noises, each of a plurality of latent pixels in the latent texture map including a respective latent value; and
determine a spherical harmonic coefficient for each of the latent values of the plurality of latent pixels in the latent texture map based on the view angle associated with the respective latent value.
13 . The apparatus of claim 12 , wherein a selected region of each of the plurality of latent images of the latent texture map includes a same set of the I. I. D. Gaussian noises.
14 . The apparatus of claim 11 , wherein the processing circuitry is configured to:
render latent textures into a first latent image of the plurality of latent images of the latent texture map, the first latent image being generated for a first view angle;
denoise the latent textures of the first latent image in a selected region of the first latent image, the selected region being defined by a mask; and
generate updated textures of the first latent image based on the denoised latent textures.
15 . The apparatus of claim 14 , wherein the processing circuitry is configured to:
determine differences between the denoised latent textures and rendered latent textures by projecting the denoised latent textures onto the rendered latent textures in the first latent image; and
generate the updated textures of the first latent image based on the determined differences.
16 . The apparatus of claim 15 , wherein the processing circuitry is configured to:
determine a spherical harmonic coefficient for each of latent values of a plurality of latent pixels in the updated textures of the first latent image with weighted least squares; and
denoise the latent texture map based on a least square of weighted updated textures of the first latent image and a weighted view angle of the first view angle.
17 . The apparatus of claim 14 , wherein the processing circuitry is configured to:
determine a weighted average texture of the updated textures of the plurality of latent images in the denoised latent texture map;
determine a gradient that indicates a minimum difference between the weighted average texture and the updated textures of each of the plurality of latent images; and
optimize the updated textures of the plurality of latent images based on the determined gradient to generate optimized textures of the plurality of latent images.
18 . The apparatus of claim 17 , wherein the processing circuitry is configured to:
decode the updated textures of the plurality of latent images into the pixel space to generate a plurality of pixel images;
determine a difference between each pair of shared regions in the plurality of pixel images; and
determine the weighted average texture as an average of weighted differences between the pairs of shared regions in the plurality of the pixel images.
19 . The apparatus of claim 17 , wherein the processing circuitry is configured to:
decode the optimized textures of the plurality of latent images into the pixel spaces to generate a plurality of RGB images in the pixel space, the plurality of RGB images including textures, a number of samples in the plurality of RGB images being greater than a number of samples in the plurality of latent images.
20 . A non-transitory computer readable storage medium storing instructions which when executed by at least one processor cause the at least one processor to perform:
receiving a three-dimensional (3D) mesh that includes a plurality of vertices and a plurality of faces formed based on the plurality of vertices;
generating a latent texture map based on a plurality of latent images of the 3D mesh in a latent space from a plurality of view angles;
denoising the latent texture map to remove noise based on a diffusion process; and
generating textures for the 3D mesh in a pixel space based on the denoised latent texture map.