Radiant and volumetric latent space encoding for volumetric rendering

1 . A method comprising:

generating, through training, a shared latent space based on (i) image data that include multiple images, wherein each image has a different viewing frame of a scene, and (ii) first and second types of embeddings, by:

for a first set of training epochs, training the shared latent space using only the first type of embeddings;

for a second set of training epochs after the first set, training the shared latent space using the first and second types of embeddings; and

for a third set of training epochs after the second set, training the shared latent space using only the second type of embeddings;

training a decoder based on the first type of embeddings;

generating an embedding based on the first type of embeddings that is representative of a novel viewing frame of the scene;

decoding, with the decoder, the shared latent space using cross-attention with the generated embedding; and

generating the novel viewing frame of the scene based on an output of the decoder.

2 . The method of claim 1 , wherein the first type of embeddings is a ray embedding and the second type of embeddings is a volumetric embedding.

3 . The method of claim 2 , wherein the novel viewing frame generated using the embedding is a predicted depth map of the scene.

4 . The method of claim 2 , wherein the novel viewing frame generated using the embedding is an image of the scene.

5 . The method of claim 1 , further comprising training a second decoder to decode the shared latent space based on the second type of embeddings.

6 . The method of claim 5 , further comprising:

decoding, with the second decoder, the shared latent space using cross-attention with a second embedding;

generating a second version of the novel viewing frame of the scene based on an output of the second decoder, wherein second version of the novel viewing frame is a predicted bitmap of the scene.

7 . A system comprising:

a preprocessing platform, comprising at least one processor and memory, configured to:

generate, through training, a shared latent space based on (i) image data that include multiple images, wherein each image has a different viewing frame of a scene, and (ii) first and second types of embeddings, by:

for a first set of training epochs, training the shared latent space using only the first type of embeddings;

for a second set of training epochs after the first set, training the shared latent space using the first and second types of embeddings; and

for a third set of training epochs after the second set, training the shared latent space using only the second type of embeddings;

train a decoder based on the first type of embeddings; and

a computer vision platform configured to:

generate an embedding based on the first type of embeddings that is representative of a novel viewing frame of the scene;

decode, with the decoder, the shared latent space using cross-attention with the generated embedding; and

generate the novel viewing frame of the scene based on an output of the decoder.

8 . The system of claim 7 , wherein the first type of embeddings is a ray embedding and the second type of embeddings is a volumetric embedding.

9 . The system of claim 8 , wherein the novel viewing frame generated using the embedding is a predicted depth map of the scene.

10 . The system of claim 8 , wherein the novel viewing frame generated using the embedding is an image of the scene.

11 . The system of claim 7 , wherein the preprocessing platform is further configured to train a second decoder to decode the shared latent space based on the second type of embeddings.

12 . The system of claim 11 , wherein the computer vision platform is further configured to:

decode, with the second decoder, the shared latent space using cross-attention with a second embedding;

generate a second version of the novel viewing frame of the scene based on an output of the second decoder, wherein second version of the novel viewing frame is a predicted bitmap of the scene.

13 . A tangible computer readable medium comprising instructions that, when executed, cause a system to:

generate, through training, a shared latent space based on (i) image data that include multiple images, wherein each image has a different viewing frame of a scene, and (ii) first and second types of embeddings, by:

for a first set of training epochs, training the shared latent space using only the first type of embeddings;

for a second set of training epochs after the first set, training the shared latent space using the first and second types of embeddings; and

for a third set of training epochs after the second set, training the shared latent space using only the second type of embeddings;

train a decoder based on the first type of embeddings;

generate an embedding based on the first type of embeddings that is representative of a novel viewing frame of the scene;

decode, with the decoder, the shared latent space using cross-attention with the generated embedding; and

generate the novel viewing frame of the scene based on an output of the decoder.

14 . The computer readable medium of claim 13 , wherein the first type of embeddings is a ray embedding and the second type of embeddings is a volumetric embedding.

15 . The computer readable medium of claim 14 , wherein the novel viewing frame generated using the embedding is a predicted depth map of the scene.

16 . The computer readable medium of claim 14 , wherein the novel viewing frame generated using the embedding is an image of the scene.

17 . The computer readable medium of claim 13 , wherein the instructions, when executed, cause the system to:

train a second decoder to decode the shared latent space based on the second type of embeddings;

decode, with the second decoder, the shared latent space using cross-attention with a second embedding;

generate a second version of the novel viewing frame of the scene based on an output of the second decoder, wherein second version of the novel viewing frame is a predicted bitmap of the scene.