Representing 3D shapes with probabilistic directed distance fields
The present disclosure provides methods, apparatuses, and computer-readable mediums for representing shapes with probabilistic directed distance fields. In some embodiments, a method includes obtaining a camera representation and a latent shape vector representation of a scene. The camera representation indicates position information and direction information of a view of the scene. The method further includes calculating, based on the latent shape vector representation of the scene, a visibility score and a depth for each ray of a plurality of rays emanating from a corresponding plurality of positions and directions. The plurality of positions and directions are determined from the camera representation of the scene. The method further includes generating renders of geometric information of the scene using the visibility score and the depth of the plurality of rays.
1. A method of representing shapes with probabilistic directed distance fields to be performed by a processor, comprising:
obtaining a camera representation and a latent shape vector representation of a scene, the camera representation indicating position information and direction information of a view of the scene;
calculating, based on the latent shape vector representation of the scene, a visibility score and a depth for each ray of a plurality of rays emanating from a corresponding plurality of positions and directions, the plurality of positions and directions being determined from the camera representation of the scene; and
generating renders of geometric information of the scene using the visibility score and the depth of the plurality of rays.
2. The method of claim 1 , further comprising:
receiving a plurality of queries requesting the visibility score and the depth for each ray of the plurality of rays emanating from the corresponding plurality of positions and directions,
wherein the calculating of the visibility score and the depth of the plurality of rays comprises calculating, in response to the receiving of a query of the plurality of queries, the visibility score and the depth of a ray of the plurality of rays corresponding to the corresponding position and direction indicated by the query.
3. The method of claim 1 , further comprising:
correcting depth information of the renders of the geometric information of the scene across at least one occlusion boundary, based on a switching mechanism over a set of estimated depth values.
4. The method of claim 1 , wherein the obtaining of the camera representation and the latent shape vector representation of the scene comprises:
encoding, using a neural encoder, an image comprising the scene.
5. The method of claim 1 , wherein the calculating of the visibility score and the depth of the plurality of rays comprises:
combining a plurality of shape representations of the scene; and
calculating the visibility score and the depth for each ray of the plurality of rays based on a combination of the plurality of shape representations of the scene.
6. The method of claim 1 , wherein the calculating of the visibility score and the depth of the plurality of rays comprises:
performing, for each ray of the plurality of rays, a single forward pass of a conditional coordinate neural network to calculate the visibility score and the depth of that ray.
7. The method of claim 1 , wherein the calculating of the visibility score and the depth of the plurality of rays comprises:
calculating a lowest distance for each ray of the plurality of rays intersecting the scene.
8. The method of claim 1 , wherein:
the visibility score indicates whether a corresponding ray intersects the scene; and
the depth indicates a distance from the corresponding position of the corresponding ray to a nearest intersection point of the corresponding ray with the scene.
9. The method of claim 1 , further comprising:
calculating, based on the latent shape vector representation of the scene, a reflectance value for each ray of the plurality of rays emanating from the corresponding plurality of positions and directions.
10. An apparatus for representing shapes with probabilistic directed distance fields, comprising:
a memory storage storing computer-executable instructions; and
a processor communicatively coupled to the memory storage, wherein the processor is configured to execute the computer-executable instructions and cause the apparatus to:
obtain a camera representation and a latent shape vector representation of a scene, the camera representation indicating position information and direction information of a view of the scene;
calculate, based on the latent shape vector representation of the scene, a visibility score and a depth for each ray of a plurality of rays emanating from a corresponding plurality of positions and directions, the plurality of positions and directions being determined from the camera representation of the scene; and
generate renders of geometric information of the scene using the visibility score and the depth of the plurality of rays.
11. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
receive a plurality of queries requesting the visibility score and the depth for each ray of the plurality of rays emanating from the corresponding plurality of positions and directions; and
calculate, in response to the receiving of a query of the plurality of queries, the visibility score and the depth of a ray of the plurality of rays corresponding to the corresponding position and direction indicated by the query.
12. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
correct depth information of the renders of the geometric information of the scene across at least one occlusion boundary, based on a switching mechanism over a set of estimated depth values.
13. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
encode, using a neural encoder, an image comprising the scene.
14. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
combine a plurality of shape representations of the scene; and
calculate the visibility score and the depth for each ray of the plurality of rays based on a combination of the plurality of shape representations of the scene.
15. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
perform, for each ray of the plurality of rays, a single forward pass of a conditional coordinate neural network to calculate the visibility score and the depth of that ray.
16. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
calculate a lowest distance for each ray of the plurality of rays intersecting the scene.
17. The apparatus of claim 10 , wherein:
the visibility score indicates whether a corresponding ray intersects the scene; and
the depth indicates a distance from the corresponding position of the corresponding ray to a nearest intersection point of the corresponding ray with the scene.
18. The apparatus of claim 10 , wherein the processor is further configured to execute further computer-executable instructions and further cause the apparatus to:
calculate, based on the latent shape vector representation of the scene, a reflectance value for each ray of the plurality of rays emanating from the corresponding plurality of positions and directions.
19. A non-transitory computer-readable storage medium storing computer-executable instructions for representing shapes with probabilistic directed distance fields by a device, the computer-executable instructions being configured, when executed by one or more processors of the device, to cause the device to:
obtain a camera representation and a latent shape vector representation of a scene, the camera representation indicating position information and direction information of a view of the scene;
calculate, based on the latent shape vector representation of the scene, a visibility score and a depth for each ray of a plurality of rays emanating from a corresponding plurality of positions and directions, the plurality of positions and directions being determined from the camera representation of the scene; and
generate renders of geometric information of the scene using the visibility score and the depth of the plurality of rays.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the computer-executable instructions further cause the device to:
receive a plurality of queries requesting the visibility score and the depth for each ray of the plurality of rays emanating from the corresponding plurality of positions and directions; and
calculate, in response to the receiving of a query of the plurality of queries, the visibility score and the depth of a ray of the plurality of rays corresponding to the corresponding position and direction indicated by the query.