System and method for estimating depth uncertainty for self-supervised 3D reconstruction

1. A method for three-dimensional (3D) scene reconstruction by an agent, comprising:

estimating an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image;

estimating a per-pixel depth of the current image via a depth estimation model, the depth estimation model including a plurality of encoder layers and a plurality of decoder layers;

generating a 3D reconstruction of the current image based on the estimated ego-motion and the estimated per-pixel depth; and

controlling an action of the agent based on the 3D reconstruction.

2. The method of claim 1 , further comprising training the depth estimation model to account for an uncertainty measurement.

3. The method of claim 2 , wherein the uncertainty measurement comprises aleatoric or epistemic uncertainty.

4. The method of claim 1 , further comprising capturing the sequence of images via a monocular camera defined on the agent.

5. The method of claim 1 , wherein the estimated ego-motion comprising a six degrees of freedom transformation associated with a transformation from the previous image to the current image and the previous image.

6. The method of claim 1 , wherein the estimated per-pixel depth includes spatial information associated with the current image.

7. The method of claim 1 , wherein:

the depth estimation model generates an intermediate depth map at each decoder layer of the plurality of decoder layers;

each intermediate depth map is upsampled and then concatenated with activations of a corresponding encoder layer and feature maps unpacked by a corresponding decoder layer; and

the activations are transmitted via a skip connection from the corresponding encoder layer.

8. An apparatus for three-dimensional (3D) scene reconstruction at an agent, comprising:

a processor; and

a memory coupled with the processor and storing instructions operable, when executed by the processor, to cause the apparatus to:

estimate an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image;

estimate a per-pixel depth of the current image via a depth estimation model, the depth estimation model including a plurality of encoder layers and a plurality of decoder layers;

generate a 3D reconstruction of the current image based on the estimated ego-motion and the estimated per-pixel depth; and

control an action of the agent based on the 3D reconstruction.

9. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to train the depth estimation model to account for an uncertainty measurement.

10. The apparatus of claim 9 , wherein the uncertainty measurement comprises aleatoric or epistemic uncertainty.

11. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to capture the sequence of images via a monocular camera defined on the agent.

12. The apparatus of claim 8 , wherein the estimated ego-motion comprising a six degrees of freedom transformation associated with a transformation from the previous image to the current image and the previous image.

13. The apparatus of claim 8 , wherein the estimated per-pixel depth includes spatial information associated with the current image.

14. The apparatus of claim 8 , wherein:

the depth estimation model generates an intermediate depth map at each decoder layer of the plurality of decoder layers;

each intermediate depth map is upsampled and then concatenated with activations of a corresponding encoder layer and feature maps unpacked by a corresponding decoder layer; and

the activations are transmitted via a skip connection from the corresponding encoder layer.

15. A non-transitory computer-readable medium having program code recorded thereon for three-dimensional (3D) scene reconstruction at an agent the program code executed by a processor and comprising:

program code to estimate an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image;

program code to estimate a per-pixel depth of the current image via a depth estimation model, the depth estimation model including a plurality of encoder layers and a plurality of decoder layers;

program code to generate a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth; and

program code to control an action of the agent based on the 3D reconstruction.

16. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to train the depth estimation model to account for an uncertainty measurement.

17. The non-transitory computer-readable medium of claim 16 , wherein the uncertainty measurement comprises aleatoric or epistemic uncertainty.

18. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to capture the sequence of images via a monocular camera defined on the agent.

19. The non-transitory computer-readable medium of claim 15 , wherein the estimated ego-motion comprising a six degrees of freedom transformation associated with a transformation from the previous image to the current image and the previous image.

20. The non-transitory computer-readable medium of claim 15 , wherein the estimated per-pixel depth includes spatial information associated with the current image.