System and method for estimating depth uncertainty for self-supervised 3D reconstruction

1. A method three-dimensional (3D) scene reconstruction by an agent, comprising:

estimating an ego-motion of the agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image, and the ego-motion comprising a six degrees of freedom transformation between the current image and the previous image;

estimating a depth of the current image via a depth estimation model, the depth estimation model previously trained to account for a depth uncertainty measurement in the current image, the depth estimation model comprising a plurality of encoder layers and a plurality of decoder layers, each decoder layer generating an intermediate depth map, each intermediate depth map being upsampled and then concatenated with both activations of a corresponding encoder layer and feature maps unpacked by a corresponding decoder layer, and the activations being transmitted via a skip connection from the corresponding encoder layer;

generating a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth; and

controlling an action of the agent based on the three-dimensional reconstruction.

2. The method of claim 1 , further comprising training the depth estimation model to account or the uncertainty measurement by one or more of:

masking out a photometric loss;

actively sampling images from a sequence of training images to focus on images with the high uncertainty; or

enforcing smoothness-sharpness constraints in different areas of the current image and the sequence of images.

3. The method of claim 2 , in which masking out the photometric loss comprises:

determining a variance of each pixel in a training image; and

masking each pixel in the training image with the variance greater than a variance threshold.

4. The method of claim 2 , in which actively sampling the images comprises:

determining a global uncertainty for each image in the sequence of training images;

increasing a sampling probability for each image with the global uncertainty that is greater than a global uncertainty threshold, and

the method further comprises sampling, during the training, images from the sequence of training images based on the sampling probability.

5. The method of claim 2 , in which enforcing smoothness-sharpness constraints comprises constraining a smoothness value for each pixel in a training image and constraining a sharpness value for each pixel in the training image.

6. The method of claim 1 , in which the uncertainty measurement comprises aleatoric or epistemic uncertainty.

7. The method of claim 1 , further comprising capturing the sequence of images via a monocular camera defined on the agent.

8. An apparatus for three-dimensional (3D) scene reconstruction at an agent, comprising:

a processor;

a memory coupled with the processor; and

instructions stored in the memory and operable, when executed by the processor, to cause the apparatus:

to estimate an ego-motion of an agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image, and the ego-motion comprising a six degrees of freedom transformation between the current image and the previous image;

to estimate a depth of the current image via a depth estimation model, the depth estimation model previously trained to account for a depth uncertainty measurement in the current image, the depth estimation model generating an intermediate depth map at each decoder layer of a plurality of decoder layers, each intermediate depth map being upsampled and then concatenated with both activations of a corresponding encoder layer and feature maps unpacked by a corresponding decoder layer, and the activations being transmitted via a skip connection from the corresponding encoder layer;

to generate a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth; and

to control an action of the agent based on the three-dimensional reconstruction.

9. The apparatus of claim 8 , in which execution of the instructions further cause the apparatus to train the depth estimation model to account for the uncertainty measurement by one or more of:

masking out a photometric loss;

actively sampling images from a sequence of training images to focus on images with the high uncertainty; or

enforcing smoothness-sharpness constraints in different areas of the current image and the sequence of images.

10. The apparatus of claim 9 , in which masking out the photometric loss comprises: determining a variance of each pixel in a training image; and masking each pixel in the training image with the variance greater than a variance threshold.

11. The apparatus of claim 9 , in which actively sampling the images comprises:

determining a global uncertainty for each image in the sequence of training images;

increasing a sampling probability for each image with the global uncertainty that is greater than a global uncertainty threshold, and

the apparatus further comprises sampling, during the training, images from the sequence of training images based on the sampling probability.

12. The apparatus of claim 9 , in which enforcing smoothness-sharpness constraints comprises constraining a smoothness value for each pixel in a training image and constraining a sharpness value for each pixel in the training image.

13. The apparatus of claim 8 , in which the uncertainty measurement comprises aleatoric or epistemic uncertainty.

14. The apparatus of claim 8 , in which execution of the instructions further cause the apparatus to capture the sequence of images via a monocular camera defined on the agent.

15. A non-transitory computer-readable medium having program code recorded thereon for three-dimensional (3D) scene reconstruction at an agent the program code executed by a processor and comprising:

program code to estimate an ego-motion of an agent based on a current image from a sequence of images and a previous image from the sequence of images, each image in the sequence of images being a two-dimensional (2D) image, and the ego-motion comprising a six degrees of freedom transformation between the current image and the previous image;

program code to estimate a depth of the current image via a depth estimation model, the depth estimation model previously trained to account for a depth uncertainty measurement in the current image, the depth estimation model generating an intermediate depth map at each decoder layer of a plurality of decoder layers, each intermediate depth map being upsampled and then concatenated with both activations of a corresponding encoder layer and feature maps unpacked by a corresponding decoder layer, and the activations being transmitted via a skip connection from the corresponding encoder layer;

program code to generate a 3D reconstruction of the current image based on the estimated ego-motion and the estimated depth; and

program code to control an action of the agent based on the three-dimensional reconstruction.

16. The non-transitory computer-readable medium of claim 15 , further comprising program code to train the depth estimation model to account for the uncertainty measurement, the program code to train the neural network comprising one or more of:

program code to mask out a photometric loss;

program code to actively sample images from a sequence of training images to focus on images with the high uncertainty; or

program code to enforce smoothness-sharpness constraints in different areas of the current image and the sequence of images.

17. The non-transitory computer-readable medium of claim 16 , in which the program code to mask out the photometric loss comprises:

program code to determine a variance of each pixel in a training image; and

program code to mask each pixel in the training image with the variance greater than a variance threshold.

18. The non-transitory computer-readable medium of claim 16 , in which the program code to actively sample the images comprises:

program code to determine a global uncertainty for each image in the sequence of training images;

program code to increase a sampling probability for each image with the global uncertainty that is greater than a global uncertainty threshold, and

the program code to train further comprises program code to sample, during the training, images from the sequence of training images based on the sampling probability.

19. The non-transitory computer-readable medium of claim 16 , in which the program code to enforce the smoothness-sharpness constraints comprises program code to constrain a smoothness value for each pixel in a training image and program code to constrain a sharpness value for each pixel in the training image.

20. The non-transitory computer-readable medium of claim 15 , in which the uncertainty measurement comprises aleatoric or epistemic uncertainty.