Monocular depth supervision from 3D bounding boxes

1. A method for navigating a vehicle through an environment, comprising:

identifying a first location of a dynamic object in a two-dimensional (2D) image of the environment based on identifying a prior location of the dynamic object in a first three-dimensional (3D) estimate of the environment, the 2D image comprising the dynamic object and a static object;

assigning a first weight to each pixel in the 2D image associated the dynamic object based on identifying the first location of the dynamic object;

assigning a second weight to each pixel in the 2D image associated with the static object, the first weight being greater than the second weight;

generating, via a depth estimation network, a depth map of the environment based on the 2D image, the depth map comprising:

a dynamic object depth estimate for the dynamic object, the dynamic object depth estimate being associated with a first accuracy that is based on the first weight; and

a static object depth estimate for the static object, the static object depth estimate being associated with a second accuracy that is based on the second weight,

the first accuracy of the dynamic object depth-estimate being greater than the second accuracy of the static object depth estimate;

generating a second 3D estimate of the environment based on the depth map;

identifying a second location of the dynamic object in the second 3D estimate; and

controlling an action of the vehicle based on identifying the second location.

2. The method of claim 1 , in which:

the dynamic object comprises a pedestrian, a neighboring vehicle, or a cyclist; and

the static object comprises a road, a sidewalk, or a building.

3. The method of claim 1 , further comprising training the depth estimation network of the vehicle by:

adjusting the weight of each pixel associated with the dynamic object in the 2D image; and

training the depth estimation network based on ground-truth information and the adjusted weights.

4. The method of claim 3 , further comprising identifying the third location, during training, based on annotated ground truth information.

5. The method of claim 3 , further comprising:

identifying the second location of the dynamic object based on a 3D bounding box identifying the second location of the dynamic object in the 3D estimate;

converting the 3D bounding box to a 2D bounding box; and

identifying the first location of the dynamic object in the 2D image based on the 2D bounding box.

6. The method of claim 3 , further comprising:

adjusting the weight of each pixel associated with the dynamic object based on a first number of pixels comprising depth information and a number of pixels corresponding to the location of the dynamic object in the 2D image; or

adjusting the weight of each pixel associated with the dynamic object based on a photometric loss and a supervised depth error loss.

7. The method of claim 1 , further comprising capturing the 2D image via a monocular camera integrated with the vehicle.

8. An apparatus for navigating a vehicle through an environment, comprising:

a processor;

a memory coupled with the processor; and

instructions stored in the memory and operable, when executed by the processor, to cause the apparatus:

to identify a first location of a dynamic object in a two-dimensional (2D) image of the environment based on identifying a second location of the dynamic object in a first three-dimensional (3D) estimate of the environment, the 2D image comprising the dynamic object and a static object;

to assign a first weight to each pixel in the 2D image associated the dynamic object and a second weight to each pixel in the 2D image associated with the static object, the first weight being greater than the second weight based on identifying the second location

to generate, via a depth estimation network, a depth map of the environment based on the 2D image, the depth map comprising:

a dynamic object depth estimate for the dynamic object, the dynamic object depth estimate being associated with a first accuracy that is based on the first weight; and

a static object depth estimate for the static object, the static object depth estimate being associated with a second accuracy that is based on the second weight,

the first accuracy of the dynamic object depth-estimate being greater than the second accuracy of the static object depth estimate;

to generate a second 3D estimate of the environment based on the depth map;

to identify a third location of the dynamic object in the 3D estimate; and

to control an action of the vehicle based on identifying the third location.

9. The apparatus of claim 8 , in which:

the dynamic object comprises a pedestrian, a neighboring vehicle, or a cyclist; and

the static object comprises a road, a sidewalk, or a building.

10. The apparatus of claim 8 , in which execution of the instructions further cause the apparatus to train the depth estimation network of the vehicle by:

adjusting the weight of each pixel associated with the dynamic object in the 2D image; and

training the depth estimation network based on ground-truth information and the adjusted weights.

11. The apparatus of claim 10 , in which execution of the instructions further cause the apparatus to train the depth estimation network of the vehicle by identifying the location, during training, based on annotated ground truth information.

12. The apparatus of claim 10 , in which execution of the instructions further cause the apparatus to:

identify the second location of the dynamic object based on a 3D bounding box identifying the second location of the dynamic object in the 3D estimate;

convert the 3D bounding box to a 2D bounding box; and

identify the first location of the dynamic object in the 2D image based on the 2D bounding box.

13. The apparatus of claim 10 , in which the instructions further cause the apparatus to train the depth estimation network of the vehicle by:

adjusting the weight of each pixel associated with the dynamic object based on a first number of pixels comprising depth information and a number of pixels corresponding to the location of the dynamic object in the 2D image; or

adjusting the weight of each pixel associated with the dynamic object based on a photometric loss and a supervised depth error loss.

14. The apparatus of claim 8 , in which execution of the instructions further cause the apparatus to capture the 2D image via a monocular camera integrated with the vehicle.

15. A non-transitory computer-readable medium having program code recorded thereon for navigating a vehicle through an environment, the program code executed by a processor and comprising:

program code to capture identify a first location of a dynamic object in a two-dimensional (2D) image of the environment based on identifying a second location of the dynamic object in a first three-dimensional (3D) estimate of the environment, the 2D image comprising the dynamic object and a static object;

program code to assign a first weight to each pixel in the 2D image associated the dynamic object and a second weight to each pixel in the 2D image associated with the static object, the first weight being greater than the second weight based on identifying the second location;

program code to generate, via a depth estimation network, a depth map of the environment based on the 2D image, the depth map comprising:

a dynamic object depth estimate for the dynamic object, the dynamic object depth estimate being associated with a first accuracy that is based on the first weight; and

a static object depth estimate for the static object, the static object depth estimate being associated with a second accuracy that is based on the second weight,

the first accuracy of the dynamic object depth-estimate being greater than the second accuracy of the static object depth estimate;

program code to generate a second 3D estimate of the environment based on the depth map;

program code to identify a third location of the dynamic object in the second 3D; and

program code to control an action of the vehicle based on identifying the third.

16. The non-transitory computer-readable medium of claim 15 , in which:

the dynamic object comprises a pedestrian, a neighboring vehicle, or a cyclist; and

the static object comprises a road, a sidewalk, or a building.

17. The non-transitory computer-readable medium of claim 15 , in which the program code further comprises program code to train the depth estimation network of the vehicle by:

adjusting the weight of each pixel associated with the dynamic object in the 2D image; and

training the depth estimation network based on ground-truth information and the adjusted weights.

18. The non-transitory computer-readable medium of claim 17 , in which the program code further comprises program code to train the depth estimation network of the vehicle by identifying the location, during training, based on annotated ground truth information.

19. The non-transitory computer-readable medium of claim 17 , in which the program code further comprises:

program code to identify the second location of the dynamic object based on a 3D bounding box identifying the second location of the dynamic object in the 3D estimate;

program code to convert the 3D bounding box to a 2D bounding box; and

program code to identify the first location of the dynamic object in the 2D image based on the 2D bounding box.

20. The non-transitory computer-readable medium of claim 17 , in which the program code further comprises program code to train the depth estimation network of the vehicle by:

adjusting the weight of each pixel associated with the dynamic object based on a first number of pixels comprising depth information and a number of pixels corresponding to the location of the dynamic object in the 2D image; or

adjusting the weight of each pixel associated with the dynamic object based on a photometric loss and a supervised depth error loss.