System and method for monocular depth estimation from semantic information
A method for controlling an ego agent includes capturing a two-dimensional (2D) image of an environment adjacent to the ego agent. The method also includes generating a semantically segmented image of the environment based on the 2D image. The method further includes generating a depth map of the environment based on the semantically segmented image. The method additionally includes generating a three-dimensional (3D) estimate of the environment based on the depth map. The method also includes controlling an action of the ego agent based on the identified location.
1. A method for controlling an ego agent comprising:
capturing a first two-dimensional (2D) image associated with a current view of an environment corresponding to a current location of the ego agent;
generating, via a semantic encoder, a semantically segmented image of the first 2D image;
generating, via a depth estimation network, a depth map of the current view of the environment based on the semantically segmented image, the depth estimation network being trained via a disposable two-stream network including a red-green-blue (RGB) encoder, a semantic encoder, and a depth decoder shared between the RGB encoder and the semantic encoder, the depth estimation network being limited to the semantic encoder and the depth decoder during inference;
generating a three-dimensional (3D) reconstruction of the first 2D image based on the depth map and a current pose of the ego agent, the current pose being based on the first 2D image and the second 2D image, the second 2D image associated with a view of the environment that is different from the current view; and
controlling an action of the ego agent based on the 3D estimate of the environment.
2. The method of claim 1 , in which, during training:
the RGB encoder encodes a group of RGB training images;
the semantic encoder encodes a group of semantically segmented training images, and
respective images from the group of RGB training images and the group of semantically segmented training images are paired inputs to the disposable two-stream network.
3. The method of claim 2 , in which:
the depth decoder generates the depth map; and
the RGB encoder and the semantic encoder share a latent feature space.
4. The method of claim 2 , in which one or more semantically segmented training images of the group of semantically segmented training images are virtual images.
5. The method of claim 1 , in which the ego agent is an autonomous vehicle.
6. The method of claim 1 , further comprising capturing the first 2D image and the second 2D image via a monocular camera integrated with the ego agent.
7. An apparatus for controlling an ego agent, comprising:
a processor;
a memory coupled with the processor; and
instructions stored in the memory and operable, when executed by the processor, to cause the apparatus:
to capture a first two-dimensional (2D) image associated with a current view of an environment corresponding to a current location of the ego agent;
to generate, via a semantic encoder, a semantically segmented image of the first 2D image;
to generate, via a depth estimation network, a depth map of the current view of the environment based on the semantically segmented image, the depth estimation network being trained via a disposable two-stream network including a red-green-blue (RGB) encoder, a semantic encoder, and a depth decoder shared between the RGB encoder and the semantic encoder, the depth estimation network being limited to the semantic encoder and the depth decoder during inference;
to generate a three-dimensional (3D) reconstruction of the first 2D image based on the depth map and a current pose of the ego agent, the current pose being based on the first 2D image and the second 2D image, the second 2D image associated with a view of the environment that is different from the current view; and
to control an action of the ego agent based on the 3D estimate of the environment.
8. The apparatus of claim 7 , in which, during training:
the RGB encoder encodes a group of RGB training images;
the semantic encoder encodes a group of semantically segmented training images, and
respective images from the group of RGB training images and the group of semantically segmented training images are paired inputs to the disposable two-stream network.
9. The apparatus of claim 8 , in which:
the depth decoder generates the depth map; and
the RGB encoder and the semantic encoder share a latent feature space.
10. The apparatus of claim 8 , in which one or more semantically segmented training images of the group of semantically segmented training images are virtual images.
11. The apparatus of claim 7 , in which the ego agent is an autonomous vehicle.
12. The apparatus of claim 7 , in which the instructions further cause the apparatus to capture the first 2D image and the second 2D image via a monocular camera integrated with the ego agent.
13. A non-transitory computer-readable medium having program code recorded thereon for controlling an ego agent, the program code executed by a processor and comprising:
program code to capture a first two-dimensional (2D) image associated with a current view of an environment corresponding to a current location of the ego agent;
program code to generate, via a semantic encoder, a semantically segmented image of the first 2D image;
program code to generate, via a depth estimation network, a depth map of the current view of the environment based on the semantically segmented image, the depth estimation network being trained via a disposable two-stream network including a red-green-blue (RGB) encoder, a semantic encoder, and a depth decoder shared between the RGB encoder and the semantic encoder, the depth estimation network being limited to the semantic encoder and the depth decoder during inference;
program code to generate a three-dimensional (3D) reconstruction of the first 2D image based on the depth map and a current pose of the ego agent, the current pose being based on the first 2D image and the second 2D image, the second 2D image associated with a view of the environment that is different from the current view; and
program code to control an action of the ego agent based on the 3D estimate of the environment.
14. The non-transitory computer-readable medium of claim 13 ,
in
which, during training:
the RGB encoder encodes a group of RGB training images;
the semantic encoder encodes a group of semantically segmented training images, and
respective images from the group of RGB training images and the group of semantically segmented training images are paired inputs to the disposable two-stream network.
15. The non-transitory computer-readable medium of claim 14 , in which:
the depth decoder generates the depth map; and
the RGB encoder and the semantic encoder share a latent feature space.
16. The non-transitory computer-readable medium of claim 14 , in which one or more semantically segmented training images of the group of semantically segmented training images are virtual images.
17. The non-transitory computer-readable medium of claim 13 , in which:
the ego agent is an autonomous vehicle; and
the program code further comprises program code to capture the first 2D image and the second 2D image via a monocular camera integrated with the ego agent.