Network architecture for the joint learning of monocular depth prediction and completion
System, methods, and other embodiments described herein relate to determining depths of a scene from a monocular image. In one embodiment, a method includes generating depth features from sensor data according to whether the sensor data includes sparse depth data. The method includes selectively injecting the depth features into a depth model. The method includes generating a depth map from at least a monocular image using the depth model that is guided by the depth features when injected. The method includes providing the depth map as depth estimates of objects represented in the monocular image.
1. A depth system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a network module including instructions that, when executed by the one or more processors, cause the one or more processors to:
generate depth features from sensor data according to whether the sensor data includes sparse depth data, including encoding the sparse depth data from a range sensor to form the depth features,
selectively inject the depth features into a depth model via skip connections of the depth model when the depth features are available according to a presence of the sparse depth data in the sensor data,
generate a depth map from at least a monocular image using the depth model that is guided by the depth features representing explicit depths from the range sensor when injected, and
provide the depth map as depth estimates of objects represented in the monocular image.
2. The depth system of claim 1 , wherein the network module includes instructions to generate the depth features including instructions to use a sparse auxiliary network that is a convolutional encoder to generate the depth features from the sparse depth data during inference, and wherein the sparse depth data is part of the sensor data that is acquired from the range sensor.
3. The depth system of claim 1 , wherein the network module includes instructions to selectively inject the depth features including instructions to, in response to determining that the sensor data includes the sparse depth data, inject the depth features into the depth model by concatenating the depth features with image features from an encoder of the depth model and provide concatenated features into a decoder of the depth model.
4. The depth system of claim 3 , wherein the network module includes instructions to inject the depth features including instructions to apply learned weights to the depth features and the image features prior to concatenating via the skip connections of the depth model.
5. The depth system of claim 1 , wherein the network module includes instructions to generate the depth map including instructions to apply the depth model to the monocular image by using an encoder of the depth model to encode image features and to use a decoder of the depth model to decode the depth features into the depth map, and
wherein the network module includes instructions to decode the image features at separate spatial resolutions as provided by the skip connections between the encoder and the decoder in combination with an output of a previous layer of the decoder.
6. The depth system of claim 1 , wherein the network module includes instructions to acquire the sensor data including at least the monocular image from at least one sensor of a device,
wherein the network module includes instructions to generate the depth features including instructions to determine whether the sensor data includes sparse depth data in addition to the monocular image and activating a sparse auxiliary network to generate the depth features from the sparse depth data when the sparse depth data is present.
7. The depth system of claim 1 , wherein providing the depth map includes controlling a device to navigate through a surrounding environment according to the depth map that identifies distances to objects in the surrounding environment.
8. The depth system of claim 1 , wherein the depth system is integrated within a device for autonomously controlling a vehicle.
9. A non-transitory computer-readable medium including instructions that, when executed by one or more processors cause the one or more processors to:
generate depth features from sensor data according to whether the sensor data includes sparse depth data, including encoding the sparse depth data from a range sensor to form the depth features,
selectively inject the depth features into a depth model via skip connections of the depth model when the depth features are available according to a presence of the sparse depth data in the sensor data,
generate a depth map from at least a monocular image using the depth model that is guided by the depth features representing explicit depths from the range sensor when injected, and
provide the depth map as depth estimates of objects represented in the monocular image.
10. The non-transitory computer-readable medium of claim 9 , wherein the instructions to generate the depth features include instructions to use a sparse auxiliary network that is a convolutional encoder to generate the depth features from the sparse depth data, and wherein the sparse depth data is part of the sensor data that is acquired from a range sensor.
11. The non-transitory computer-readable medium of claim 9 , wherein the instructions to selectively inject the depth features include instructions to, in response to determining that the sensor data includes the sparse depth data, inject the depth features into the depth model by concatenating the depth features with image features from an encoder of the depth model and provide concatenated features into a decoder of the depth model.
12. The non-transitory computer-readable medium of claim 11 , wherein the instructions to inject the depth features include instructions to apply learned weights to the depth features and the image features prior to concatenating via the skip connections of the depth model.
13. The non-transitory computer-readable medium of claim 9 , wherein instructions to generate the depth map include instructions to apply the depth model to the monocular image by using an encoder of the depth model to encode image features and to use a decoder of the depth model to decode the depth features into the depth map, and
wherein the instructions to decode the image features at separate spatial resolutions as provided by the skip connections between the encoder and the decoder in combination with an output of a previous layer of the decoder.
14. A method, comprising:
generating depth features from sensor data according to whether the sensor data includes sparse depth data, including encoding the sparse depth data from a range sensor to form the depth features;
selectively injecting the depth features into a depth model via skip connections of the depth model when the depth features are available according to a presence of the sparse depth data in the sensor data;
generating a depth map from at least a monocular image using the depth model that is guided by the depth features representing explicit depths from the range sensor when injected; and
providing the depth map as depth estimates of objects represented in the monocular image.
15. The method of claim 14 , wherein generating the depth features includes using a sparse auxiliary network that is a convolutional encoder to generate the depth features from the sparse depth data during inference, and wherein the sparse depth data is part of the sensor data that is acquired from a range sensor.
16. The method of claim 14 , wherein selectively injecting the depth features includes, in response to determining that the sensor data includes the sparse depth data, injecting the depth features into the depth model by concatenating the depth features with image features from an encoder of the depth model and providing concatenated features into a decoder of the depth model.
17. The method of claim 16 , wherein injecting the depth features includes applying learned weights to the depth features and the image features prior to concatenating via the skip connections of the depth model.
18. The method of claim 14 , wherein generating the depth map includes applying the depth model to the monocular image by using an encoder of the depth model to encode image features and to use a decoder of the depth model to decode the depth features into the depth map, and wherein using the decoder includes decoding the image features at separate spatial resolutions as provided by the skip connections between the encoder and the decoder in combination with an output of a previous layer of the decoder.
19. The method of claim 14 , further comprising:
acquiring the sensor data including at least the monocular image from at least one sensor of a device, wherein generating the depth features includes determining whether the sensor data includes sparse depth data in addition to the monocular image and activating a sparse auxiliary network to generate the depth features from the sparse depth data when the sparse depth data is present.
20. The method of claim 14 , wherein providing the depth map includes controlling a device to navigate through a surrounding environment according to the depth map that identifies distances to objects in the surrounding environment.