Systems and methods for depth estimation using monocular images

1. A depth system for generating depth estimates of an environment depicted in a monocular image, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing:

a depth module including instructions that when executed by the one or more processors cause the one or more processors to, in response to receiving the monocular image, process the monocular image according to a depth model to generate a depth map by:

encoding the monocular image according to encoding layers of the depth model including iteratively encoding features of the monocular image to generate feature maps at successively refined representations using packing blocks within the encoding layers,

decoding the feature maps according to decoding layers of the depth model including iteratively decoding the features maps associated with separate ones of the packing blocks using unpacking blocks of the decoding layers to generate the depth map; and

an image module including instructions that when executed by the one or more processors cause the one or more processors to provide the depth map as the depth estimates of objects represented in the monocular image.

2. The depth system of claim 1 , wherein the depth module includes instructions to encode the features including instructions to fold spatial dimensions of the feature maps into additional channels according to the packing blocks to capture sub-pixel information across the encoding layers, and

wherein the depth module includes instructions to encode spatial information of the feature maps and aggregate the features in the feature maps according to the packing blocks.

3. The depth system of claim 2 , wherein the depth module includes instructions to encode the feature maps according to the packing blocks including instructions to add channels to an intermediate feature map of the feature maps according to a space-to-depth operation, a three-dimensional convolutional operation, a reshaping operation, and a two-dimensional convolutional operation to generate the intermediate feature map, and

wherein the depth module includes instructions to encode the feature maps according to the space-to-depth operation that adds the additional channels to capture the sub-pixel information.

4. The depth system of claim 1 , wherein the depth module includes instructions to decode the feature maps including instructions to relay the feature maps to the unpacking blocks using skip connections between the packing blocks and the unpacking blocks,

wherein the depth module includes instructions to decode the feature maps including instructions to unfold feature channels in the feature maps into spatial dimensions using the decoding layers including the unpacking blocks, and

wherein the depth module includes instructions to relay the feature maps including instructions to provide encoded features from separate ones of the encoding layers as residual information of high-level features in the monocular image to the decoding layers.

5. The depth system of claim 4 , wherein the depth module includes instructions to decode according to the unpacking blocks including instructions to, at the separate unpacking blocks, applying a two-dimensional convolutional operation, a three-dimensional convolutional operation, a reshaping operation, and a depth-to-space operation that applies sub-pixel convolutions to up-sample the spatial dimensions and super-resolve spatial information into the depth map, and

wherein the depth module includes instructions to decode the feature maps including instructions to generate intermediate depth maps that provide inverse depths and that are inputs to subsequent ones of the decoding layers.

6. The depth system of claim 1 , wherein the depth module includes instructions to generate the depth map including instructions to generate a set of depth maps at multiple different scales,

wherein the set of depth maps are inverse depth maps separately derived from the decoding layers, and

wherein the image module includes instructions to provide the depth map including instructions to provide the depth map to one or more vehicle systems within a vehicle to facilitate automated functions of the vehicle.

7. The depth system of claim 1 , wherein the depth model is a machine learning algorithm comprised of the encoder layers and the decoder layers, the encoding layers including convolutional layers, the packing blocks, and residual blocks, and the decoding layers including the unpacking blocks, convolutional layers, and output layers for providing a set of depth maps.

8. The depth system of claim 1 , wherein the image module includes instructions to train the depth model using a training algorithm that is self-supervised and accounts for motion of a camera between successive monocular training images in training data using a loss function that accounts for structure from motion (SfM), and

wherein the image module includes instructions to train the depth model including instructions to execute a pose model to account for the motion, and to adjust parameters in the depth model including at least sub-pixel convolutions in the decoding layers and encoding hyper-parameters in the encoding layers.

9. A non-transitory computer-readable medium for generating depth estimates of an environment depicted in a monocular image and including instructions that when executed by one or more processors cause the one or more processors to:

process the monocular image according to a depth model to generate a depth map by:

encoding the monocular image according to encoding layers of the depth model including iteratively encoding features of the monocular image to generate feature maps at successively refined representations using packing blocks within the encoding layers, and

decoding the feature maps according to decoding layers of the depth model including iteratively decoding the features maps associated with separate ones of the packing blocks using unpacking blocks of the decoding layers to generate the depth map; and

provide the depth map as the depth estimates of objects represented in the monocular image.

10. The non-transitory computer-readable medium of claim 9 , wherein the instructions to encode the features include instructions to fold spatial dimensions of the feature maps into additional channels according to at least a space-to-depth operation of the packing blocks to capture sub-pixel information across the encoding layers, and

wherein the instructions include instructions to encode spatial information of the feature maps and aggregate the features in the feature maps according to the packing blocks.

11. The non-transitory computer-readable medium of claim 9 , wherein the instructions to decode the feature maps include instructions to relay the feature maps to the unpacking blocks using skip connections between the packing blocks and the unpacking blocks,

wherein the instructions to decode the feature maps include instructions to unfold feature channels in the feature maps into spatial dimensions using the decoding layers including at least a depth-to-space operation of the unpacking blocks.

12. The non-transitory computer-readable medium of claim 9 , wherein the instructions to generate the depth map include instructions to generate a set of depth maps at multiple different scales,

wherein the set of depth maps are inverse depth maps separately derived from the decoding layers, and

wherein the instructions to provide the depth map include instructions to provide the depth map to one or more vehicle systems within a vehicle to facilitate automated functions of the vehicle.

13. The non-transitory computer-readable medium of claim 9 , wherein the depth model is a machine learning algorithm comprised of the encoder layers and the decoder layers, the encoding layers including convolutional layers, the packing blocks, and residual blocks, and the decoding layers including the unpacking blocks, convolutional layers, and output layers for providing a set of depth maps.

14. A method of generating depth estimates of an environment depicted in a monocular image, comprising:

in response to receiving the monocular image, processing the monocular image according to a depth model to generate a depth map by:

encoding the monocular image according to encoding layers of the depth model including iteratively encoding features of the monocular image to generate feature maps at successively refined representations using packing blocks within the encoding layers,

decoding the feature maps according to decoding layers of the depth model including iteratively decoding the features maps associated with separate ones of the packing blocks using unpacking blocks of the decoding layers to generate the depth map; and

providing the depth map as the depth estimates of objects represented in the monocular image.

15. The method of claim 14 , wherein encoding the features includes folding spatial dimensions of the feature maps into additional channels according to the packing blocks to capture sub-pixel information across the encoding layers, and

wherein encoding includes aggregating the features in the feature maps according to the packing blocks.

16. The method of claim 15 , wherein encoding the feature maps according to the packing blocks includes adding channels to an intermediate feature map of the feature maps according to a space-to-depth operation, a three-dimensional convolutional operation, a reshaping operation, and a two-dimensional convolutional operation to generate the intermediate feature map, and

wherein encoding the feature maps according to the space-to-depth operation adds the additional channels to capture the sub-pixel information.

17. The method of claim 14 , wherein decoding the feature maps includes relaying the feature maps to the unpacking blocks using skip connections between the packing blocks and the unpacking blocks,

wherein decoding the feature maps includes unfolding feature channels in the feature maps into spatial dimensions using the decoding layers including the unpacking blocks, and

wherein relaying the feature maps includes providing encoded features from separate ones of the encoding layers as residual information of high-level features in the monocular image to the decoding layers.

18. The method of claim 17 , wherein decoding according to the unpacking blocks includes, at the separate unpacking blocks, applying a two-dimensional convolutional operation, a three-dimensional convolutional operation, a reshaping operation, and a depth-to-space operation that applies sub-pixel convolutions to up-sample the spatial dimensions and super-resolve spatial information into the depth map, and

wherein decoding the feature maps includes to generate intermediate depth maps that provide inverse depths and that are inputs to subsequent ones of the decoding layers.

19. The method of claim 14 , wherein generating the depth map includes instructions to generate a set of depth maps at multiple different scales,

wherein the set of depth maps are inverse depth maps separately derived from the decoding layers, and

wherein the depth model is a machine learning algorithm comprised of the encoder layers and the decoder layers, the encoding layers including convolutional layers, the packing blocks, and residual blocks, and the decoding layers including the unpacking blocks, convolutional layers, and output layers for providing the set of depth maps.

20. The method of claim 14 , further comprising:

training the depth model using a training algorithm that is self-supervised and accounts for motion of a camera between successive monocular training images in training data using a loss function that accounts for structure from motion (SfM),

wherein training the depth model includes executing a pose model to account for the motion, and adjusting parameters in the depth model including at least sub-pixel convolutions in the decoding layers and encoding hyper-parameters in the encoding layers.