Network architecture for ego-motion estimation

1. A self-supervised system for estimating ego-motion based on a plurality of input images, comprising:

one or more processors;

a memory, communicably connected to the one or more processors and storing:

a depth module including instructions that when executed by the one or more processors cause the one or more processors to determine a depth estimation D t based on a target image and a depth estimation D s based on a source image; and

a pose module including instructions that when executed by the one or more processors cause the one or more processors to determine an ego-motion estimation in a form of a six degrees-of-freedom (6 DOF) transformation between the target image and the source image by inputting the depth estimations (D t , D s ), the target image, and the source image into a two-stream network architecture trained to output the 6 DOF transformation based at least in part on the depth estimations (D t , D s ), the target image, and the source image,

the two-stream network architecture comprising:

an appearance stream convolution neural network (CNN) that convolves the source image and the target image; and

a structure stream CNN that convolves the depth estimations (D t , D s ),

wherein the pose module further includes instructions to fuse outputs of the appearance stream CNN and the structure stream CNN into a unified output to produce the 6 DOF transformation.

2. The self-supervised system of claim 1 , further comprising:

a synthesizer module including instructions that when executed by the one or more processors cause the one or more processors to synthesize a predicted image based at least in part on the ego-motion estimation, the depth estimation D t and the source image,

wherein the memory further stores instructions to compare the predicted image against the target image to determine photometric loss for the self-supervised system and adjust parameters of the self-supervised system to reduce the photometric loss by optimizing an associated loss function.

3. The self-supervised system of claim 2 , wherein the loss function includes an appearance based matching term defined as a linear combination between an L1 loss and a structural similarity (SSIM) patch-based loss, an edge-aware smoothness loss term, and a mask term that filters out stationary pixels and pixels with little photometric variation.

4. The self-supervised system of claim 1 , wherein the self-supervised system is trained using training data that is augmented with noise.

5. The self-supervised system of claim 4 , wherein the noise comprises random noise patches sized 81×81 to 101×101 at a noise augmentation level of 20%-40% coverage.

6. The self-supervised system of claim 1 , wherein the source image and the target image are both monocular images.

7. A method for estimating ego-motion based on a plurality of input images in a self-supervised system, comprising:

receiving a source image and a target image;

determining a depth estimation D t based on the target image;

determining a depth estimation D s based on a source image;

determining an ego-motion estimation in a form of a six degrees-of-freedom (6 DOF) transformation between the target image and the source image by inputting the depth estimations (D t , D s ), the target image, and the source image into a two-stream network architecture trained to output the 6 DOF transformation based at least in part on the depth estimations (D t , D s ), the target image, and the source image,

wherein determining the ego-motion estimation comprises:

convolving the source image and target image via an appearance stream convolution neural network (CNN);

convolving the depth estimations (D t , D s ) via a structure stream CNN; and

fusing outputs of the appearance stream CNN and the structure stream CNN into a unified output to produce the 6 DOF transformation.

8. The method of claim 7 , further comprising:

synthesizing a predicted image based at least in part on the ego-motion estimation, the depth estimation D t and the source image;

comparing the predicted image against the target image to determine systemic photometric loss; and

adjusting parameters of the self-supervised system to reduce the systemic photometric loss by optimizing an associated loss function.

9. The method of claim 8 , wherein the loss function includes an appearance based matching term defined as a linear combination between an L1 loss and a structural similarity (SSIM) patch-based loss, an edge-aware smoothness loss term, and a mask term that filters out stationary pixels and pixels with little photometric variation.

10. The method of claim 7 , further comprising training a network implementation of the method using training data that is augmented with noise.

11. The method of claim 10 , wherein the noise comprises random noise patches sized 81×81 to 101×101 at a noise augmentation level of 20%-40% coverage.

12. The method of claim 7 , wherein the source image and the target image are both monocular images.

13. A non-transitory computer-readable medium for estimating ego-motion based on a plurality of input images in a self-supervised system, including instructions that, when executed by one or more processors, cause the one or more processors to:

receive a source image and a target image;

determine a depth estimation D t based on the target image;

determine a depth estimation D s based on a source image;

determine an ego-motion estimation in a form of a six degrees-of-freedom (6 DOF) transformation between the target image and the source image by inputting the depth estimations (D t , D s ), the target image, and the source image into a two-stream network architecture trained to output the 6 DOF transformation based at least in part on the depth estimations (D t , D s ), the target image, and the source image; and

determine the ego-motion estimation by:

convolving the source image and target image via an appearance stream convolution neural network (CNN);

convolving the depth estimations (D t , D s ) via a structure stream CNN; and

fusing outputs of the appearance stream CNN and the structure stream CNN into a unified output to produce the 6 DOF transformation.

14. The non-transitory computer-readable medium of claim 13 , further including instructions to:

synthesize a predicted image based at least in part on the ego-motion estimation, the depth estimation D t and the source image;

compare the predicted image against the target image to determine systemic photometric loss; and

adjusting parameters of the self-supervised system to reduce the systemic photometric loss by optimizing an associated loss function.

15. The non-transitory computer-readable medium of claim 14 , wherein the loss function includes an appearance based matching term defined as a linear combination between an L1 loss and a structural similarity (SSIM) patch-based loss, an edge-aware smoothness loss term, and a mask term that filters out stationary pixels and pixels with little photometric variation.

16. The non-transitory computer-readable medium of claim 13 , further including instructions to train a network implementation of the non-transitory computer-readable medium by using training data that is augmented with noise.

17. The non-transitory computer-readable medium of claim 16 , wherein the noise comprises random noise patches sized 81×81 to 101×101 at a noise augmentation level of 20%-40% coverage.