System and method for self-supervised depth and ego-motion overfitting
Systems and methods to improve machine learning by explicitly over-fitting environmental data obtained by an imaging system, such as a monocular camera are disclosed. The system includes training self-supervised depth and pose networks in monocular visual data collected from a certain area over multiple passes. Pose and depth networks may be trained by extracting data from multiple images of a single environment or trajectory, allowing the system to overfit the image data.
1. A system for generating depth estimates of an environment, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a depth system including instructions that when executed by the one or more processors cause the one or more processors to generate a plurality of depth maps by:
receiving a plurality of monocular images, each of the plurality of monocular images capturing substantially the same environment of interest;
processing each of the monocular images according to an overfit depth model;
filtering the plurality of depth maps to remove non-static objects by comparing non-consecutive depth maps of a route trajectory;
an image module including instructions that when executed by the one or more processors cause the one or more processors to generate a permanent three-dimensional reconstruction of the environment and the route trajectory including locations of obstacles based on the plurality of depth maps; and
a planner module to plan a trajectory of a car that safely navigates the obstacles.
2. The system of claim 1 wherein the overfit depth model comprises a convolutional neural network.
3. The system of claim 1 wherein the overfit depth model comprises a pose network.
4. The system of claim 3 wherein the pose network outputs a relative pose between a pair of monocular images.
5. The system of claim 1 wherein the overfit depth model comprises a depth network.
6. The system of claim 5 wherein the depth network projects two dimensional pixels into three dimensional points.
7. A method of generating depth estimates of an environment, the method comprising:
receiving a plurality of monocular images, each of the plurality of monocular images capturing substantially the same environment of interest;
processing each of the monocular images according to an overfit depth model to generate a plurality of depth maps;
filtering the plurality of depth maps to remove non-static objects by comparing non-consecutive depth maps of a route trajectory;
generating from the plurality of depth maps a permanent three-dimensional reconstruction of the environment and the route trajectory including locations of obstacles; and
planning a trajectory of a car that safely navigates the obstacles.
8. The method of claim 7 wherein the overfit depth model comprises a convolutional neural network.
9. The method of claim 7 wherein the overfit depth model comprises a pose network.
10. The method of claim 9 wherein the pose network outputs a relative pose between a pair of monocular images.
11. The method of claim 7 wherein the overfit depth model comprises a depth network.
12. The method of claim 11 wherein the depth network projects two dimensional pixels into three dimensional points.
13. A non-transitory computer-readable medium for generating depth estimates of an environment and including instructions that when executed by one or more processors cause the one or more processors to:
receive a plurality of monocular images, each of the plurality of monocular images capturing substantially the same environment of interest;
process each of the monocular images according to an overfit depth model to generate a plurality of depth maps;
filter the plurality of depth maps to remove non-static objects by comparing non-consecutive depth maps of a route trajectory;
generate from the plurality of depth maps a permanent three-dimensional reconstruction of the environment and the route trajectory including locations of obstacles; and
plan a trajectory of a car that safely navigates the obstacles.
14. The non-transitory computer-readable medium of claim 13 wherein the overfit depth model comprises a convolutional neural network.
15. The non-transitory computer-readable medium of claim 13 wherein the overfit depth model comprises a pose network.
16. The non-transitory computer-readable medium of claim 15 wherein the pose network outputs a relative pose between a pair of monocular images.
17. The non-transitory computer-readable medium of claim 1 wherein the overfit depth model comprises a depth network.