Real-time obstacle detection with a calibrated camera and known ego-motion
View Patent ↗A method and system of real-time obstacle detection from a moving vehicle is provided. The method and system use a calibrated image capturing device. The method and system use a motion estimation technique to pick points with reliable image motion flows, and performs very fast sparse matching between the image motion flows and true motion flows calculated from the ego-motion of the image capturing device. Any mismatch between the image motion flows and the true motion flows are verified over time to achieve robust obstacle detection.
1. A method for real-time obstacle detection from a vehicle moving relative to a road, comprising:
calibrating an image capturing device, such that any image point can be mapped to a real-world point;
receiving one or more images from the image capturing device;
receiving information about the ego-motion of the image capturing device;
selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;
applying a motion estimation method on the one or more real-world points, the one or more images, and the known plane to determine an image motion flow of the one or more real-world points;
determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and
determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.
2. The method of claim 1 , wherein applying a motion estimation technique comprises applying an optical flow method with statistical fusion.
3. The method of claim 2 , wherein receiving information from the one or more sensors comprises receiving information from at least one of a gyro and a GPS.
4. The method of claim 1 , wherein receiving information about the ego-motion of the image capturing device comprises receiving information from one or more sensors onboard the vehicle.
5. The method of claim 1 , wherein receiving information about the ego-motion of the image capturing device comprises determining the information based on the one or more images.
6. The method of claim 1 , wherein determining that the one or more real-world point is one or more obstacle points by comparing the image motion flow and the true motion flow comprises:
if the image motion flow does not match the true motion flow, determining that the one or more real-world points are one or more obstacle points; and
if the image motion flow matches the true motion flow, determining that the one or more real-world points are not obstacle points.
7. The method of claim 1 , further comprising:
tracking data of a plurality of instances of the one or more obstacle points over a time interval; and
determining that the one or more obstacle points is one or more actual obstacles based on the data.
8. The method of claim 7 , wherein determining that the one or more obstacle points is one or more actual obstacles based on the data comprises determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on other detection methods.
9. The method of claim 8 , wherein determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on other detection methods comprises determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on at least one of radar obstacle detection, lidar obstacle detection, and ultrasonic obstacle detection.
10. The method of claim 7 , further comprising classifying a type of the one or more actual obstacles using automated detection.
11. The method of claim 7 , further comprising classifying a type of the one or more actual obstacles using a graphical user interface.
12. The method of claim 1 , wherein calibrating the image capturing device comprises calibrating a video camera.
13. A system for real-time obstacle detection from a vehicle moving relative to a road, comprising:
an image capturing device;
means for calibrating an image capturing device, such that any image point can be mapped to a real-world point;
means for receiving one or more images from the image capturing device;
means for receiving information about the ego-motion of the image capturing device;
means for selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;
means for applying a motion estimation method on the one or more real-world points, the one or more images, and the road to determine an image motion flow of the one or more real-world points;
means for determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and
means for determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.
14. The system of claim 13 , wherein the image capturing device comprises a video camera.
15. The system of claim 13 , further comprising:
means for tracking data of a plurality of instances of the one or more obstacle points over a time interval; and
means for determining that the one or more obstacle points is one or more actual obstacles based on the data.
16. The system of claim 13 , further comprising one or more onboard vehicle sensors for determining the ego-motion of the image capturing device.
17. The system of claim 16 , wherein the one or more onboard vehicle sensors comprises a gyro sensor.
18. The system of claim 16 , wherein the one or more onboard vehicle sensors comprises a GPS sensor.
19. A computer readable medium encoded with computer executable instructions for performing a method for real-time obstacle detection from a vehicle moving relative to a road, comprising:
calibrating an image capturing device, such that any image point can be mapped to a real-world point;
receiving one or more images from the image capturing device;
receiving information about the ego-motion of the image capturing device;
selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;
applying a motion estimation method on the one or more real-world points, the one or more images, and the road to determine an image motion flow of the one or more real-world points;
determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and
determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.