Image-based pedestrian detection
Object detection systems and methods can include identifying an object of interest within an image obtained from a camera, obtaining a first supplemental portion of data associated with the object of interest determining an estimated location of the object of interest within three-dimensional space based at least in part on the first supplemental portion of data and a known relative location of the camera, determining a portion of the LIDAR point data corresponding to the object of interest based at least in part on the estimated location of the object of interest within three-dimensional space, and providing one or more of at least a portion of the image corresponding to the object of interest and the portion of LIDAR point data corresponding to the object of interest as an output.
1. A method for detecting pedestrians in images, comprising:
identifying, by a computing system comprising one or more electronic devices, a pedestrian within an image obtained from a camera positioned relative to a vehicle;
determining, by the computing system, an estimated support location for the pedestrian within the image;
determining, by the computing system, a supplemental portion of data correlated with the pedestrian, the supplemental portion of data obtained from a sensor other than the camera;
determining, by the computing system, an estimated location of the pedestrian within three-dimensional space based at least in part on (i) the estimated support location for the pedestrian within the image, (ii) a known relative location of the camera to the vehicle, and (iii) the supplemental portion of data;
obtaining, by the computing system, map data descriptive of a geographic area surrounding the vehicle, the map data including relative locations of ground surfaces proximate to the vehicle;
determining, by the computing system, a revised estimated location of a pedestrian within three-dimensional space based at least in part on the estimated location and the map data; and
providing, by the computing system, at least a portion of the image corresponding to the pedestrian and the estimated location of the pedestrian within the three-dimensional space to an object classification application.
2. The method of claim 1 wherein the supplemental portion of data comprises a portion of LIDAR point data corresponding to the pedestrian.
3. The method of claim 2 , wherein determining, by the computing system, the portion of LIDAR point data comprises identifying, by the computing system, a cluster of LIDAR data points of a predetermined size near a location within a point cloud corresponding to the estimated location of the pedestrian within the three-dimensional space.
4. The method of claim 1 , wherein determining, by the computing system, the estimated support location for the pedestrian within the image comprises determining, by the computing system, a location within the image corresponding to feet of the pedestrian.
5. The method of claim 1 , wherein determining, by the computing system, the estimated support location for the pedestrian within the image comprises:
determining, by the computing system, a location within the image corresponding to a head of the pedestrian; and
determining, by the computing system, the estimated support location for the pedestrian based at least in part on the location within the image corresponding to the head of the pedestrian and an estimated height of the pedestrian.
6. The method of claim 1 , wherein determining a revised estimated location of the pedestrian within three-dimensional space based at least in part on the estimated location and the map data comprises adjusting the estimated location to a nearest point within three-dimensional space corresponding to one of the relative locations of ground surfaces identified within the map data.
7. The method of claim 1 , wherein identifying, by the computing system, a pedestrian within an image obtained from a camera positioned relative to a vehicle comprises:
generating, by the computing system, a multi-scale image pyramid of multiple image samples having different scaling factors; and
analyzing, by the computing system, successive image patches within each of the multiple image samples using a sliding window of fixed size to identify pedestrians within the successive image patches.
8. The method of claim 1 , wherein identifying, by the computing system, a pedestrian within an image obtained from a camera positioned relative to a vehicle comprises:
determining, by the computing system, a region proposal within the image, the region proposal having a smaller area than the image; and
identifying, by the computing system, the pedestrian within the region proposal.
9. The method of claim 8 , wherein the region proposal is determined at least in part from locations within the image corresponding to locations for which corresponding LIDAR point data is also obtained.
10. The method of claim 8 , wherein the region proposal is determined at least in part from locations within the image determined to correspond to foreground as opposed to background.
11. A detection system, comprising:
a camera positioned relative to a vehicle and configured to obtain images within an area proximate to the vehicle;
a LIDAR sensor positioned relative to a vehicle and configured to obtain LIDAR point data within an area proximate to the vehicle;
one or more electronic devices coupled to the camera and the LIDAR sensor, the one or more electronic devices configured to receive image data from the camera and LIDAR point data from the LIDAR sensor and perform operations comprising:
identifying an object of interest within an image obtained from the camera;
determining an estimated location of the object of interest within three-dimensional space based at least in part on a known relative location of the camera;
obtaining map data descriptive of a geographic area surrounding the vehicle; and
determining a revised estimated location of an object of interest within three-dimensional space based at least in part on the estimated location and the map data;
determining a portion of the LIDAR point data corresponding to the object of interest based at least in part on the estimated location of the object of interest within three-dimensional space; and
providing one or more of at least a portion of the image corresponding to the object of interest and the portion of the LIDAR point data corresponding to the object of interest as an output.
12. The detection system of claim 11 , further comprising determining an estimated support location for the object of interest within the image, wherein:
when a lower bound of the object of interest is visible within the image, determining an estimated support location for the object of interest within the image comprises determining a location within the image corresponding to the lower bound of the object of interest; and
when the lower bound of the object of interest is not visible, determining an estimated support location for the object of interest within the image comprises: determining a location within the image corresponding to an upper bound of the object of interest; and determining the estimated support location for the object of interest based at least in part on the location within the image corresponding to the upper bound of the object of interest and an estimated height of the object of interest.
13. The detection system of claim 11 , wherein determining a revised estimated location of an object of interest within three-dimensional space based at least in part on the estimated location and the map data comprises adjusting the estimated location to a nearest point within three-dimensional space corresponding to one of the relative locations of ground surfaces identified within the map data.
14. The detection system of claim 11 , wherein identifying an object of interest within an image obtained from the camera comprises:
generating a multi-scale image pyramid of multiple image samples having different scaling factors; and
analyzing successive image patches within each of the multiple image samples using a sliding window of fixed size to identify objects of interest within the successive image patches.
15. The detection system of claim 11 , wherein identifying an object of interest within an image obtained from the camera comprises:
determining a region proposal within the image, the region proposal having a smaller area than the image; and
identifying an object of interest within the region proposal.
16. An autonomous vehicle, comprising:
one or more cameras configured to obtain images within an environment proximate to an autonomous vehicle;
a field programmable gate array (FPGA) device coupled to the one or more cameras, the FPGA device configured to receive an image from the one or more cameras, generate a multi-scale image pyramid of multiple image samples having different scaling factors, and analyze successive image patches within each of the multiple image samples using a sliding window of fixed size to identify objects of interest within the successive image patches;
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
determining a supplemental portion of data correlated with each object of interest, each supplemental portion of data obtained from a sensor other than the one or more cameras, each supplemental portion of data comprising map data descriptive of a geographic area surrounding the autonomous vehicle and including relative locations of ground surfaces proximate to the autonomous vehicle; and
providing at least a portion of the image corresponding to each object of interest and the supplemental portion of data associated with each object of interest to an object classification application.
17. The autonomous vehicle of claim 16 , further comprising:
a LIDAR sensor positioned relative to the autonomous vehicle and configured to obtain LIDAR point data within an area proximate to the autonomous vehicle; and
wherein the operations further comprise providing at least a portion of the LIDAR point data corresponding to an estimated location of each object of interest within three-dimensional space to the object classification application.
18. The autonomous vehicle of claim 16 , wherein the FPGA device is further configured to determine a region proposal within each image sample, each region proposal having a smaller area than its corresponding image sample, and to analyze successive image patches within the region proposal for each of the multiple image samples by using a sliding window of fixed size to identify objects of interest within the successive image patches.
19. The autonomous vehicle of claim 18 , wherein the region proposal for each image sample is determined at least in part from locations within the image corresponding to locations for which corresponding LIDAR point data is also obtained.
20. The autonomous vehicle of claim 18 , wherein the region proposal is determined at least in part from locations within the image determined to correspond to foreground as opposed to background.