System and method for determining object characteristics in real-time
System and method for object detection. Images from cameras are provided to an inference engine to detect objects in real time, providing the images to an inference engine to detect the non-background and background pixels of the objects in the images, determining the position and size of the objects in the images based on contemporaneously gathered LiDAR data and the relationship of non-background to background pixels.
1. A method for determining a size and a location of an object surrounding an autonomous vehicle (AV) in real time comprising:
receiving two-dimensional (2D) sensor data from a sensor associated with the AV;
associating a first confidence score with each of the 2D sensor data;
determining a bounding box, a classification, and a second confidence score representing the object located within the 2D sensor data;
receiving point cloud data temporally and geographically associated with the bounding box;
classifying pixels in the bounding box as non-background pixels or background pixels; and
determining the location and the size of the object based on a relationship between an amount of the non-background pixels and the background pixels and the point cloud data;
wherein the point cloud data are independent of the 2D sensor data.
2. The method as in claim 1 wherein the sensor comprises a camera.
3. The method as in claim 1 wherein determining the bounding box, the classification, and the confidence score comprises providing the sensor data to a real-time object detection inference engine.
4. The method as in claim 3 wherein the real-time object detection inference engine comprises a one-stage object detector.
5. The method as in claim 1 wherein classifying the pixels as the non-background pixels and the background pixels comprises providing the sensor data to a semantic segmentation inference engine.
6. The method as in claim 5 wherein the semantic segmentation inference engine comprises an encoder-decoder engine.
7. The method as in claim 1 further comprising filtering the background pixels and the non-background pixels.
8. The method as in claim 7 wherein filtering comprises a median filter.
9. The method as in claim 7 wherein filtering comprises a mode filter.
10. The method as in claim 1 wherein the sensor comprises four long range cameras.
11. A system for determining a size and a location of at least one object surrounding an autonomous vehicle (AV) in real time comprising:
a bounding box processor configured for:
receiving two-dimensional (2D) sensor data and a first confidence score respectively associated with each of the 2D sensor data from a sensor associated with the AV; and
determining a bounding box, a classification, and a second confidence score representing the object located within the 2D sensor data;
a class processor configured for classifying pixels in the bounding box as non-background pixels or background pixels; and
a 3D positioner configured for:
receiving point cloud data temporally and geographically associated with the bounding box; and
determining the location and the size of the object based on a relationship between an amount of the non-background pixels and the background pixels and the point cloud data;
wherein the point cloud data are independent of the 2D sensor data.
12. The system as in claim 11 wherein the sensor comprises a camera.
13. The system as in claim 11 wherein the bounding box processor is configured for providing the sensor data to a real-time object detection inference engine.
14. The system as in claim 13 wherein the real-time object detection inference engine comprises a one-stage object detection engine.
15. The system as in claim 11 wherein the 3D positioner is configured for providing the sensor data to a semantic segmentation inference engine.
16. The system as in claim 15 wherein the semantic segmentation inference engine comprises an encoder-decoder engine.
17. The system as in claim 11 further comprising a filter configured for filtering the background pixels and the non-background pixels.
18. The system as in claim 17 wherein the filter comprises a median filter.
19. The system as in claim 17 wherein the filter comprises a mode filter.
20. The system as in claim 12 wherein the sensor comprises four long range cameras.
21. Method of claim 1 wherein the first confidence score is based on LiDAR reflectivity.
22. System of claim 1 wherein the first confidence score is based on LIDAR reflectivity.