Systems and methods for enhancement of 3D object detection using point cloud semantic segmentation and attentive anchor generation
Devices, systems, methods, and media are disclosed for performing an object detection task comprising: obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space and are assigned a respective element classification label; clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label; generating, based on a location of the first cluster within the semantic segmentation map, at least one anchor that defines a respective probable object location of a first dynamic object; and generating, based on the semantic segmentation map and the at least one anchor, a respective bounding box and object instance classification label for the first dynamic object.
1. A method of performing an object detection task comprising:
obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space, the array of elements including elements that are each assigned a respective element classification label selected from a set of possible classification labels that correspond to different classifications of dynamic objects;
clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label;
generating, based on a location of the first cluster within the semantic segmentation map, a plurality of anchors, each of the plurality of anchors defining a different respective probable object location of a first dynamic object, wherein generating the plurality of anchors comprises:
computing an approximate location for the first dynamic object in the semantic segmentation map based on the locations of the elements of the first cluster;
generating a lower resolution map corresponding to the semantic segmentation map, and mapping the approximate location for the first dynamic object to a corresponding coarse element location in the lower resolution map:
generating a plurality of candidate anchors each indicating a different respective probable location of the first dynamic object relative to the coarse element location; and
mapping at least some of the plurality of candidate anchors to respective element locations of a higher resolution map to provide the plurality of anchors; and
generating, based on the semantic segmentation map and the at least one plurality of anchors, a respective bounding box and object instance classification label for the first dynamic object.
2. The method of claim 1 wherein computing the approximate location for the first dynamic object comprises determining a mean element location for the first cluster of elements based on the respective locations of the elements of the first cluster within the semantic segmentation map.
3. The method of claim 1 comprising sampling the plurality of candidate anchors to select only a subset of the plurality of candidate anchors to include in the mapping to the respective element locations of the higher resolution map.
4. The method of claim 1 wherein generating the plurality of candidate anchors comprises selecting, for each candidate anchor: an anchor geometry, an anchor orientation, and an anchor offset relative to the coarse element location.
5. The method of claim 1 , wherein clustering groups of the elements is performed to identify, in addition to the first cluster of elements, a plurality of further clusters that include elements that have each been assigned the same respective element classification label,
the method comprising, for each of the plurality of further clusters:
computing an approximate location in the semantic segmentation map for a respective dynamic object corresponding to the further cluster based on the location of the further cluster within the semantic segmentation map;
mapping the approximate location for the respective dynamic object to a corresponding coarse element location in the lower resolution map;
generating a respective plurality of candidate anchors each indicating a different respective probable location of the respective dynamic object; and
mapping at least some of the respective plurality of candidate anchors to respective element locations in the higher resolution map to provide a respective plurality of anchors for the further cluster, each anchor of the respective plurality of anchors defining a respective probable object location of the respective dynamic object in the higher resolution map,
the method further comprising:
generating a respective bounding box and object instance classification label for each of the respective dynamic objects represented in the plurality of further clusters based on the plurality of anchors provided for each of the plurality of further clusters.
6. The method of claim 5 comprising, prior to generating the respective bounding boxes and object instance classification labels for the first dynamic object and the respective dynamic objects represented in the plurality of further clusters, generating additional anchors according to a defined set of ad-hoc rules, each of the additional anchors defining a respective probable object location in the higher resolution map, wherein the generating the respective bounding boxes and object instance classification labels is also based on the additional anchors.
7. The method of claim 1 wherein obtaining the semantic segmentation map comprises obtaining a Light Detection and Ranging (LIDAR) frame of the real-world space using a LIDAR sensor and using a semantic segmentation model to assign the element classification labels used for the elements of the semantic segmentation map.
8. The method of claim 7 comprising applying a 3D to 2D conversion operation on an output of semantic segmentation model to generate the semantic segmentation map, wherein the semantic segmentation map represents a birds-eye-view (BEV) of the real-world space, and wherein the at least one anchor defines the respective probable object location of the first dynamic object with respect to the semantic segmentation map.
9. The method of claim 7 wherein the semantic segmentation map represents a 3D volume of the real-world space, and wherein the at least one anchor defines the respective probable object location of the first dynamic object with respect to the semantic segmentation map.
10. The method of claim 1 comprising controlling one or more of a steering and a speed of an autonomous vehicle based on the respective bounding box and object instance classification label for the first dynamic object.
11. A system comprising a processor device coupled to a memory, the memory storing executable instructions that when executed by the processor device configure the system to perform an object detection task comprising:
obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space, the array of elements including elements that are each assigned a respective element classification label selected from a set of possible classification labels that correspond to different classifications of dynamic objects;
clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label;
generating, based on a location of the first cluster within the semantic segmentation map, a plurality of anchors, each of the plurality of anchors defining a respective probable object location of a first dynamic object, wherein generating the plurality of anchors comprises:
computing an approximate location for the first dynamic object in the semantic segmentation map based on the locations of the elements of the first cluster;
generating a lower resolution map corresponding to the semantic segmentation map, and mapping the approximate location for the first dynamic object to a corresponding coarse element location in the lower resolution map;
generating a plurality of candidate anchors each indicating a different respective probable location of the first dynamic object relative to the coarse element location; and
mapping at least some of the plurality of candidate anchors to respective element locations of a higher resolution map to provide the plurality of anchors; and
generating, based on the semantic segmentation map and the at least one plurality of anchors, a respective bounding box and object instance classification label for the first dynamic object.
12. The system of claim 11 wherein computing the approximate location for the first dynamic object comprises determining a mean element location for the first cluster of elements based on the respective locations of the elements of the first cluster within the semantic segmentation map.
13. The system of claim 11 , the object detection task comprising sampling the plurality of candidate anchors to select only a subset of the plurality of candidate anchors to include in the mapping to the respective element locations of the higher resolution map.
14. The system of claim 11 wherein clustering groups of the elements is performed to identify, in addition to the first cluster of elements, a plurality of further clusters that include elements that have each been assigned the same respective element classification label,
the object detection task comprising, for each of the plurality of further clusters:
computing an approximate location in the semantic segmentation map for a respective dynamic object corresponding to the further cluster based on the location of the further cluster within the semantic segmentation map;
mapping the approximate location for the respective dynamic object to a corresponding coarse element location in the lower resolution map;
generating a respective plurality of candidate anchors each indicating a different respective probable location of the respective dynamic object; and
mapping at least some of the respective plurality of candidate anchors to respective element locations in the higher resolution map to provide a respective plurality of anchors for the further cluster, each anchor of the respective plurality of anchors defining a respective probable object location of the respective dynamic object in the higher resolution map,
the object detection task further comprising:
generating a respective bounding box and object instance classification label for each of the respective dynamic objects represented in the plurality of further clusters based on the plurality of anchors provided for each of the plurality of further clusters.
15. The system of claim 14 , the object detection task comprising, prior to generating the respective bounding boxes and object instance classification labels for the first dynamic object and the respective dynamic objects represented in the plurality of further clusters, generating additional anchors according to a defined set of ad-hoc rules, each of the additional anchors defining a respective probable object location in the higher resolution map, wherein the generating the respective bounding boxes and object instance classification labels is also based on the additional anchors.
16. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor device of a computing system, cause the computing system to perform a method comprising:
obtaining a semantic segmentation map representing a real-world space, the semantic segmentation map including an array of elements that each represent a respective location in the real-world space, the array of elements including elements that are each assigned a respective element classification label selected from a set of possible classification labels that correspond to different classifications of dynamic objects;
clustering groups of the elements based on the assigned respective element classification labels to identify at least a first cluster of elements that have each been assigned the same respective element classification label;
generating, based on a location of the first cluster within the semantic segmentation map, a plurality of anchors, each of the plurality of anchors defining a respective probable object location of a first dynamic object, wherein generating the plurality of anchors comprises:
computing an approximate location for the first dynamic object in the semantic segmentation map based on the locations of the elements of the first cluster;
generating a lower resolution map corresponding to the semantic segmentation map, and mapping the approximate location for the first dynamic object to a corresponding coarse element location in the lower resolution map;
generating a plurality of candidate anchors each indicating a different respective probable location of the first dynamic object relative to the coarse element location; and
mapping at least some of the plurality of candidate anchors to respective element locations of a higher resolution map to provide the plurality of anchors; and
generating, based on the semantic segmentation map and the plurality of anchors, a respective bounding box and object instance classification label for the first dynamic object.