Sequential fusion for 3D object detection

1. A method comprising:

matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image;

generating, using an image-based neural network, semantic segmentation scores for pixels of the image;

decorating, using the one or more processors, the points in the 3D point cloud with the semantic segmentation scores; and

estimating, using the decorated points, oriented 3D bounding boxes for one or more objects;

wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

encoding, using a point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of the one or more objects; and

estimating, using the features, oriented 3D bounding boxes for the one or more objects;

wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

2. The method of claim 1 , wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

projecting the 3D point cloud into a birds eye view (BEV); and

decorating points in the BEV with the semantic segmentation scores.

3. The method of claim 1 , wherein matching temporally the points in the 3D point cloud with the image includes projecting a most recent point cloud to a semantic data output from a previous image in time.

4. The method of claim 1 , wherein each particular point is transformed from a light detection and ranging (LiDAR) vehicle coordinate frame to a camera coordinate frame and a segmentation score vector is obtained for each pixel in the image where the particular point is projected in the camera coordinate frame.

5. The method of claim 1 , further comprising:

determining that one or more points project on two images simultaneously; and

randomly choosing the segmentation score vector from one of the two images.

6. The method of claim 1 , wherein the points in each pillar include the values x_c, y_c, z_c, x_p and y_p where the values x_c, y_c, z_c denote a distance to an arithmetic mean of all points in the pillar and x_p and y_p denote the offset from the pillar x, y center, and where the points in each pillar are augmented with the semantic segmentation scores.

7. The method of claim 1 , wherein the semantic segmentation scores includes semantic segmentation scores reduced to one or more classes in a set of classes that includes at least one of car, bike, pedestrian, barrier, traffic cone, drivable surface or background classes.

8. The method of claim 1 , further comprising:

generating, using the image-based neural network, two-dimensional (2D) or three-dimensional (3D) bounding boxes; and

decorating the points with classification scores for the 2D or 3D bounding boxes in addition to the semantic segmentation scores or instead of the semantic segmentation scores.

9. The method of claim 8 , further comprising:

determining that the two or more bounding boxes are overlapping and include a common point;

computing an average of the classification scores for the two or more overlapping bounding boxes; and

decorating the common point with the average of the classification scores.

10. A system comprising:

one or more processors of a vehicle; and

memory storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

matching temporally points in a three-dimensional (3D) point cloud with an image;

generating, using an image-based neural network, semantic segmentation scores for pixels of the image;

decorating the points in the 3D point cloud with the semantic segmentation scores; and

estimating, using the decorated points, oriented 3D bounding boxes for one or more objects;

wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

encoding, using a point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of the one or more objects; and

estimating, using the features, oriented 3D bounding boxes for the one or more objects

wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

11. The system of claim 10 , wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

projecting the 3D point cloud into a birds eye view (BEV); and

decorating points in the BEV with the semantic segmentation scores.

12. The system of claim 10 , wherein matching temporally the points in the 3D point cloud with the image includes projecting a most recent point cloud to a semantic segmentation score output from a previous image in time.

13. The system of claim 10 , wherein each particular point is transformed from a light detection and ranging (LiDAR) vehicle coordinate frame to a camera coordinate frame and a segmentation score vector is obtained for each pixel in the image where the particular point is projected in the camera coordinate frame.

14. The system of claim 13 , further comprising:

determining that one or more points project on two images simultaneously; and

randomly choosing the segmentation score vector from one of the two images.

15. The system of claim 10 , wherein the points in each pillar include the values x_c, y_c, z_c, x_p and y_p where the values x_c, y_c, z_c denote a distance to an arithmetic mean of all points in the pillar and x_p and y_p denote the offset from the pillar x, y center, and where the points in each pillar are augmented with the semantic segmentation scores.

16. The system of claim 10 , wherein the semantic segmentation scores are reduced to one or more classes in a set of classes that includes at least one of car, bike, pedestrian, barrier, traffic cone, drivable surface or background classes.

17. The system of claim 10 , further comprising:

generating, using the image-based neural network, two-dimensional (2D) or three-dimensional (3D) bounding boxes; and

decorating the points with classification scores for the 2D or 3D bounding boxes in addition to the semantic segmentation scores or instead of the semantic segmentation scores.

18. The system of claim 17 , further comprising:

determining that the two or more bounding boxes are overlapping and include a common point;

computing an average of the classification scores for the two or more overlapping bounding boxes; and

decorating the common point with the average of the classification scores.

19. A non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

matching temporally points in a three-dimensional (3D) point cloud with an image;

generating, using an image-based neural network, semantic segmentation scores for pixels of the image;

decorating the points in the 3D point cloud with the semantic segmentation scores; and

estimating, using the decorated points, oriented 3D bounding boxes for one or more objects;

wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

encoding, using a point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of the one or more objects; and

estimating, using the features, oriented 3D bounding boxes for the one or more objects wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

20. A method comprising:

matching temporally, using one or more processors of a vehicle, points within a three-dimensional (3D) set of points with an image;

generating, using an image-based neural network, semantic segmentation scores for pixels of the image;

decorating, using the one or more processors, the points within the 3D set of points with the semantic segmentation scores;

encoding, using a point pillars network, the decorated points into point pillars;

generating, using a deep learning network that receives the point pillars as input, features indicative of one or more objects; and

estimating, using the features, oriented 3D bounding boxes for the one or more objects.

21. The method of claim 20 , wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

22. The method of claim 21 , wherein the points in each pillar include values of a distance to an arithmetic mean of all points in the pillar and an offset from a center of the pillar, wherein the points in each pillar are augmented with the semantic segmentation scores.

23. The method of claim 20 , further comprising:

decorating the points within the 3D set of points with classification scores for the oriented 3D bounding boxes in addition to the semantic segmentation scores or instead of the semantic segmentation scores;

determining that the oriented 3D bounding boxes are overlapping and include a common point;

computing an average of the classification scores for the two or more overlapping bounding boxes; and

decorating the common point with the average of the classification scores.