Sequential fusion for 3D object detection

1. A method comprising:

matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image;

generating, using a neural network, a semantic segmentation scores for the image;

decorating, using the one or more processors, the points in the 3D point cloud with the semantic segmentation scores to generate decorated points; and

estimating, using a point pillars network and the decorated points, oriented 3D bounding boxes for the one or more objects.

2. The method of claim 1 , wherein estimating, using a point pillars network and the decorated points, oriented 3D bounding boxes for the one or more objects comprises:

encoding, using the point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of one or more objects;

estimating, using a detection head with the features as input, oriented 3D bounding boxes for the one or more objects.

3. The method of claim 1 , wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

projecting the 3D point cloud into a birds eye view (BEV); and

decorating points in the BEV with the semantic data.

4. The method of claim 1 , wherein matching temporally the points in the 3D point cloud with the image includes projecting a most recent point cloud to a semantic segmentation scores output from a previous image in time.

5. The method of claim 1 , wherein each particular point is transformed from a light detection and ranging (LiDAR) ego-vehicle coordinate frame to a camera coordinate frame and a segmentation score vector is obtained for each pixel in the image where the particular point is projected in the camera coordinate frame.

6. The method of claim 5 , further comprising:

determining that one or more points project on two images simultaneously; and

randomly choosing the segmentation score vector from one of the two images.

7. The method of claim 2 , wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

8. The method of claim 7 , wherein the points in each pillar include the values x_c, y_c, z_c, x_p and y_p where the values x_c, y_c, z_c denote a distance to an arithmetic mean of all points in the pillar and x_p and y_p denote the offset from the pillar x, y center, and where the points in each pillar are augmented with the semantic segmentation scores.

9. The method of claim 1 , wherein the semantic segmentation scores include semantic segmentation scores reduced to one or more classes in a set of classes that includes at least one of car, bike, pedestrian, barrier, traffic cone, drivable surface or background classes.

10. The method of claim 1 , further comprising:

generating, using the image-based neural network, two-dimensional (2D) or three-dimensional (3D) bounding boxes; and

decorating the points with classification scores for the 2D or 3D bounding boxes in addition to the semantic data or instead of the semantic data.

11. The method of claim 10 , further comprising:

determining that the two or more bounding boxes are overlapping and include a common point;

computing an average of the classification scores for the two or more overlapping bounding boxes; and

decorating the common point with the average of the classification scores.

12. A system comprising:

one or more processors of a vehicle; and

memory storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

matching temporally points in a three-dimensional (3D) point cloud with an image;

generating, using a neural network, semantic segmentation scores for the image;

decorating the points in the 3D point cloud with the semantic segmentation scores to generate decorated points; and

estimating, using a point pillars network and the decorated points, oriented 3D bounding boxes for the one or more objects.

13. The system of claim 12 , wherein estimating, using a point pillars network and the decorated points, oriented 3D bounding boxes for the one or more objects comprises:

encoding, using the point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of one or more objects;

estimating, using a detection head with the features as input, oriented 3D bounding boxes for the one or more objects.

14. The system of claim 12 , wherein decorating the points in the 3D point cloud with the semantic segmentation scores further comprises:

projecting the 3D point cloud into a birds eye view (BEV); and

decorating points in the BEV with the semantic data.

15. The system of claim 12 , wherein matching temporally the points in the 3D point cloud with the image includes projecting a most recent point cloud to a semantic segmentation scores output from a previous image in time.

16. The system of claim 12 , wherein each particular point is transformed from a light detection and ranging (LiDAR) ego-vehicle coordinate frame to a camera coordinate frame and a segmentation score vector is obtained for each pixel in the image where the particular point is projected in the camera coordinate frame.

17. The system of claim 16 , further comprising:

determining that one or more points project on two images simultaneously; and

randomly choosing the segmentation score vector from one of the two images.

18. The system of claim 13 , wherein encoding the decorated points into point pillars includes augmenting point pillar values with the semantic segmentation scores.

19. The system of claim 18 , wherein the points in each pillar include the values x_c, y_c, z_c, x_p and y_p where the values x_c, y_c, z_c denote a distance to an arithmetic mean of all points in the pillar and x_p and y_p denote the offset from the pillar x, y center, and where the points in each pillar are augmented with the semantic segmentation scores.

20. The system of claim 12 , wherein the semantic segmentation scores include semantic segmentation scores reduced to one or more classes in a set of classes that includes at least one of car, bike, pedestrian, barrier, traffic cone, drivable surface or background classes.

21. The system of claim 12 , further comprising:

generating, using the image-based neural network, two-dimensional (2D) or three-dimensional (3D) bounding boxes; and

decorating the points with classification scores for the 2D or 3D bounding boxes in addition to the semantic data or instead of the semantic segmentation scores.

22. The system of claim 21 , further comprising:

determining that the two or more bounding boxes are overlapping and include a common point;

computing an average of the classification scores for the two or more overlapping bounding boxes; and

decorating the common point with the average of the classification scores.

23. A non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

matching temporally points in a three-dimensional (3D) point cloud with an image;

generating, using a neural network, semantic segmentation scores for the image;

decorating the points in the 3D point cloud with the semantic segmentation scores to generate decorated points; and

estimating, using a point pillars network and the decorated points, oriented 3D bounding boxes for the one or more objects.

24. The non-transitory computer-readable storage medium of claim 23 , wherein decorating the points in the 3D point cloud with the semantic data further comprises:

encoding, using the point pillars network, the decorated points into point pillars;

generating, using a deep learning network with the point pillars as input, features indicative of one or more objects;

estimating, using a detection head with the features as input, oriented 3D bounding boxes for the one or more objects.