Deep learning for object detection using pillars

1. A system comprising:

one or more computer processors; and

one or more non-transitory storage media storing instructions which, when executed by the one or more computer processors, cause performance of operations comprising:

receiving a set of measurements from a sensor of a vehicle, the set of measurements comprising a plurality of data points that represent a plurality of objects in a three-dimensional (3D) space surrounding the vehicle, wherein each data point of the plurality of data points is represented by a set of 3D spatial coordinates;

dividing the 3D space into a plurality of pillars, wherein each pillar of the plurality of pillars comprises a slice of the 3D space that extends from a respective portion of a two-dimensional (2D) plane of the 3D space;

assigning each data point of the plurality of data points to a pillar in the plurality of pillars;

generating a 2D pseudo-image based on the plurality of pillars, wherein each location on the 2D pseudo-image corresponds to a respective pillar of the plurality of pillars and represents features of the data points in the respective pillar, wherein the 2D pseudo-image comprises more channels than channels of a standard RGB image;

processing the 2D pseudo-image using 2D convolutions to generate an intermediate output;

detecting the plurality of objects in the 3D space surrounding the vehicle based on an analysis of the intermediate output by an object detection neural network; and

operating the vehicle based upon the detecting of the plurality of objects in the 3D space surrounding the vehicle.

2. The system of claim 1 , further comprising:

determining whether a first count of a plurality of non-empty pillars exceeds a first threshold value P;

in accordance with a determination that the first count of the plurality of non-empty pillars exceeds the first threshold value P, selecting P non-empty pillars from the plurality of non-empty pillars;

in accordance with a determination that the first count of the plurality of non-empty pillars is less than the first threshold value P:

generating a second subset of pillars, wherein a sum of the first count and a second count of the second subset of pillars is equal to P;

selecting P non-empty pillars from the plurality of non-empty pillars and the second subset of pillars;

for each non-empty pillar of the P non-empty pillars:

determining whether a third count of data points assigned to the each non-empty pillar exceeds a second threshold value N;

in accordance with a determination that the third count of the data points assigned to the each non-empty pillar exceeds the second threshold value N, selecting N data points;

in accordance with a determination that the third count of the data points in the each non-empty pillar is less than the second threshold value N, assigning the each non-empty pillar a plurality of zero coordinate data points, wherein the sum of a fourth count of the plurality of zero coordinate data points and the third count equals N.

3. The system of claim 2 , wherein the first threshold value P is adaptive.

4. The system of claim 2 , wherein the second threshold value N is adaptive.

5. The system of claim 2 , wherein generating the 2D pseudo-image comprises:

for each non-empty pillar of the plurality of non-empty pillars:

generating a plurality of modified data points based upon the plurality of data points corresponding to the each non-empty pillar; and

processing the plurality of non-empty pillars comprising respective pluralities of modified data points to generate the 2D pseudo-image.

6. The system of claim 5 , wherein each of the modified data points is generated based on a relative distance between the data point corresponding to the each non-empty pillar and a center of the each non-empty pillar.

7. The system of claim 1 , wherein each of the data points is represented by modified 3D spatial coordinates and a cylindrical Euclidean distance from the sensor to the data point.

8. The system of claim 7 , wherein each of the data points is, is represented by the modified 3D spatial coordinates, the cylindrical Euclidean distance, a reflectance, and a time stamp.

9. The system of claim 1 , wherein generating the 2D pseudo-image comprises processing the plurality of pillars using a pillar processing module to generate the 2D pseudo-image.

10. The system of claim 1 , wherein detecting the plurality of objects in the 3D space surrounding the vehicle based on the analysis of the 2D pseudo-image comprises:

processing the 2D pseudo-image using a two-dimensional (2D) backbone neural network to generate the intermediate output that characterizes features of the 2D pseudo-image; and

processing the intermediate output using the object detection neural network to detect the plurality of objects in the 3D space surrounding the vehicle.

11. A method comprising:

receiving a set of measurements from a sensor of a vehicle, the set of measurements comprising a plurality of data points that represent a plurality of objects in a three-dimensional (3D) space surrounding the vehicle, wherein each data point of the plurality of data points is represented by a set of 3D spatial coordinates;

dividing the 3D space into a plurality of pillars, wherein each pillar of the plurality of pillars comprises a slice of the 3D space that extends from a respective portion of a two-dimensional (2D) plane of the 3D space;

assigning each data point of the plurality of data points to a pillar in the plurality of pillars;

generating a 2D pseudo-image based on the plurality of pillars, wherein each location on the 2D pseudo-image corresponds to a respective pillar of the plurality of pillars and represents features of the data points in the respective pillar, wherein the 2D pseudo-image comprises more channels than channels of a standard RGB image;

processing the 2D pseudo-image using 2D convolutions to generate an intermediate output;

detecting the plurality of objects in the 3D space surrounding the vehicle based on an analysis of the intermediate output by an object detection neural network; and

operating the vehicle based upon the detecting of the plurality of objects in the 3D space surrounding the vehicle,

wherein the method is performed by one or more special-purpose computing devices.

12. The method of claim 11 , further comprising:

determining whether a first count of a plurality of non-empty pillars exceeds a first threshold value P;

in accordance with a determination that the first count of the plurality of non-empty pillars exceeds the first threshold value P, selecting P non-empty pillars from the plurality of non-empty pillars;

in accordance with a determination that the first count of the plurality of non-empty pillars is less than the first threshold value P:

generating a second subset of pillars, wherein a sum of the first count and a second count of the second subset of pillars is equal to P;

selecting P non-empty pillars from the plurality of non-empty pillars and the second subset of pillars;

for each non-empty pillar of the P non-empty pillars:

determining whether a third count of data points assigned to the each non-empty pillar exceeds a second threshold value N;

in accordance with a determination that the third count of the data points assigned to the each non-empty pillar exceeds the second threshold value N, selecting N data points;

in accordance with a determination that the third count of the data points in the each non-empty pillar is less than the second threshold value N, assigning the each non-empty pillar a plurality of zero coordinate data points, wherein the sum of a fourth count of the plurality of zero coordinate data points and the third count equals N.

13. The method of claim 12 , wherein the first threshold value P is adaptive.

14. The method of claim 12 , wherein generating the 2D pseudo-image comprises:

for each non-empty pillar of the plurality of non-empty pillars:

generating a plurality of modified data point based upon the plurality of data points corresponding to the each non-empty pillar; and

processing the plurality of non-empty pillars comprising respective pluralities of modified data points to generate the 2D pseudo-image.

15. The method of claim 14 , wherein each of the modified data points is generated based on a relative distance between the data point corresponding to the each non-empty pillar and a center of the each non-empty pillar.

16. The method of claim 11 , wherein each data point is represented by the modified 3D spatial coordinates and a cylindrical Euclidean distance from the sensor to the data point.

17. The method of claim 16 , wherein each of the data points is represented by the modified 3D spatial coordinates, the cylindrical Euclidean distance, a reflectance, and a time stamp.

18. The method of claim 11 , wherein generating the 2D pseudo-image comprises processing the plurality of pillars using a pillar processing module to generate the 2D pseudo-image.

19. The method of claim 11 , wherein detecting the plurality of objects in the 3D space surrounding the vehicle based on the analysis of the pseudo-image comprises:

processing the pseudo-image using a backbone neural network to generate the intermediate output that characterizes features of the pseudo-image; and

processing the intermediate output using the object detection neural network to detect the plurality of objects in the 3D space surrounding the vehicle.

20. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause performance of the method recited in claim 11 .