Deep learning for object detection using pillars

1. A system comprising:

one or more computer processors; and

one or more non-transitory storage media storing instructions which, when executed by the one or more computer processors, cause performance of operations comprising:

receiving a set of measurements from a sensor of a vehicle, the set of measurements comprising a plurality of data points that represent a plurality of objects in a three-dimensional (3D) space surrounding the vehicle, wherein each data point of the plurality of data points is represented by a set of 3D spatial coordinates;

dividing the 3D space into a plurality of pillars, wherein each pillar of the plurality of pillars comprises a slice of the 3D space that extends from a respective portion of a two-dimensional (2D) plane of the 3D space;

assigning each data point of the plurality of data points to a pillar in the plurality of pillars;

for each data point in each pillar, modifying the 3D spatial coordinates to be relative to a center of the pillar;

generating a pseudo-image based on the plurality of pillars;

detecting the plurality of objects in the 3D space surrounding the sensor based on an analysis of the pseudo-image; and

operating the vehicle based upon the detecting of the plurality of objects in the 3D space surrounding the vehicle.

2. The system of claim 1 , further comprising:

determining whether a first count of a plurality of non-empty pillars exceeds a first threshold value P;

in accordance with a determination that the first count of the plurality of non-empty pillars exceeds the first threshold value P, selecting P non-empty pillars from the plurality of non-empty pillars;

in accordance with a determination that the first count of the plurality of non-empty pillars is less than the first threshold value P:

generating a second subset of pillars, wherein a sum of the first count and a second count of the second subset of pillars is equal to P;

selecting P non-empty pillars from the plurality of non-empty pillars and the second subset of pillars;

for each non-empty pillar of the P non-empty pillars:

determining whether a third count of data points assigned to the each non-empty pillar exceeds a second threshold value N;

in accordance with a determination that the third count of the data points assigned to the each non-empty pillar exceeds the second threshold value N, selecting N data points;

in accordance with a determination that the third count of the data points in the each non-empty pillar is less than the second threshold value N, assigning the each non-empty pillar a plurality of zero coordinate data points, wherein the sum of a fourth count of the plurality of zero coordinates and the third count equals N.

3. The system of claim 2 , wherein the first threshold value P is adaptive.

4. The system of claim 2 , wherein the second threshold value N is adaptive.

5. The system of claim 1 , wherein each of the data points is represented by the modified 3D spatial coordinates and a cylindrical Euclidean distance from the sensor to the data point.

6. The system of claim 5 , wherein each of the data points is, is represented by the modified 3D spatial coordinates, the cylindrical Euclidean distance, a reflectance, and a time stamp.

7. The system of claim 1 , wherein generating the pseudo-image comprises processing the plurality of pillars using a pillar processing module to generate the pseudo-image.

8. The system of claim 1 , wherein detecting the one or more objects in the 3D space surrounding the sensor based on the analysis of the pseudo image comprises:

processing the pseudo-image using a two-dimensional (2D) backbone neural network to generate an intermediate output that characterizes features of the pseudo-image; and

processing the intermediate output using an object detection neural network to detect the one or more objects in the 3D space surrounding the sensor.

9. A method comprising:

receiving a set of measurements from a sensor of a vehicle, the set of measurements comprising a plurality of data points that represent a plurality of objects in a three-dimensional (3D) space surrounding the vehicle, wherein each data point of the plurality of data points is represented by a set of 3D spatial coordinates;

dividing the 3D space into a plurality of pillars, wherein each pillar of the plurality of pillars comprises a slice of the 3D space that extends from a respective portion of a two-dimensional (2D) plane of the 3D space;

assigning each data point of the plurality of data points to a pillar in the plurality of pillars;

for each data point in each pillar, modifying the 3D spatial coordinates to be relative to a center of the pillar;

generating a pseudo-image based on the plurality of pillars;

detecting the plurality of objects in the 3D space surrounding the sensor based on an analysis of the pseudo-image; and

operating the vehicle based upon the detecting of the plurality of objects in the 3D space surrounding the vehicle,

wherein the method is performed by one or more special-purpose computing devices.

10. The method of claim 9 , further comprising:

determining whether a first count of a plurality of non-empty pillars exceeds a first threshold value P;

in accordance with a determination that the first count of the plurality of non-empty pillars exceeds the first threshold value P, selecting P non-empty pillars from the plurality of non-empty pillars;

in accordance with a determination that the first count of the plurality of non-empty pillars is less than the first threshold value P:

generating a second subset of pillars, wherein a sum of the first count and a second count of the second subset of pillars is equal to P;

selecting P non-empty pillars from the plurality of non-empty pillars and the second subset of pillars;

for each non-empty pillar of the P non-empty pillars:

determining whether a third count of data points assigned to the each non-empty pillar exceeds a second threshold value N;

in accordance with a determination that the third count of the data points assigned to the each non-empty pillar exceeds the second threshold value N, selecting N data points;

in accordance with a determination that the third count of the data points in the each non-empty pillar is less than the second threshold value N, assigning the each non-empty pillar a plurality of zero coordinate data points, wherein the sum of a fourth count of the plurality of zero coordinates and the third count equals N.

11. The method of claim 9 , wherein the first threshold value P is adaptive.

12. The method of claim 9 , wherein each data point is represented by the modified 3D spatial coordinates a cylindrical Euclidean distance from the sensor to the data point.

13. The method of claim 12 , wherein each of the data points is is represented by the modified 3D spatial coordinates, the cylindrical Euclidean distance, a reflectance, and a time stamp.

14. The method of claim 9 , wherein generating the pseudo-image comprises processing the plurality of pillars using a pillar processing module to generate the pseudo-image.

15. The method of claim 9 , wherein detecting the one or more objects in the 3D space surrounding the sensor based on the analysis of the pseudo image comprises:

processing the pseudo-image using a two-dimensional (2D) backbone neural network to generate an intermediate output that characterizes features of the pseudo-image; and

processing the intermediate output using an object detection neural network to detect the one or more objects in the 3D space surrounding the sensor.

16. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause performance of the method recited in claim 9 .