Range-view LIDAR-based object detection
View Patent ↗Systems and methods for detecting and classifying objects that are proximate to an autonomous vehicle can include receiving, by one or more computing devices, LIDAR data from one or more LIDAR sensors configured to transmit ranging signals relative to an autonomous vehicle, generating, by the one or more computing devices, a data matrix comprising a plurality of data channels based at least in part on the LIDAR data, and inputting the data matrix to a machine-learned model. A class prediction for each of one or more different portions of the data matrix and/or a properties estimation associated with each class prediction generated for the data matrix can be received as an output of the machine-learned model. One or more object segments can be generated based at least in part on the class predictions and properties estimations. The one or more object segments can be provided to an object classification and tracking application.
1. A computer-implemented method of detecting objects of interest comprising:
receiving, by a computing system comprising one or more computing devices, LIDAR data comprising a plurality of LIDAR data points from one or more LIDAR sensors configured to transmit ranging signals relative to an autonomous vehicle;
generating, by the computing system, a data matrix comprising a plurality of data channels based, at least in part, on the LIDAR data, wherein at least one of the plurality of data channels within the data matrix comprises LIDAR Background Subtraction foreground data indicative of whether each of the plurality of LIDAR data points is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data received from the one or more LIDAR sensors;
inputting, by the computing system, the data matrix comprising a plurality of data channels to a machine-learned model;
receiving, by the computing system as a first output of the machine-learned model, a class prediction for each of one or more different portions of the data matrix;
receiving, by the computing system as a second output of the machine-learned model, a properties estimation associated with each class prediction generated for the data matrix;
generating, by the computing system, one or more object segments based at least in part on the class predictions and properties estimations; and
providing, by the computing system, the one or more object segments to an object classification and tracking application.
2. The computer-implemented method of claim 1 , wherein the LIDAR data comprises LIDAR sweep data corresponding to LIDAR point data received around an approximately 360 degree horizontal view around the autonomous vehicle, and wherein the method further comprises generating, by the computing system, the LIDAR sweep data based on the LIDAR data received from the one or more LIDAR sensors.
3. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises LIDAR point range data indicative of how far each of the plurality of LIDAR data points is from the one or more LIDAR sensors, and wherein at least another one of the plurality of data channels within the data matrix comprises LIDAR point height data indicative of a height above ground for each of the plurality of LIDAR data points.
4. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises intensity data indicative of an energy intensity of a returned ranging signal received back at the one or more LIDAR sensors after the one or more LIDAR sensors transmits the ranging signals relative to the autonomous vehicle.
5. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises absence of LIDAR return data indicative of data matrix cells for which no ranging signal was returned after the one or more LIDAR sensors transmits the ranging signals relative to the autonomous vehicle.
6. The computer-implemented method of claim 1 , wherein the data matrix comprises at least five data channels comprising LIDAR point range data indicative of how far each of the plurality of LIDAR data points is from the one or more LIDAR sensors, LIDAR point height data indicative of a height above ground of each of the plurality of LIDAR data points, intensity data indicative of an energy intensity of a returned ranging signal received back at the one or more LIDAR sensors after transmission, absence of LIDAR return data indicative of data matrix cells for which no ranging signal was returned after transmission by the one or more LIDAR sensors, and the LIDAR Background Subtraction foreground data indicative of whether each of the plurality of LIDAR data points is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data from the one or more LIDAR sensors.
7. The computer-implemented method of claim 1 , wherein the machine-learned model comprises a convolutional neural network.
8. The computer-implemented method of claim 1 , the method further comprising:
predicting, by the computing system, an instance segmentation for each of one or more detected instances based at least in part on the class predictions and the properties estimations; and
generating, by the computing system, a bounding box estimation for each instance segmentation based at least in part on the class predictions and the properties estimations.
9. The computer-implemented method of claim 8 , wherein predicting the instance segmentation comprises predicting the instance segmentation based at least in part on a class probability and an instance center determined based on the class predictions and the properties estimations; and
wherein generating the bounding box estimation comprises generating the bounding box estimation based at least in part on an instance center, an orientation, a width, and a height determined based at least in part on the properties estimations.
10. An object detection system comprising:
one or more processors;
a machine-learned prediction model, wherein the prediction model has been trained to receive a data matrix comprising multiple channels of LIDAR-associated data and, in response to receipt of the data matrix comprising multiple channels of LIDAR-associated data, output one or more class predictions for different portions of the data matrix; and
at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining a data matrix comprising multiple channels of LIDAR-associated data, wherein at least one of the multiple channels of LIDAR-associated data comprises LIDAR Background Subtraction foreground data indicative of whether a LIDAR data point is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data received from the one or more LIDAR sensors;
inputting the data matrix comprising multiple channels of LIDAR-associated data into the machine-learned prediction model; and
receiving, as output of the machine-learned prediction model, one or more class predictions for one or more different portions of the data matrix.
11. The object detection system of claim 10 , wherein the machine-learned prediction model has been further trained to output a properties estimation for one or more types of predicted classes identified from the one or more class predictions for the one or more different portions of the data matrix, and
wherein the operations further comprise receiving, as an output of the machine-learned prediction model, a properties estimation associated with each of the one or more class predictions for the one or more different portions of the data matrix.
12. The object detection system of claim 11 , wherein the operations further comprise generating one or more object segments based at least in part on the class predictions and the properties estimations.
13. The object detection system of claim 10 , wherein the output of the machine-learned prediction model includes a class prediction for each cell of the data matrix comprising LIDAR point data.
14. The object detection system of claim 10 , wherein the machine-learned prediction model comprises a convolutional neural network.
15. The object detection system of claim 10 , wherein the operations further comprise:
predicting an instance segmentation for each of one or more detected instances based at least in part on a class probability and an instance center generated at least in part from the class predictions and the properties estimations; and
generating a bounding box estimation for each instance segmentation based at least in part on an instance center, an orientation, a width, and a height generated at least in part from the properties estimations.
16. An autonomous vehicle comprising:
a sensor system comprising at least one LIDAR sensor configured to transmit ranging signals relative to the autonomous vehicle and to generate LIDAR data comprising a plurality of LIDAR data points; and
a vehicle computing system comprising:
one or more processors; and
at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
receiving LIDAR data from the sensor system;
generating a data matrix comprising a plurality of data channels based at least in part on the LIDAR data, wherein at least one of the plurality of data channels within the data matrix comprises LIDAR Background Subtraction foreground data indicative of whether each of the plurality of LIDAR data points is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to LIDAR data from the sensor system;
providing the data matrix comprising the plurality of data channels as input to a machine-learned model;
receiving, as a first output of the machine-learned model, a class prediction for each cell of the data matrix; and
receiving, as a second output of the machine-learned model, a properties estimation for each cell of the data matrix.
17. The autonomous vehicle of claim 16 , wherein the operations further comprise:
generating one or more object segments based at least in part on the class predictions and the properties estimations, and
wherein the object segments are used in classifying and tracking objects of interest to the operation of the autonomous vehicle.
18. The autonomous vehicle of claim 16 , wherein the plurality of data channels within the data matrix further comprise one or more of:
LIDAR point range data indicative of how far each of the plurality of LIDAR data points is from the at least one LIDAR sensor,
LIDAR point height data indicative of a height above ground of each of the plurality of LIDAR data points,
intensity data indicative of an energy intensity of a returned ranging signal received back at the at least one LIDAR sensor after the at least one LIDAR sensor transmits ranging signals relative to the autonomous vehicle, and
absence of LIDAR return data indicative of data matrix cells for which no ranging signal is returned after the at least one LIDAR sensor transmits ranging signals relative to the autonomous vehicle.
19. The autonomous vehicle of claim 16 , wherein the operations further comprise:
predicting an instance segmentation for each of one or more detected instances based at least in part on a class probability and an instance center generated at least in part from the class predictions and the properties estimations; and
generating a bounding box estimation for each instance segmentation based at least in part on an instance center, an orientation, a width, and a height generated at least in part from the properties estimations.