Object detection and state estimation from deep learned per-point radar representations
A current set of radar data may be combined with previous sets of radar data to create a combined set of radar data. Each of these sets of radar data and the combined set of radar data may include various data points and each of these data points may be associated with certain specific features or classifications. The set of combined data may then be pre-processed to identify distances to associate with certain data points, locations to associate with those data points, and velocities associated with the data points such that each of the respective data points may be mapped and tagged with data that identifies identify positions, velocities, and other physical details of the respective data points. This pre-processed data may then be processed by a machine learning process to identify kinematic information that may then be provided to a tracking system of an AV.
1 . A method for evaluating radar data, the method comprising:
generating generated current data from a currently received radar signals;
pre-processing both the generated current data and data associated with a set of prior data;
generating a combined dataset that includes data points associated with a combination of the generated current data and the set of prior data, wherein locations of the data points of the combined dataset are identified;
extracting per-point feature data from a set of accessed data based on the identified locations of the data points;
training a machine learning (ML) process;
generating per-point outputs of the ML process based on the ML process receiving the per-point feature data and the combined dataset, the per-point outputs associated with the per-point feature data and the data points of the combined data set, wherein the per-point feature data and the data points of the combined data set are associated with point-wise representations;
organizing the per-point feature data from the set of accessed data and information from the combined dataset into a set of formatted data that associates the data points with respective per-point features;
extracting data from the set of formatted data a set of shallow features, wherein each feature on the set of shallow features corresponds to a single data point of the data points, wherein the set of shallow features does not include data that associates the single point with another data point of the data points;
associating each feature of the set of shallow features to a specific grid cell of a grid around the autonomous vehicle to generate a grid cell to point mapping;
generating a set of deep global features that associate the set of shallow features with the grid cell to point mapping, wherein the deep global features include data that links one point to other points;
post-processing the per-point outputs of the ML process to transform point-wise representations to object representations; and
using the object representations to maneuver or operate an autonomous vehicle.
2 . The method of claim 1 , further comprising concatenating the set of shallow features with the deep global features to create a set of augmented point features.
3 . The method of claim 2 , further comprising generating a set of machine learned (ML) outputs from the augmented point features.
4 . The method of claim 3 , further comprising shifting a location of the single data point toward a centroid associated with an object and the single data point.
5 . The method of claim 4 , further comprising applying a density-based clustering operation on the shifted single data point.
6 . The method of claim 5 , further comprising generating object attributes.
7 . The method of claim 6 , wherein the object attributes include or are associated with one or more objects representing, a point-to-cluster association, a bounding box, or object class.
8 . The method of claim 1 , wherein the set of deep global features includes the data that associates the single point with the other point of the data points.
9 . A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor for implementing a method for evaluating radar data, the method comprising:
generating generated current data from a currently received radar signals;
pre-processing both the generated current data and data associated with a set of prior data;
generating a combined dataset that includes data points associated with a combination of the generated current data and the set of prior data, wherein locations of the data points of the combined dataset are identified;
extracting per-point feature data from a set of accessed data based on the identified locations of the data points;
training a machine learning (ML) process;
generating per-point outputs of the ML process based on the ML process receiving the per-point feature data and the combined dataset, the per-point outputs associated with the per-point feature data and the data points of the combined data set, wherein the per-point feature data and the data points of the combined data set are associated with point-wise representations;
organizing the per-point feature data from the set of accessed data and information from the combined dataset into a set of formatted data that associates the data points with respective per-point features;
extracting data from the set of formatted data a set of shallow features, wherein each feature on the set of shallow features corresponds to a single data point of the data points, wherein the set of shallow features does not include data that associates the single point with another data point of the data points;
associating each feature of the set of shallow features to a specific grid cell of a grid around the autonomous vehicle to generate a grid cell to point mapping;
generating a set of deep global features that associate the set of shallow features with the grid cell to point mapping, wherein the deep global features include data that links one point to other points;
post-processing the per-point outputs of the ML process to transform point-wise representations to object representations; and
using the object representations to maneuver or operate an autonomous vehicle.
10 . An apparatus for evaluating radar data, the apparatus comprising:
a radar device that emits and receives radar signals;
a memory; and
a processor that executes instructions out of the memory to:
generate generated current data from the received radar signals;
pre-process both the generated current data and data associated with a set of prior data, wherein the pre-process preserves angles, topological orientations of data points, and information associated with motion vectors of objects in an area around a vehicle while reducing information associated with specific conditions under which the generated current data was generated;
generate a combined dataset that includes data points associated with a combination of the generated current data and the prior data, wherein locations of the data points of the combined dataset are identified;
extract per-point feature data from a set of accessed data based on the identified locations of the data points;
train a machine learning (ML) process;
generate per-point outputs of the ML process based on the ML process receiving the per-point feature data and the combined dataset, the per-point outputs associated with the per-point feature data and the data points of the combined data set, wherein the per-point feature data and the data points of the combined data set are associated with point-wise representations;
organize the per-point feature data from the set of accessed data and information from the combined dataset into a set of formatted data that associates the data points with respective per-point features;
extract data from the set of formatted data a set of shallow features, wherein each feature on the set of shallow features corresponds to a single data point of the data points, wherein the set of shallow features does not include data that associates the single point with another data point of the data points;
associate each feature of the set of shallow features to a specific grid cell of a grid around the autonomous vehicle to generate a grid cell to point mapping;
generate a set of deep global features that associate the set of shallow features with the grid cell to point mapping, wherein the deep global features include data that links one point to other points;
post-process the per-point outputs of the ML process to transform point-wise representations to object representations; and
use the object representations to maneuver or operate an autonomous vehicle.