Generating point clouds with appended features for use in perception
The technologies described herein relate to a radar system that is configured to generate point clouds based upon radar tensors generated by the radar system. More specifically, the radar system is configured to identify bins in radar tensors that correspond to objects in an environment of the radar system, and to use energy values in other bins to construct a point cloud. A computing system detects objects in an environment of the radar system based upon the point clouds.
1 . A radar system comprising:
processing circuitry that is configured to perform acts comprising:
generating a radar tensor based upon radar returns detected by the radar system, wherein the radar tensor has bins that comprise energy values;
identifying a first bin in the bins that corresponds to an object in an environment of the radar system, wherein the first bin has a first energy value that satisfies a predefined threshold, and further wherein the first bin is identified as corresponding to the object in the environment based upon the first energy value;
computing a point cloud entry based upon the first energy value;
selecting a second bin in the bins based upon the first bin, wherein the second bin is adjacent to the first bin and has a second energy value that fails to satisfy the predefined threshold;
computing a value of a feature based upon the second energy value of the second bin, wherein computing the value of the feature comprises providing the second energy value and neighbor energy values from additional bins in a predefined neighborhood of the first bin to a submanifold sparse convolutional neural network, and wherein the submanifold sparse convolutional neural network outputs the value of the feature based on the second energy value and the neighbor energy values;
appending the value of the feature to the point cloud entry to generate an updated point cloud entry; and
transmitting a point cloud that comprises the updated point cloud entry to a computing system that is configured to detect and identify objects in the environment of the radar system based upon the point cloud,
wherein the computing system comprises a perception neural network, and wherein weight matrices of the submanifold sparse convolutional neural network are co-optimized with weight matrices of the perception neural network using joint training.
2 . The radar system of claim 1 , wherein the second bin comprises a predetermined number of bins that surround the first bin.
3 . The radar system of claim 2 , wherein the submanifold sparse convolutional neural network outputs values for several features based upon the second energy value, and further wherein the values for the several features are appended to the point cloud entry to generate the updated point cloud entry.
4 . The radar system of claim 1 , wherein the second bin is selected based upon a location of the second bin relative to the first bin in the radar tensor.
5 . The radar system of claim 1 , the acts further comprising:
subsequent to the first bin being identified, defining the predefined neighborhood of the first bin, wherein the predefined neighborhood of the first bin comprises a plurality of bins, wherein the plurality of bins comprises the second bin, and further wherein the plurality of bins have a plurality of energy values, wherein the plurality of bins immediately surround the first bin; and
selecting the plurality of bins based upon the plurality of bins being included in the predefined neighborhood of the first bin, wherein the value of the feature is computed based upon the plurality of energy values of the plurality of bins.
6 . The radar system of claim 5 , wherein a plurality of feature values are computed based upon the plurality of energy values of the plurality of bins.
7 . The radar system of claim 1 being included in an autonomous vehicle (AV).
8 . The radar system of claim 1 , wherein an autonomous vehicle (AV) performs a driving maneuver based upon output of the perception neural network.
9 . The radar system of claim 1 , wherein the point cloud entry comprises values for location, a value for velocity, and a value for magnitude of a signal corresponding to the first energy value.
10 . The radar system of claim 1 , wherein selecting the second bin comprises selecting a plurality of bins from a predefined neighborhood of the first bin.
11 . A method performed by a radar system, the method comprising:
generating a radar tensor based upon radar returns detected by the radar system, wherein the radar tensor has bins that comprise energy values;
identifying a first bin in the bins that corresponds to an object in an environment of the radar system, wherein the first bin has a first energy value that satisfies a predefined threshold, and further wherein the first bin is identified as corresponding to the object in the environment based upon the first energy value; computing a point cloud entry based upon the first energy value;
selecting a second bin in the bins based upon the first bin, wherein the second bin is adjacent to the first bin and has a second energy value that fails to satisfy the predefined threshold;
computing a value of a feature based upon the second energy value of the second bin, wherein computing the value of the feature comprises providing the second energy value and neighbor energy values from additional bins in a predefined neighborhood of the first bin to a submanifold sparse convolutional neural network, and wherein the submanifold sparse convolutional neural network outputs the value of the feature based on the second energy value and the neighbor energy values;
appending the value of the feature to the point cloud entry to generate an updated point cloud entry; and
transmitting a point cloud that comprises the updated point cloud entry to a computing system that is configured to detect and identify objects in the environment of the radar system based upon the point cloud,
wherein the computing system comprises a perception neural network, and wherein weight matrices of the submanifold sparse convolutional neural network are co-optimized with weight matrices of the perception neural network using joint training.
12 . The method of claim 11 , wherein the second bin comprises a predetermined number of bins that surround the first bin.
13 . The method of claim 12 , wherein the submanifold sparse convolutional neural network outputs values for several features based upon the second energy value, and further wherein the values for the several features are appended to the point cloud entry to generate the updated point cloud entry.
14 . The method of claim 11 , wherein the second bin is selected based upon a location of the second bin relative to the first bin in the radar tensor.
15 . The method of claim 11 , further comprising:
subsequent to the first bin being identified, defining the predefined neighborhood of the first bin, wherein the predefined neighborhood of the first bin comprises a plurality of bins, wherein the plurality of bins comprises the second bin, and further wherein the plurality of bins have a plurality of energy values, wherein the plurality of bins immediately surround the first bin; and
selecting the plurality of bins based upon the plurality of bins being included in the predefined neighborhood of the first bin, wherein the value of the feature is computed based upon the plurality of energy values of the plurality of bins.
16 . The method of claim 15 , wherein a plurality of feature values are computed based upon the plurality of energy values of the plurality of bins.
17 . The method of claim 11 being included in an autonomous vehicle (AV).
18 . The method of claim 11 , wherein an autonomous vehicle (AV) performs a driving maneuver based upon output of the perception neural network.
19 . The method claim 11 , wherein selecting the second bin comprises selecting a plurality of bins from a predefined neighborhood of the first bin.
20 . An autonomous vehicle comprising:
a radar system;
a mechanical system; and
a computing system that is in communication with the radar system and the mechanical system, wherein the radar system is configured to perform acts comprising:
generating a radar tensor based upon radar returns detected by the radar system, wherein the radar tensor has bins that comprise energy values;
identifying a first bin in the bins that corresponds to an object in an environment of the radar system, wherein the first bin has a first energy value that satisfies a predefined threshold, and further wherein the first bin is identified as corresponding to the object in the environment based upon the first energy value;
computing a point cloud entry based upon the first energy value;
selecting a second bin in the bins based upon the first bin, wherein the second bin is adjacent to the first bin and has a second energy value that fails to satisfy the predefined threshold;
computing a value of a feature based upon the second energy value of the second bin, wherein computing the value of the feature comprises providing the second energy value and neighbor energy values from additional bins in a predefined neighborhood of the first bin to a submanifold sparse convolutional neural network, and wherein the submanifold sparse convolutional neural network outputs the value of the feature based on the second energy value and the neighbor energy values;
appending the value of the feature to the point cloud entry to generate an updated point cloud entry; and
transmitting a point cloud that comprises the updated point cloud entry to the computing system;
wherein the computing system is configured to perform acts comprising:
detecting and identifying an object in an environment of the AV based upon the point cloud; and
controlling the mechanical system based upon the detected and identified object,
wherein the computing system comprises a perception neural network, and wherein weight matrices of the submanifold sparse convolutional neural network are co-optimized with weight matrices of the perception neural network using joint training.