Multi-view deep neural network for LiDAR perception
A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.
1 . One or more processors comprising one or more circuits to:
receive initial classification data generated using a first representation of sensor data and representing one or more classifications;
generate refined classification data based at least on a neural network processing a projected representation of the initial classification data and a second representation of the sensor data different from the first representation of the sensor data; and
cause performance of one or more operations corresponding to a machine based at least on the refined classification data.
2 . The one or more processors of claim 1 , wherein the one or more circuits are further to generate the projected representation of the initial classification data based at least on projecting one or more detected three-dimensional locations labeled with one or more classifications extracted based at least on a range image corresponding to the first representation of the sensor data.
3 . The one or more processors of claim 1 , wherein the one or more circuits are further to generate the projected representation of the initial classification data based at least on projecting a representation of labeled object geometry generated based at least on associating the first representation of the sensor data with the initial classification data.
4 . The one or more processors of claim 1 , wherein the one or more circuits are further to generate the projected representation of the initial classification data based at least on transforming a semantically labeled range image associating the first representation of the sensor data with the initial classification data.
5 . The one or more processors of claim 1 , wherein the one or more circuits are further to generate the refined classification data based at least on the neural network processing one or more first channels comprising one or more height values represented by the second representation of the sensor data and one or more second channels comprising one or more view transformed classifications represented by the projected representation of the initial classification data.
6 . The one or more processors of claim 1 , wherein the one or more circuits are further to generate the refined classification data based at least on the neural network processing at least one of one or more view transformed confidence maps or one or more view transformed segmentation masks corresponding to the projected representation of the initial classification data.
7 . The one or more processors of claim 1 , wherein the first representation of the sensor data comprises one or more range image images corresponding to a point cloud, and the second representation of the sensor data comprises one or more height maps corresponding to the point cloud.
8 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
9 . A method comprising:
receiving initial classification data generated using sensor data and representing one or more classifications;
generating refined classification data based at least on a neural network processing a projected representation of one or more labeled points that associate the sensor data with the initial classification data; and
causing performance of one or more operations corresponding to a machine based at least on the refined classification data.
10 . The method of claim 9 , further comprising generating the one or more labeled points based at least on labeling one or more detected three-dimensional locations with one or more classifications extracted based at least on a range image corresponding to the sensor data.
11 . The method of claim 9 , further comprising generating the one or more labeled points based at least on associating one or more per-pixel classifications corresponding to the initial classification data with one or more corresponding pixels of a range image corresponding to the sensor data.
12 . The method of claim 9 , further comprising generating the projected representation of the one or more labeled points based at least on transforming a semantically labeled range image associating the sensor data with the initial classification data.
13 . The method of claim 9 , further comprising generating the refined classification data based at least on the neural network processing one or more first channels comprising one or more height values represented by the sensor data and one or more second channels comprising one or more view transformed classifications represented by the projected representation of the one or more labeled points.
14 . The method of claim 9 , further comprising generating the refined classification data based at least on the neural network processing at least one of one or more view transformed confidence maps or one or more view transformed segmentation masks corresponding to the initial classification data.
15 . The method of claim 9 , the initial classification data generated using a first representation of the sensor data comprising one or more range images corresponding to a point cloud, further comprising generating the refined classification data based at least on the neural network further processing a second representation of the sensor data comprising one or more height maps corresponding to the point cloud.
16 . The method of claim 9 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
17 . A system comprising one or more processing units to cause performance of one or more operations corresponding to a machine based at least on refined classification data, the refined classification data generated based at least on a neural network processing initial classification data and a first representation of sensor data, the initial classification data generated using a second representation of the sensor data different from the first representation of the sensor data.
18 . The system of claim 17 , wherein the one or more processing units are further to generate the refined classification data based at least on the neural network processing a projected representation of the initial classification data.
19 . The system of claim 17 , wherein the one or more processing units are further to generate the refined classification data based at least on the neural network processing a projected representation of the initial classification data generated based at least on projecting one or more detected three-dimensional locations labeled with the initial classification data.
20 . The system of claim 17 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.