Dynamic object detection using LiDAR data for autonomous machine systems and applications
In various examples, systems and methods of the present disclosure detect and/or track objects in an environment using projection images generated from LiDAR. For example, a machine learning model—such as a deep neural network (DNN)—may be used to compute a motion mask indicative of motion corresponding to points representing objects in an environment. Various input channels may be provided as input to the machine learning model to compute a motion mask. One or more comparison images may be generated based on comparing depth values projected from a current range image to a coordinate space of a previous range image to depth values of the previous range image. The machine learning model may use the one or more projection images, the one or more comparison images, and/or the one or more range images to compute a motion mask and/or a motion vector output representation.
1. At least one processor comprising:
one or more circuits to:
generate, using sensor data obtained using one or more sensors of an ego-machine, a first range image corresponding to a first time and a second range image corresponding to a second time subsequent to the first time;
generate a first projected image based at least on projecting first depth values from the first range image to a first coordinate space of the second range image and a second projected image based at least on projecting second depth values from the second range image to a second coordinate space of the first range image;
compute, from the first projected image and the second projected image, input data to a deep neural network (DNN), the input data indicating differences in depth between the first range image and the second range image and differences in spatial correspondences of scene points across the first coordinate space and the second coordinate space; and
compute, based at least on the DNN processing the input data, output data indicative of motion at one or more pixels of the second range image.
2. The at least one processor of claim 1 , further comprising one or more circuits to:
generate a comparison image based at least on part on comparing the second depth values from the second range image in the second coordinate space of the first range image to depth values of the first range image; and
wherein the input data is computed from the comparison image.
3. The at least one processor of claim 1 , the input data includes a first representation of one or more regions of the first projected image and a second representation of one or more regions of the second projected image.
4. The at least one processor of claim 1 , wherein the first projected image and the second projected image are generated using tracked ego-motion of the ego-machine between the first time and the second time.
5. The at least one processor of claim 1 , wherein the input data includes a first channel corresponding to a first distance between a first location for the ego-machine at the first time and a first (three-dimensional) 3D point that projects to a same pixel as a second 3D point when viewed from a second location for the ego-machine at the second time, and a second channel corresponding to a second distance between the second location and the second 3D point.
6. The at least one processor of claim 1 , wherein the output data computed using the DNN is representative of a motion mask, the motion mask having one or more first values being indicative of one or more first objects depicted using the second range image being in motion at the second time and having one or more second values being indicative of one or more second objects depicted using the second range image being static at the second time.
7. The at least one processor of claim 6 , wherein the motion mask comprises, for each of the one or more pixels of the second range image, a confidence value between 0 and 1, the confidence value being indicative of whether a feature or object represented using a respective pixel of the one or more pixels corresponds to an object in motion at the second time.
8. The at least one processor of claim 1 , wherein the one or more circuits are further to perform, using the output data indicative of motion, one or more control operations corresponding to the ego-machine based at least on the motion at the one or more pixels of the second range image.
9. The at least one processor of claim 1 , wherein the DNN is trained to predict a likelihood that one or more pixels correspond to a static or a dynamic object.
10. The at least one processor of claim 1 , wherein the at least one processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
11. A method comprising:
generating, using a sensor of an ego-machine, a first range image at a first time and a second range image at a second time subsequent the first time;
generating a first projected image based at least on projecting one or more first depth values from the first range image to a first coordinate space of the second range image;
generating a second projected image based at least on projecting one or more second depth values from the second range image to a second coordinate space of the first range image;
computing, from the first projected image and the second projected image, input data to a deep neural network (DNN), the input data indicating differences in depth between the first range image and the second range image and differences in spatial correspondences of scene points across the first coordinate space and the second coordinate space; and
computing, based at least on a deep neural network (DNN) the DNN processing the input data, output data indicative of motion at one or more pixels of the second range image.
12. The method of claim 11 , further comprising:
generating a comparison image based at least in part on comparing the second depth values from the second range image in the second coordinate space of the first range image to depth values of the first range image,
wherein the input data is computed from the comparison image.
13. The method of claim 11 , wherein the input data includes a first representation of one or more regions of the first projected image and a second representation of one or more regions of the second projected image.
14. The method of claim 11 , wherein the generating the first projected image and the second projected image are based at least on tracked ego-motion of the ego-machine between the first time and the second time.
15. A system comprising:
one or more processors to perform operations including:
generating, using a sensor of an ego-machine, a first range image at a first time and a second range image at a second time subsequent the first time;
generating a first projected image based at least on projecting one or more first depth values from the first range image to a first coordinate space of the second range image;
generating a second projected image based at least on projecting one or more second depth values from the second range image to a second coordinate space of the first range image;
computing, from the first projected image and the second projected image, input data to a deep neural network (DNN), the input data indicating differences in depth between the first range image and the second range image and differences in spatial correspondences of scene points across the first coordinate space and the second coordinate space; and
computing, using the DNN having the input data, output data indicative of motion at one or more pixels of the second range image.
16. The system of claim 15 , wherein the operations further include:
generating a comparison image based at least on part on comparing the second depth values from the second range image in the second coordinate space of the first range image to depth values of the first range image,
wherein the computing the input data is computed from the comparison image.
17. The system of claim 15 , wherein, for each range image of n number of range images prior to the second range image, a respective first projected image, second projected image, and comparison image are generated using the second range image.
18. The system of claim 15 , wherein the generating the first projected image and the second projected image are based at least on tracked ego-motion of the ego-machine between the first time and the second time.
19. The system of claim 15 , wherein the operations further include computing, based at least on the output data computed using the DNN, one or more object detections.
20. The system of claim 15 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.