Detecting hazards based on disparity maps using machine learning for autonomous machine systems and applications
In various examples, systems and methods for machine learning based hazard detection for autonomous machine applications using stereo disparity are presented. Disparity between a stereo pair of images is used to generate a path disparity model. Using the path disparity model, a machine learning model can recognize when a pixel in the first image corresponds to a pixel in the second image even though the pixel in the two images does not have identical characteristics. Similarities in extracted feature vectors can be computed and represented by a vector similarity metric that is input to a machine learning classifier, along with feature information extracted from the stereo image pair, to differentiate hazard pixels from non-hazard pixels. In some embodiments, a V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.
1 . One or more processors comprising:
one or more circuits to:
generate a disparity map indicative of disparities between pixels of a first image generated using a first sensor and pixels of a second image generated using a second sensor, the first sensor and the second sensor having at least partially overlapping fields of view including at least a portion of a path of an ego-machine;
generate a path disparity model based at least on the disparity map;
extract, based at least on executing one or more first machine learning models, a first feature map of the first image and a second feature map of the second image;
compute, based at least on executing the one or more first machine learning models, a first feature vector corresponding to a first pixel of the first image based at least on the first feature map of the first image and a second feature vector corresponding to a second pixel of the second image based at least on the second feature map of the second image, the second pixel corresponding to the first pixel;
determine a similarity between the first feature vector and the second feature vector that is associated with the first feature vector based at least on applying the path disparity model to the first pixel;
classify at least one of the first pixel or the second pixel as corresponding to one or more hazards based at least on the first feature vector and the second feature vector; and
perform one or more operations for controlling the ego-machine along the path based at least on the one or more hazards.
2 . The one or more processors of claim 1 , wherein at least one of the first pixel or the second pixel is classified based at least on the one or more first machine learning models processing the first feature vector and the second feature vector; or
wherein at least one of the first pixel or the second pixel is classified based at least on one or more second machine learning models processing the first feature vector and the second feature vector, wherein the one or more second machine learning models are different from the one or more first machine learning models.
3 . The one or more processors of claim 1 , wherein at least one of the first pixel or the second pixel is classified based at least on the one or more first machine learning models determining an indication of confidence that the first pixel or the second pixel corresponds to a hazard pixel, and the one or more first machine learning models are end-to-end trained.
4 . The one or more processors of claim 1 , wherein the one or more circuits are further to:
classify one or more other pixels of the first image as corresponding to the one or more hazards;
apply a clustering algorithm to the first pixel and the one or more other pixels to generate one or more clusters of pixels; and
define one or more locations of the one or more hazards based at least on the one or more clusters of pixels.
5 . The one or more processors of claim 4 , wherein the one or more locations are defined by generating one or more bounding shapes around at least one respective cluster of pixels of the one or more clusters of pixels.
6 . The one or more processors of claim 1 , wherein at least one of the first pixel or the second pixel is classified based at least on a combined feature vector, wherein the combined feature vector is generated by combining the first feature vector and the second feature vector.
7 . The one or more processors of claim 1 , wherein the first pixel is within a region of interest that includes the path of the ego machine.
8 . The one or more processors of claim 1 , wherein the one or more first machine learning models are trained based at least on ground truth data that comprises classification labels corresponding to a drivable region, a non-drivable region, and hazards.
9 . The one or more processors of claim 8 , wherein a delineation of the non-drivable region from the drivable region is determined based at least on freespace information.
10 . The one or more processors of claim 1 , wherein the path disparity model is represented in a disparity space including a first axis corresponding to disparity and a second axis corresponding to an image row of at least one of the first image or the second image.
11 . The one or more processors of claim 1 , wherein the generation of the path disparity model includes applying a line fitting algorithm to at least a portion of the disparity map.
12 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center, or
a system implemented at least partially using cloud computing resources.
13 . A system comprising:
a plurality of sensors to generate sensor data;
one or more processing units comprising processing circuitry to:
generate a disparity map indicative of disparities between pixels of a pair of stereo images generated using the plurality of sensors, the plurality of sensors comprising a first sensor and a second sensor having at least partially overlapping fields of view including at least a portion of a path of an ego-machine;
generate a path disparity model based at least on the disparity map;
extract, based at least on executing one or more first machine learning models, a first feature map of a first image of the pair of stereo images and a second feature map of a second image of the pair of stereo images;
compute, based at least on executing the one or more first machine learning models, a first feature vector for a first pixel of the first image of the pair of stereo images based at least on the first feature map of the first image of the pair of stereo images and a second feature vector for a second pixel of the second image of the pair of stereo images based at least on the second feature map of the second image of the pair of stereo images, the second pixel of the second image of the pair of stereo images corresponding to the first pixel of the pair of stereo images;
apply the path disparity model to the first pixel to determine an association between the first feature vector and the second feature vector;
classify at least one of the first pixel or the second pixel as a hazard pixel corresponding to a hazard object on the path of the ego-machine based at least on the association between the first feature vector and the second feature vector; and
generate an output indicating a presence of the hazard object on the path of the ego-machine.
14 . The system of claim 13 , wherein the processing circuitry is further to:
cluster one or more pixels classified as the hazard pixel to generate one or more clusters of pixels; and
define a location of the hazard object based at least on the one or more clusters of pixels.
15 . The system of claim 13 , wherein the determination of the association between the first feature vector and the second feature vector is performed, at least in part, by projecting the first feature vector into the second image at a pixel location determined based at least on the path disparity model.
16 . The system of claim 13 , wherein the association between the first feature vector and the second feature vector is determined based at least on the one or more first machine learning models processing the first feature vector and the second feature vector; or
wherein the association between the first feature vector and the second feature vector is determined based at least on one or more second machine learning models processing the first feature vector and the second feature vector, wherein the one or more second machine learning models are different than the one or more first machine learning models.
17 . The system of claim 13 , wherein the generation of the path disparity model includes applying a line fitting algorithm to at least a portion of the disparity map.
18 . The system of claim 13 , wherein at least one of the first pixel or the second pixel is classified based at least on the one or more first machine learning models determining an indication of confidence that the first pixel or the second pixel corresponds to a hazard pixel, and the one or more first machine learning models are trained using end-to-end trained.
19 . The system of claim 13 , wherein at least one of the first pixel or the second pixel is classified based at least on a combined feature vector, wherein the combined feature vector is generated by combining the first feature vector and the second feature vector.
20 . The system of claim 13 , wherein the one or more first machine learning models are trained based at least on ground truth data that comprises classification labels corresponding to a drivable region, a non-drivable region, and hazards.
21 . The system of claim 13 , wherein the processing circuitry is further to filter the pair of stereo images to a region of interest that includes the path of the ego-machine.
22 . The system of claim 13 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
23 . A method comprising:
determining a location of a first pixel of a first image corresponding to a hazard object on a path of an ego-machine based at least on classifying the first pixel as a hazard pixel, wherein the first pixel is classified as a hazard pixel based at least on a first feature vector corresponding to the first pixel of the first image and a second feature vector corresponding to a second pixel of a second image corresponding to the first pixel, the first feature vector and the second feature vector determined to correspond to one another based at least on applying a path disparity model to the first pixel, the path disparity model being computed based at least on a disparity map generated based at least on the first image and the second image; and
based at least on the location of the first pixel corresponding to the hazard object on the path of the ego-machine,
performing one or more operations for controlling the ego-machine along the path; or
generating an output indicating a presence of the hazard object on the path of the ego-machine.