Detecting hazards based on disparity maps using computer vision for autonomous machine systems and applications
In examples, system and methods for stereo disparity based hazard detection for autonomous machine applications are presented. Example embodiments may assist an ego-machine in detecting hazards within its path of travel. The systems and methods may use disparity between a stereo pair of images to generate a baseline path disparity model and further identify hazards from detected disparities that deviate from that path disparity model. A disparity map for the image pair is constructed in which each pixel represents a disparity for a corresponding element of the image captured. Blockwise division may be optionally used to subdivide the disparity map into a plurality of smaller disparity maps, each corresponding to a block of pixels of the disparity map. A V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.
1 . At least one processor comprising:
one or more circuits to:
generate a disparity map based at least on sensor data obtained using two or more sensors of an ego-machine, the two or more sensors having at least partially overlapping fields of view that include at least a portion of a path of the ego-machine;
compute, based at least on the disparity map, one or more updated disparity maps including a path disparity model;
classify one or more pixels of the one or more updated disparity maps in a region of interest that includes the path of the ego-machine as one or more hazard pixels in response to a difference in disparity between the path disparity model and the one or more pixels exceeding a disparity threshold;
apply a clustering algorithm to the one or more pixels classified as the one or more hazard pixels to generate one or more clusters of hazard pixels; and
identify one or more hazards on the path of the ego-machine as being represented by the sensor data based at least on correlating the one or more clusters of hazard pixels with the sensor data.
2 . The at least one processor of claim 1 , wherein the one or more updated disparity maps comprise a first axis that corresponds to a disparity and a second axis that corresponds to an image row of an image represented by the sensor data.
3 . The at least one processor of claim 1 , wherein the disparity threshold for at least a portion of the one or more updated disparity maps increases as distance from the two or more sensors decreases.
4 . The at least one processor of claim 1 , wherein the correlating of the one or more clusters of hazard pixels includes generating a bounding shape corresponding to a location of the one or more clusters of hazard pixels in the sensor data.
5 . The at least one processor of claim 1 , wherein the computation of the one or more updated disparity maps includes applying a line fitting algorithm to the disparity map to compute the path disparity model.
6 . The at least one processor of claim 1 , wherein the clustering algorithm comprises a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm.
7 . The at least one processor of claim 1 , wherein the disparity threshold varies as a function of distance from the two or more sensors.
8 . The at least one processor of claim 1 , wherein the two or more sensors include at least one of a stereo camera, a monocular camera, a surround camera, a fisheye camera, or a wide view camera.
9 . The at least one processor of claim 1 , wherein the one or more circuits are further to:
identify the region of interest represented by initial sensor data; and
filter the initial sensor data to generate the sensor data representative of the region of interest.
10 . The at least one processor of claim 1 , wherein the one or more updated disparity maps include a plurality of updated disparity maps, at least one updated disparity map of the plurality of updated disparity maps corresponding to a block of pixels of the disparity map, and further wherein a respective disparity road model is generated for each updated disparity map of the plurality of updated disparity maps.
11 . The at least one processor of claim 10 , wherein a first block of pixels corresponding to a location closer to the two or more sensors of the ego-machine is larger than a second block of pixels corresponding to a location further from the two or more sensors.
12 . The at least one processor of claim 1 , wherein the path comprises at least one of:
a paved road;
an unpaved road;
a highway;
a driveway;
a portion of a parking lot;
a trail;
a track;
a walking path;
a delineated portion of an environment; or
a runway.
13 . The at least one processor of claim 1 , wherein the at least one processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
14 . A system comprising:
a plurality of sensors having at least partially overlapping fields of view; and
one or more processors comprising processing circuitry to:
generate a disparity map corresponding to at least a portion of a path of an ego-machine based at least on sensor data obtained using the plurality of sensors;
classify one or more pixels of the disparity map in a region of interest that includes a path of the ego-machine as one or more hazard pixels in response to a disparity difference of the one or more pixels to a path disparity model computed from the disparity map exceeding a disparity threshold;
cluster the one or more hazard pixels to define one or more hazard pixel clusters;
determine one or more locations of one or more hazards on the path of the ego-machine based at least on correlating the one or more hazard pixel clusters with the sensor data; and
perform one or more operations based at least on the one or more locations of the one or more hazards on the path of the ego-machine.
15 . The system of claim 14 , wherein the plurality of sensors include at least one of a stereo camera, a monocular camera, a surround camera, a fisheye camera, or a wide view camera.
16 . The system of claim 14 , wherein the disparity map comprises a first axis that corresponds to a disparity and a second axis that corresponds to an image row of an image represented by the sensor data.
17 . The system of claim 14 , wherein the disparity threshold for at least a portion of the disparity map increases as distance from the plurality of sensors decreases.
18 . The system of claim 14 , wherein the processing circuitry generates a bounding shape corresponding to a location of the one or more hazard pixel clusters in the sensor data.
19 . The system of claim 14 , wherein the processing circuitry applies a line fitting algorithm to the disparity map to compute the path disparity model.
20 . The system of claim 14 , wherein the processing circuitry applies a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm to cluster the one or more hazard pixels.
21 . The system of claim 14 , wherein the disparity threshold varies for each image row of the disparity map as a function of distance from the plurality of sensors.
22 . The system of claim 14 , wherein the processing circuitry is further to:
identify the region of interest represented by initial sensor data; and
filter the initial sensor data to generate the sensor data representative of the region of interest.
23 . The system of claim 14 , wherein the disparity map include a plurality of updated disparity maps each corresponding to a block of pixels of the disparity map, and further wherein a respective path disparity model is generated for each updated disparity map of the plurality of updated disparity maps, wherein a first block of pixels corresponding to a location closer to the plurality of sensors of the ego-machine is larger than a second block of pixels corresponding to a location further from the plurality of sensors.
24 . The system of claim 14 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
25 . A method comprising:
generating a disparity map having a path disparity model based at least on sensor data obtained using two or more sensors of an ego-machine having at least partially overlapping fields;
classifying one or more pixels of the disparity map in a region of interest that includes a path of the ego-machine as one or more hazard pixels in response to a disparity distance of the one or more pixels of the disparity map to the path disparity model exceeding a disparity threshold; and
determining a location of one or more hazard objects on the path of the ego-machine corresponding to the one or more pixels classified as the one or more hazard pixels.
26 . The method of claim 25 , further comprising:
clustering the one or more pixels classified as the one or more hazard pixels to define the one or more hazard objects.