Freespace detection using machine learning for autonomous systems and applications
Systems and methods are disclosed that relate to freespace detection using machine learning models. First data that may include object labels may be obtained from a first sensor and freespace may be identified using the first data and the object labels. The first data may be annotated to include freespace labels that correspond to freespace within an operational environment. Freespace annotated data may be generated by combining the one or more freespace labels with second data obtained from a second sensor, with the freespace annotated data corresponding to a viewable area in the operational environment. The viewable area may be determined by tracing one or more rays from the second sensor within the field of view of the second sensor relative to the first data. The freespace annotated data may be input into a machine learning model to train the machine learning model to detect freespace using the second data.
1 . A method comprising:
obtaining sensor data of a first sensor modality corresponding to an operational environment, the sensor data generated using one or more sensors associated with an ego-machine; and
identifying, using a machine learning model and the sensor data, freespace information indicating one or more portions of the operational environment that is navigable using the ego-machine,
wherein the machine learning model was previously trained at least by:
propagating one or more freespace labels from training data of a second sensor modality to training data of the first sensor modality to obtain one or more second freespace labels corresponding to the training data of the first sensor modality; and
updating one or more parameters of the machine learning model using the training data of the first sensor modality and the one or more second freespace labels.
2 . The method of claim 1 , wherein the sensor data is generated using one or more RADAR sensors.
3 . The method of claim 1 , further comprising:
performing one or more operations using the ego-machine based at least on the identified freespace information.
4 . The method of claim 1 , wherein the sensor data is obtained relative to the ego-machine in a rig coordinate frame.
5 . The method of claim 1 , further comprising:
obtaining, from the machine learning model, a predicted occupancy probability associated with one or more locations within the operational environment.
6 . The method of claim 5 , wherein the predicted occupancy probability is segmented into a pixel-by-pixel representation of the one or more locations within the operational environment.
7 . A computing system comprising:
a first sensor that generates first data, the first data including one or more freespace labels corresponding to freespace in an operational environment, the freespace indicating an area of the operational environment that is navigable for an ego-machine;
a second sensor that generates second data, the second data annotated with the one or more freespace labels propagated from the first data; and
a machine learning model that is trained using the second data including the one or more freespace labels to detect the freespace.
8 . The computing system of claim 7 , wherein a viewable area in the operational environment is determined relative to the second sensor by tracing one or more rays from the second sensor within a field of view of the second sensor relative to the first data.
9 . The computing system of claim 7 , wherein the second data is arranged to align with the first data prior to propagating the one or more freespace labels from the first data to the second data.
10 . The computing system of claim 7 , wherein the first data is a LiDAR point cloud obtained using a LiDAR device, the second data is RADAR data, and the second sensor includes a RADAR device.
11 . The computing system of claim 7 , wherein annotating the first data to obtain the one or more freespace labels is performed at least by user review.
12 . The computing system of claim 7 , wherein the second data is arranged to correspond with the first data prior to propagating the one or more freespace labels from the first data to the second data.
13 . The computing system of claim 7 , wherein the one or more freespace labels are determined based on at least a current status of the operational environment.
14 . The computing system of claim 13 , wherein the current status of the one or more freespace labels includes at least one of a visibility status or an occupancy status.
15 . The computing system of claim 7 , wherein the computing system comprises one or more of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting at least one virtual reality content, augmented reality content, or mixed reality content;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
16 . A processor comprising processing circuitry to perform operations comprising:
obtaining sensor data of a first sensor modality corresponding to an operational environment, the sensor data generated using one or more sensors associated with an ego-machine;
identifying, using a machine learning model, freespace corresponding to the sensor data generated using one or more sensors, wherein the machine learning model was previously trained at least by:
propagating one or more freespace labels from training data of a second sensor modality to training data of the first sensor modality to obtain one or more second freespace labels corresponding to the training data of the first sensor modality; and
updating one or more parameters of the machine learning model using the training data of the first sensor modality and the one or more second freespace labels.
17 . The processor of claim 16 , wherein the sensor data is generated using one or more RADAR sensors.
18 . The processor of claim 16 , the operations further comprising:
performing one or more operations using the ego-machine based at least on the identified freespace.
19 . The processor of claim 16 , wherein the data is obtained relative to the ego-machine in a rig coordinate frame.