Systems and methods for dataset and model management for multi-modal auto-labeling and active learning
Datasets for autonomous driving systems and multi-modal scenes may be automatically labeled using previously trained models as priors to mitigate the limitations of conventional manual data labeling. Properly versioned models, including model weights and knowledge of the dataset on which the model was previously trained, may be used to run an inference operation on unlabeled data, thus automatically labeling the dataset. The newly labeled dataset may then be used to train new models, including sparse data sets, in a semi-supervised or weakly-supervised fashion.
1. A method for managing a dataset comprising:
receiving, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent;
inferring, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects;
labeling the unlabeled object with the new label to generate a third dataset; and
training an untrained network using the third dataset to generate a new model.
2. The method of claim 1 wherein the first dataset comprises data from a group of sensors.
3. The method of claim 2 wherein the group of sensors include two or more of a LiDAR sensor, a RADAR sensor, an RGB camera, monocular camera, and/or stereo camera.
4. The method of claim 1 wherein the first trained network comprises a versioned model.
5. The method of claim 4 wherein the versioned model comprises a weighted model.
6. The method of claim 4 wherein the versioned model comprises an identification of a training dataset.
7. The method of claim 1 further comprising tracking the new model as auto-labeled.
8. The method of claim 1 further comprising tracking the first trained network and first dataset.
9. The method of claim 1 wherein the untrained network is trained in a semi-supervised manner.
10. The method of claim 1 wherein the first dataset is labeled according to a first ontology.
11. The method of claim 10 wherein the first dataset is also labeled according to a second ontology.
12. An apparatus for managing a dataset comprising:
at least one processor; and
at least one memory coupled with the at least one processor and storing instructions operable, when executed by the at least one processor, to cause the apparatus to:
receive, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent;
infer, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects;
label the unlabeled object with the new label to generate a third dataset; and
train an untrained network using the third dataset to generate a new model.
13. The apparatus of claim 12 wherein the dataset comprises data from a group of sensors.
14. The apparatus of claim 13 wherein the group of sensors include two or more of a LiDAR sensor, a RADAR sensor, an RGB camera, monocular camera, and/or stereo camera.
15. The apparatus of claim 12 wherein the first trained network comprises a versioned model.
16. The apparatus of claim 15 wherein the versioned model comprises a weighted model.
17. The apparatus of claim 15 wherein the versioned model comprises an identification of a training dataset.
18. The apparatus of claim 12 wherein the new model is tracked as auto-labeled.
19. The apparatus of claim 12 wherein the untrained network is trained in a semi-supervised manner.
20. A non-transitory computer-readable medium for managing a dataset and including instructions that when executed by one or more processors cause the one or more processors to:
receive, at a device that is remotely located from an agent, sensor data collected by one or more sensors associated with the agent and a first dataset including an unlabeled object and one or more labeled objects in a set of frames captured via the one or more sensors over a period of time, the dataset and the sensor being received based on a connection bandwidth at the agent being greater than a bandwidth threshold, the one or more labeled objects being labeled by a first trained network at the agent;
inferring, at a second trained network at the device, a new label for the unlabeled object based on the one or more labeled object in the first dataset and a second dataset including one or more objects similar to the unlabeled object, the new label being different than a respective label associated each of the one or more labeled objects;
label the unlabeled object with the new label to generate a third dataset; and
train an untrained network using the third dataset to generate a new model.