Auto-labeling of driving logs using analysis-by-synthesis and unsupervised domain adaptation
Acquiring labeled data can be a significant bottleneck in the development of machine learning models that are accurate and efficient enough to enable safety-critical applications, such as automated driving. The process of labeling of driving logs can be automated. Unlabeled real-world driving logs, which include data captured by one or more vehicle sensors, can be automatically labeled to generate one or more labeled real-world driving logs. The automatic labeling can include analysis-by-synthesis on the unlabeled real-world driving logs to generate simulated driving logs, which can include reconstructed driving scenes or portions thereof. The automatic labeling can further include simulation-to-real automatic labeling on the simulated driving logs and the unlabeled real-world driving logs to generate one or more labeled real-world driving logs. The automatically labeled real-world driving logs can be stored in one or more data stores for subsequent training, validation, evaluation, and/or model management.
1. A method of automatically labeling driving logs, the method comprising:
receiving one or more unlabeled real-world driving logs, the one or more unlabeled real-world driving logs including data captured by one or more vehicle sensors;
automatically labeling the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs, the automatically labeling including:
analysis-by-synthesis on the one or more unlabeled real-world driving logs to generate one or more simulated driving logs, whereby the one or more simulated driving logs include reconstructed driving scenes or portions thereof; and
simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs; and
storing the one or more labeled real-world driving logs in one or more data stores of labeled driving logs.
2. The method of claim 1 , wherein the one or more simulated driving logs include photo-realistic renderings of an observed real-world scenario with one or more ground truth labels.
3. The method of claim 2 , wherein the one or more ground truth labels include semantic segmentation, depth, two-dimensional object detection bounding boxes, three-dimensional object detection bounding boxes, or object class labels.
4. The method of claim 1 , wherein analysis-by-synthesis includes using a simulator together with computer vision techniques or computer graphing techniques.
5. The method of claim 1 , wherein the one or more unlabeled real-world driving logs include camera data, radar data, LIDAR data, or sonar data.
6. The method of claim 1 , wherein the simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs includes:
using an unsupervised domain adaptation algorithm on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to learn a model specifically for generating labels for one or more desired tasks;
applying the model to the one or more unlabeled real-world driving logs to determine task-specific labels; and
applying the task-specific labels to the one or more unlabeled real-world driving logs to generate the one or more labeled real-world driving logs.
7. The method of claim 6 , wherein the simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs includes:
performing predictive post-processing on the generated one or more labeled real-world driving logs, whereby potentially noisy labels introduced in the simulation-to-real automatic labeling are reduced or corrected.
8. The method of claim 7 , wherein the predictive post-processing includes spatio-temporal consistencies, geometric consistencies, cross-modal consistencies, and label-specific reasoning.
9. The method of claim 7 , wherein the predictive post-processing includes temporal reasoning, SLAM label inference, cross-model reasoning, or label-specific reasoning using pre-trained convolutional neural network models.
10. The method of claim 1 , wherein the one or more labeled real-world driving logs include raw measurements captured by the one or more vehicle sensors and from ground truth labels that have been automatically labeled by the simulation-to-real automatic labeling.
11. The method of claim 1 , further including indexing the one or more labeled real-world driving logs.
12. A system for automatically labeling driving logs, the system comprising:
one or more processors, the one or more processors being programmed to initiate executable operations comprising:
receiving one or more unlabeled real-world driving logs, the one or more unlabeled real-world driving logs including data captured by one or more vehicle sensors;
automatically labeling the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs, the automatically labeling including:
analysis-by-synthesis on the one or more unlabeled real-world driving logs to generate one or more simulated driving logs, whereby the one or more simulated driving logs include reconstructed driving scenes or portions thereof; and
simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs; and
storing the one or more labeled real-world driving logs in one or more data stores of labeled driving logs.
13. The system of claim 12 , wherein the one or more simulated driving logs include photo-realistic renderings of an observed real-world scenario along with one or more ground truth labels, and wherein the ground truth labels include semantic segmentation, depth, two-dimensional object detection bounding boxes, three-dimensional object detection bounding boxes, or object class labels.
14. The system of claim 12 , wherein analysis-by-synthesis includes using a simulator together with computer vision techniques or computer graphing techniques.
15. The system of claim 12 , wherein the unlabeled real-world driving logs include camera data, radar data, LIDAR data, or sonar data.
16. The system of claim 12 , wherein the simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs includes:
using an unsupervised domain adaptation algorithm on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to learn a model specifically for generating labels for one or more desired tasks;
applying the model to the one or more unlabeled real-world driving logs to determine task-specific labels; and
applying the task-specific labels to the one or more unlabeled real-world driving logs to generate the one or more labeled real-world driving logs.
17. The system of claim 16 , wherein the simulation-to-real automatic labeling on the one or more simulated driving logs and the one or more unlabeled real-world driving logs to generate one or more labeled real-world driving logs includes:
performing predictive post-processing on the generated one or more labeled real-world driving logs, whereby potentially noisy labels introduced in the simulation-to-real automatic labeling are reduced or corrected.
18. The system of claim 17 , wherein the predictive post-processing includes spatio-temporal consistencies, geometric consistencies, cross-modal consistencies, and label-specific reasoning.
19. The system of claim 17 , wherein the predictive post-processing includes temporal reasoning, SLAM label inference, cross-model reasoning, or label-specific reasoning using pre-trained convolutional neural network models.
20. The system of claim 12 , wherein the one or more labeled real-world driving logs include raw measurements captured by the one or more vehicle sensors and from ground truth labels that have been automatically labeled by the simulation-to-real automatic labeling.