IP Library › Granted Patent US 12,333,389
Granted Patent B2
US 12,333,389 · App. 17/124,413 · Granted Jun 17, 2025

Autonomous vehicle system for intelligent on-board selection of data for training a remote machine learning model

Inventors: Shaojun Zhu (Pittsburgh, PA); Richard L. Kwant (San Bruno, CA); Nicolas Cebron (Sunnyvale, CA)
Assignee: Volkswagen Group of America Investments, LLC
G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,333,389
App. No.
17/124,413
Granted
Jun 17, 2025
Kind
B2
Abstract

Systems and methods for on-board selection of data logs for training a machine learning model. The methods include, by an autonomous vehicle, receiving sensor data logs corresponding to surroundings of the autonomous vehicle from a plurality of sensors, identifying one or more events within each sensor data log. The methods also include, for each sensor data log: analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training a machine learning model, and transmitting that sensor data log to a remote computing device for training the machine learning model if that sensor data log satisfies one or more usefulness criteria for training the machine learning model. The features can include spatial features, temporal features, bounding box inconsistencies, or map-based features.

Claims (112)

1. A method for on-board selection of data logs for training a machine learning model, comprising, by an on-board computing device of an autonomous vehicle:

receiving, from a plurality of sensors, a plurality of sensor data logs corresponding to surroundings of the autonomous vehicle;

identifying one or more events within each of the plurality of sensor data logs;

identifying a property of a plurality of different properties of the machine learning model that is to be trained;

identifying a first event of the one or more events which is missing or under-represented in training data for training the identified property of the machine learning model;

for each sensor data log of the plurality of sensor data logs associated with the identified first event:

analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training the identified property of the machine learning model by comparing a first event identified using a first sensor data log collected by a first sensor of the plurality of sensors and a corresponding event identified using a second sensor data log collected by a second sensor of the plurality of sensors, wherein the features comprise at least spatial features,

analyzing whether a difference between the first event and the corresponding event is more than a threshold,

determining that the first sensor data log and the second sensor data log are spatially inconsistent if the difference between the first event and the corresponding event is more than the threshold,

determining that the first sensor data log or the second sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model if an actual accuracy of the machine learning model will improve upon training using spatially inconsistent first sensor data log and second sensor data log, and

in response to determining that that sensor data log satisfies one or more usefulness criteria for training the machine learning model, transmitting that sensor data log to a remote computing device for training the machine learning model;

performing operations by a control system to control, using the machine learning model trained using the transmitted sensor data log, movement of the autonomous vehicle along a navigation route;

receiving information relating to an effectiveness of the sensor data log in training the machine learning model; and

modifying, based on the received information, a weight or frequency of use associated with at least one data log selection strategy of a plurality of different data log selection strategies that can be used during said analyzing features.

2. The method of claim 1 , further comprising, in response to determining that that sensor data log does not satisfy the one or more usefulness criteria for training the identified property of the machine learning model, discarding that sensor data log.

3. The method of claim 1 , wherein identifying the one or more events within each of the plurality of sensor data logs comprises detection of an object within a sensor data log.

4. The method of claim 1 , wherein the identified property comprises at least one of the following: a desired accuracy of the machine learning model, a convergence of the machine learning model, a statistical fit of the machine learning model, identification of a problem being solved using the machine learning model, or a training status of the machine learning model.

5. The method of claim 1 , wherein the analyzing comprises

checking a consistency of ones of the features related to the identified events within ones of the plurality of sensor data logs that were acquired using at least two sensors of a same sensor modality or different sensor modalities;

assigning a label of a plurality of different labels to the sensor data log based on the consistency; and

basing said determining at least on the label.

6. A method for on-board selection of data logs for training a machine learning model, comprising, by an on-board computing device of an autonomous vehicle:

receiving, from a plurality of sensors, a plurality of sensor data logs corresponding to surroundings of the autonomous vehicle;

identifying one or more events within each of the plurality of sensor data logs;

identifying a property of a plurality of different properties of the machine learning model that is to be trained;

identifying a first event of the one or more events which is missing or under-represented in training data for training the identified property of the machine learning model;

for each sensor data log of the plurality of sensor data logs associated with the identified first event:

analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training the identified property of the machine learning model by comparing a first event identified using a first sensor data log collected by a first sensor of the plurality of sensors and a corresponding event identified using a second sensor data log collected by a second sensor of the plurality of sensors, wherein the features comprise at least spatial features,

analyzing whether a difference between the first event and the corresponding event is more than a threshold,

determining that the first sensor data log and the second sensor data log are not spatially inconsistent if the difference between the first event and the corresponding event is less than the threshold, and

determining that the first second data log or the second sensor data log does not satisfy the one or more usefulness criteria for training the machine learning model if the statistical fit of the machine learning model indicates overfitting;

in response to determining that that sensor data log satisfies one or more usefulness criteria for training the machine learning model, transmitting that sensor data log to a remote computing device for training the machine learning model;

performing operations by a control system to control, using the machine learning model trained using the transmitted sensor data log, movement of the autonomous vehicle along a navigation route;

receiving information relating to an effectiveness of the sensor data log in training the machine learning model; and

modifying, based on the received information, a weight or frequency of use associated with at least one data log selection strategy of a plurality of different data log selection strategies that can be used during said analyzing features.

7. The method of claim 1 , wherein:

the features comprise temporal features;

analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model comprises identifying an event using a first sensor data log collected by a first sensor of the plurality of sensors; and

the method further comprises determining whether the first event can be identified using a second sensor data log collected by the first sensor immediately after collection of the first sensor data log.

8. The method of claim 7 , further comprising:

determining that the first sensor data log and the second sensor data log are temporally inconsistent if the event cannot be identified using the second sensor data log; and

determining that the first sensor data log or the second sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model if an actual accuracy of the machine learning model will improve upon training using temporally inconsistent first and second sensor data logs.

9. The method of claim 7 , further comprising:

determining that the first sensor data log and the second sensor data log are not temporally inconsistent if the event can be identified using the second sensor data log; and

determining that the first sensor data log or the second sensor data log does not satisfy the one or more usefulness criteria for training the identified property of the machine learning model if the statistical fit of the machine learning model indicates overfitting.

10. The method of claim 1 , wherein:

the features comprise bounding box inconsistencies;

analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model comprises identifying an object within a bounding box using a first sensor data log collected by a first sensor of the plurality of sensors; and

the method further comprises:

tracking the object for predicting a bounding box for the object in a second sensor data log collected by the first sensor immediately after collection of the first sensor data log, and

determining whether the object can be identified within the predicted bounding box using the second sensor data log.

11. The method of claim 10 , further comprising:

determining that the first sensor data log and the second sensor data log include bounding box inconsistencies if the object cannot be identified within the predicted bounding box using the second sensor data log; and

determining that the first sensor data log or the second sensor data log satisfies the one or more usefulness criteria for training the machine learning model if an actual accuracy of the machine learning model will improve upon training using the first and second data logs with bounding box inconsistencies.

12. The method of claim 10 , further comprising:

determining that the first sensor data log and the second sensor data log do not include bounding box inconsistencies if the object can be identified within the predicted bounding box using the second sensor data log; and

determining that the first sensor data log or the second sensor data log does not satisfy the one or more usefulness criteria for training the machine learning model if the statistical fit of the machine learning model indicates overfitting.

13. The method of claim 1 , wherein:

the features comprise map-based features;

analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training the identified property of the machine learning model comprises determining that an event identified using that sensor data log violates the map-based features; and

the method further comprises determining that the sensor log satisfies the one or more usefulness criteria when it violates the map-based features.

14. The method of claim 1 , further comprising:

receiving, from the remote server, information relating to an effectiveness of that sensor data log for training the machine learning model; and

updating, using the received information, the analyzing of features of the identified one or more events within a subsequently received sensor data log for determining whether the subsequently received sensor data log satisfies the one or more usefulness criteria for training the machine learning model.

15. A system for on-board selection of data logs for training a machine learning model comprising:

an autonomous vehicle comprising:

a plurality of sensors,

a processor, and

a non-transitory computer-readable medium comprising one or more programming instructions that when executed by the processor, cause the processor to:

receive, from the plurality of sensors, a plurality of sensor data logs corresponding to surroundings of the autonomous vehicle,

identify one or more events within each of the plurality of sensor data logs,

identify a property of a plurality of different properties of the machine learning model that is to be trained,

identify a first event of the one or more events which is missing or under-represented in training data for training the identified property of the machine learning model,

for each second data log of the plurality of sensor data logs associated with the identified first event:

analyze features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training the identified property of the machine learning model by comparing a first event identified using a first sensor data log collected by a first sensor of the plurality of sensors and a corresponding event identified using a second sensor data log collected by a second sensor of the plurality of sensors, wherein the features comprise at least spatial features,

analyze whether a difference between the first event and the corresponding event is more than a threshold,

determine that the first sensor data log and the second sensor data log are spatially inconsistent if the difference between the first event and the corresponding event is more than the threshold,

determine that the first sensor data log or the second sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model if an actual accuracy of the machine learning model will improve upon training using spatially inconsistent first sensor data log and second sensor data log, and

in response to determining that that sensor data log satisfies the one or more usefulness criteria for training the machine learning model, transmit that sensor data log to a remote computing device for training the machine learning model, and

control, using the machine learning model trained using the transmitted sensor data log, movement of the autonomous vehicle;

receive information relating to an effectiveness of the sensor data log in training the machine learning model; and

modify, based on the received information, a weight or frequency of use associated with at least one data log selection strategy of a plurality of different data log selection strategies that can be used during an analysis of the features of the identified one or more events.

16. The system of claim 15 , further comprising programming instructions that when executed by the processor, cause the processor to: in response to determining that that sensor data log does not satisfy one or more usefulness criteria for training the identified property of the machine learning model, discard that sensor data log.

17. The system of claim 15 , wherein the programming instructions, that when executed by the processor cause the processor to analyze the features, comprise:

check a consistency of ones of the features related to the identified events within ones of the plurality of sensor data logs that were acquired using at least two sensors of a same sensor modality or different sensor modalities;

assign a label of a plurality of different labels to the sensor data log based on the consistency; and

basing said determining whether that sensor data log satisfies one or more usefulness criteria at least on the label.

18. The system of claim 15 , wherein:

the features comprise temporal features;

the programming instructions that cause the processor to analyze features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model comprises identifying an event using first sensor data log collected by a first sensor of the plurality of sensors; and

the system further comprises programming instructions that when executed by the processor, cause the processor to:

determine whether the first event can be identified using a second sensor data log collected by the first sensor immediately after collection of the first sensor data log,

determine that the first sensor data log and the second sensor data log are temporally inconsistent if the first event cannot be identified using the second sensor data log, and

determine that the first data log or the second data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model if an actual accuracy of the machine learning model will improve upon training using temporally inconsistent first and second data logs.

19. The system of claim 15 , wherein:

the features comprise bounding box inconsistencies;

the programming instructions that cause the processor to analyze features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies the one or more usefulness criteria for training the machine learning model comprises identifying an object within a bounding box using first sensor data log collected by a first sensor of the plurality of sensors; and

the system further comprises programming instructions that when executed by the processor, cause the processor to:

track the object for predicting a bounding box for the object in a second sensor data log collected by the first sensor immediately after collection of the first sensor log,

determine whether the object can be identified within the predicted bounding box using the second sensor data log,

determine that the first sensor data log and the second sensor data log include bounding box inconsistencies if the object cannot be identified within the predicted bounding box using the second sensor data log, and

determine that the first sensor data log or the second sensor data log satisfies the one or more usefulness criteria for training the identified property of the machine learning model if an actual accuracy of the machine learning model will improve upon training using first and second data logs with bounding box inconsistencies.

20. The system of claim 15 , wherein:

the features comprise map-based features;

the programming instructions that cause the processor to analyze features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training the identified property of the machine learning model comprises determining that an event identified using that sensor data log violates the map-based features; and

the system further comprises programming instructions that when executed by the processor, cause the processor to determine that the sensor log satisfies the one or more usefulness criteria when it violates the map-based features.

21. The system of claim 15 , further comprising programming instructions that when executed by the processor, cause the processor to:

receive, from a remote server, information relating to an effectiveness of that sensor data log for training the machine learning model; and

update, using the received information, the analyzing of features of the identified one or more events within a subsequently received sensor data log for determining whether the subsequently received sensor data log satisfies the one or more usefulness criteria for training the machine learning model.

22. The method of claim 1 , further comprising, at the remote computing device:

receiving the transmitted sensor data log; and

using the transmitted sensor data log for training the machine learning model.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 9, 2024
From: ARGO AI, LLC
To: VOLKSWAGEN GROUP OF AMERICA INVESTMENTS, LLC
Reel/Frame 069177/0099 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 16, 2020
From: ZHU, SHAOJUN; KWANT, RICHARD L.; CEBRON, NICOLAS
To: ARGO AI, LLC
Reel/Frame 054673/0887 →
Continuity (1)
Related Publication 20220188695A1 · Jun 16, 2022
References Cited (48)
US 7793188B2 · Mukhopadhyay et al. · 2010 [cited by applicant]
US 9031317B2 · Yakubovich et al. · 2015 [cited by applicant]
US 10379842B2 · Malladi et al. · 2019 [cited by applicant]
US 10489222B2 · Sathyanarayana et al. · 2019 [cited by applicant]
US 10499283B2 · Elias et al. · 2019 [cited by applicant]
US 11657591B2 · Muehlenstaedt · 2023 [cited by examiner]
US 20060046711A1 · Jung · 2006 [cited by examiner]
US 20160321561A1 · Röder · 2016 [cited by examiner]
US 20170011281A1 · Dijkman et al. · 2017 [cited by applicant]
US 20170083792A1 · Rodriguez-Serrano et al. · 2017 [cited by applicant]
US 20190258904A1 · Ma et al. · 2019 [cited by applicant]
US 20200005083A1 · Collins · 2020 [cited by applicant]
US 20200081426A1 · Kane et al. · 2020 [cited by applicant]
US 20200111005A1 · Ghosh et al. · 2020 [cited by applicant]
US 20200202171A1 · Hughes et al. · 2020 [cited by applicant]
US 20200272854A1 · Caesar · 2020 [cited by applicant]
US 20200286615A1 · Hartung et al. · 2020 [cited by applicant]
US 20200380285A1 · Price et al. · 2020 [cited by applicant]
US 20200382361A1 · Chandrasekhar et al. · 2020 [cited by applicant]
US 20210125423A1 · Isaac · 2021 [cited by examiner]
US 20210129845A1 · Bonk · 2021 [cited by applicant]
US 20210134379A1 · Zhao · 2021 [cited by applicant]
US 20210156963A1 · Popov · 2021 [cited by examiner]
US 20210295201A1 · Kim et al. · 2021 [cited by applicant]
US 20220164602A1 · Frtunikj · 2022 [cited by examiner]
US 20230040068A1 · Gandhi · 2023 [cited by applicant]
CN 111914944A · 2020 [cited by applicant]
WO 2020159568A1 · 2020 [cited by applicant]
D. Eldowa, K. Elgazzar, H. S. Hassanein, T. Sharaf and S. Shah, “Assessing the Integrity of Traffic Data through Short Term State Prediction,” 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA, 20… [cited by examiner]
A. A. Sodemann, M. P. Ross and B. J. Borghetti, “A Review of Anomaly Detection in Automated Surveillance,” in IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, No. 6, pp. 12… [cited by examiner]
Castaño, F.; Beruvides, G.; Villalonga, A.; Haber, R.E. Self-Tuning Method for Increased Obstacle Detection Reliability Based on Internet of Things LiDAR Sensor Models. Sensors 2018, 18, 1508. https://doi.org/10.3390/s1… [cited by examiner]
S. Thornton and S. Dey, “Machine Learning Techniques for Vehicle Matching with Non-Overlapping Visual Features,” 2020 IEEE 3rd Connected and Automated Vehicles Symposium (CAVS), Victoria, BC, Canada, 2020, pp. 1-6, doi:… [cited by examiner]
Zhang et al., “Edge-to-edge cooperative artificial intelligence in smart cities with on-demand learning offloading” (Year: 2019). [cited by examiner]
Claviere et al., “Trajectory Tracking Control for Robotic Vehicles Using Counterexample Guided Training of Neural Networks” (Year: 2019). [cited by examiner]
Bobbili, N. P., & Cretu, A. M. (Jun. 2018). Adaptive weighting with smote for learning from imbalanced datasets: a case study for traffic offence prediction. In 2018 IEEE International Conference on CIVEMSA (pp. 1-6) (Y… [cited by examiner]
Dave, H., “Active Learning Sampling Strategies”, available at https://towardsdatascience.com/active-learning-sampling-strategies-f8d8ac7037c8 accessed on Nov. 17, 2020. [cited by applicant]
Nvidia Al, “Scalable Active Learning for Autonomous Driving”, Dec. 18, 2019, available at https://medium.com/nvidia-ai/scalable-active-learning-for-autonomous-driving-a-practical-implementation-and-a-b-test-4d315ed04b5f… [cited by applicant]
Wei, X., “Deep Active Learing for 3D Object Detection for Autonomous Driving”, Degree Project in Computer Science and Engineering, Second Cycle, Stockholm, Sweden, Nov. 22, 2019, available at http://www.diva-portal.org/… [cited by applicant]
Liu, S. et al., “Edge Computing for Autonomous Driving: Opportunities and Challenges”, Proceedings of the IEEE, vol. 107, No. 8, Aug. 2019, available at http://weisong.eng.wayne.edu/_resources/pdfs/liu19-EdgeAV.pdf. [cited by applicant]
Cai, S. et al., “Data Collection in Underwater Sensor Networks Based on Mobile Edge Computing”, IEEE Access, (2019) vol. 7, pp. 65357-65367, available at https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8719985. [cited by applicant]
Information about Related Patents and Patent Applications, see section 6 of the accompanying Information Disclosure Statement Letter, which concerns Related Patents and Patent Applications. [cited by applicant]
International Search Report and Written Opinion mailed Apr. 7, 2022, issued in International Application No. PCT/US2021/072897 (10 pages). [cited by applicant]
Huang, H et al., “On the improvement of reinforcement active learning with the involvement of cross entropy to address one-shot learning problem” PLoS One (2019) 14(6): e0217408. available at https://journals.plos.org/p… [cited by applicant]
Yoo, D. et al., “Learning Loss for Active Learning”, arXiv:1905.03677 [cs.CV], May 9, 2019. available at: https://openaccess.thecvf.com/content_CVPR_2019/papers/Yoo_Learning_Loss_for_Active_Learning_CVPR_2019_paper.pdf. [cited by applicant]
Aussel, N. et al., “Combining federated and active learning for communication-efficient distributed failure prediction in aeronautics” 2019, hal-02446200f, available at https://hal.archives-ouvertes.fr/hal-02446200/docu… [cited by applicant]
Zhang et al, Edge-to-edge cooperative artificial intelligence in smart cities with on-demand learning offloading, 2019. [cited by applicant]
Claviere et al., Trajectory tracking control for robotic vehicles using counterexample guided training of neural networks, 2019. [cited by applicant]
Bobbili, N.P. et al., Jun. 2018, Adaptive weighting with smote for learning from imbalanced datasets: a case study for traffic offence prediction, 2018 IEEE International Conference on CIVEMSA, p. 1-6. [cited by applicant]