Granted Patent B2

US 12,277,752 · App. 18/364,569 · Granted Apr 15, 2025

Inventors: Jelena Frtunikj (Bavaria, DE); Daniel Alfonsetti (Auburn, ME)

Assignee: Volkswagen Group of America Investments, LLC

G06V10/774G06F18/2115G06F18/2148G06N20/00G06V10/255G06V10/772G06V10/82G06V20/56G06V20/58G06V20/588G06V20/64

Abstract

Claims (48)

1. A method for selecting data for building a machine learning model, the method comprising:

receiving, from a plurality of sensors, a plurality of unlabeled sensor data logs corresponding to surroundings of an autonomous vehicle;

identifying one or more trends associated with a training dataset comprising a plurality of labeled data logs, the training dataset being used for training a machine learning model;

selecting a subset of the plurality of unlabeled sensor data logs that have an importance score greater than a threshold, the importance score being assigned to each of the plurality of unlabeled sensor data logs in the subset using a function determined using the one or more trends; and

using the subset of the plurality of unlabeled sensor data logs for updating the machine learning model to generate an updated model.

2. The method of claim 1 , further comprising:

identifying, using the machine learning model, within each of the plurality of unlabeled sensor data logs, a classification label;

determining a confidence score of the machine learning model trained for identifying the classification label; and

using the confidence score and the one or more trends for identifying the function, the function being configured for generating the importance score for each of the plurality of unlabeled sensor data logs.

3. The method of claim 2 , wherein the machine learning model is an object detection model.

4. The method of claim 2 , wherein the confidence score is determined based on at least one of the following associated with the machine learning model: an actual accuracy; a desired accuracy; a false positive rate; a false negative rate; a convergence; an output; a statistical fit; identification of a problem being solved; or a training status.

5. The method of claim 2 , further comprising:

receiving metadata associated with each of the plurality of labeled data logs in the training data sets; and

using the metadata for identifying the one or more trends.

6. The method of claim 5 , wherein the metadata comprises at least one of the following: a time of capture of a data log; a location of capture of a data log; information relating to environmental conditions at a location or time of capture of a data log; or information relating to one or more events a location or time of capture of a data log.

7. The method of claim 2 , wherein the one or more trends are used to identify one or more characteristics in the plurality labeled data logs of the training dataset that are underrepresented.

8. The method of claim 7 , wherein the one or more characteristics are selected from at least one of the following: a label class count; a label count by time; a label count by location; an event class count; an event class count by time; an event class count by location; an instance count by environmental conditions; or a machine leaning model uncertainty.

9. The method of claim 2 , further comprising:

receiving metadata associated with the unlabeled sensor data logs, the metadata comprising at least one of the following: a time of capture of a data log, a location of capture of a data log, information relating to environmental conditions at a location or time of capture of a data log, or information relating to one or more events a location or time of capture of a data log;

wherein the importance score is assigned to each of the plurality of unlabeled sensor data logs by assigning a score to one or more of a plurality of characteristics of each of the plurality of unlabeled sensor data logs, the plurality of characteristics comprising at least one of the following: a label class count; a label count by time; a label count by location; an event class count; an event class count by time; an event class count by location; an instance count by environmental conditions; or a machine leaning model uncertainty.

10. The method of claim 1 , further comprising discarding one or more of the plurality of unlabeled sensor data logs that have an importance score less than the threshold.

11. A system for selecting data for training a machine learning model using active learning, the system comprising:

a processor; and

a non-transitory computer-readable medium comprising one or more programming instructions that when executed by the processor, cause the processor to:

receive, from a plurality of sensors, a plurality of unlabeled sensor data logs corresponding to surroundings of an autonomous vehicle,

identify one or more trends associated with a training dataset comprising a plurality of labeled data logs, the training dataset being used for training a machine learning model,

select a subset of the plurality of unlabeled sensor data logs that have an importance score greater than a threshold, the importance score being assigned to each of the plurality of unlabeled sensor data logs in the subset using a function determined using the one or more trends, and

use the subset of the plurality of unlabeled sensor data logs for updating the machine learning model to generate an updated model.

12. The system of claim 11 , further comprising programming instructions that when executed by the processor, cause the processor to:

identify, using the machine learning model, within each of the plurality of unlabeled sensor data logs, a classification label;

determine a confidence score of the machine learning model trained for identifying the classification label; and

use the confidence score and the one or more trends for identifying the function, the function being configured for generating the importance score for each of the plurality of unlabeled sensor data logs.

13. The system of claim 12 , wherein the machine learning model is an object detection model.

14. The system of claim 12 , wherein the confidence score is determined based on at least one of the following associated with the machine learning model trained using the training dataset: an actual accuracy; a desired accuracy; a false positive rate; a false negative rate; a convergence; an output; a statistical fit; identification of a problem being solved; or a training status.

15. The system of claim 12 , further comprising programming instructions that when executed by the processor, cause the processor to:

receive metadata associated with each of the plurality of labeled data logs in the training data sets; and

use the metadata for identifying the one or more trends.

16. The system of claim 15 , wherein the metadata comprises at least one of the following: a time of capture of a data log; a location of capture of a data log; information relating to environmental conditions at a location or time of capture of a data log; or information relating to one or more events a location or time of capture of a data log.

17. The system of claim 12 , wherein the one or more trends are used to identify one or more characteristics in the plurality labeled data logs of the training dataset that are underrepresented.

18. The system of claim 17 , wherein the one or more characteristics are selected from at least one of the following: a label class count; a label count by time; a label count by location; an event class count; an event class count by time; an event class count by location; an instance count by environmental conditions; or a machine leaning model uncertainty.

19. The system of claim 12 , further comprising programming instructions that when executed by the processor, cause the processor to:

receive metadata associated with the unlabeled sensor data logs, the metadata comprising at least one of the following: a time of capture of a data log, a location of capture of a data log, information relating to environmental conditions at a location or time of capture of a data log, or information relating to one or more events a location or time of capture of a data log;

wherein the importance score is assigned to each of the plurality of unlabeled sensor data logs by assigning a score to one or more of a plurality of characteristics of each of the plurality of unlabeled sensor data logs, the plurality of characteristics comprising at least one of the following: a label class count; a label count by time; a label count by location; an event class count; an event class count by time; an event class count by location; an instance count by environmental conditions; or a machine leaning model uncertainty.

20. A computer program product comprising a non-transitory computer-readable medium that stores instructions that, when executed by a computing device, will cause the computing device to perform operations comprising:

receiving, from a plurality of sensors, a plurality of unlabeled sensor data logs corresponding to surroundings of an autonomous vehicle;

identifying one or more trends associated with a training dataset comprising a plurality of labeled data logs, the training dataset being used for training a machine learning model;

selecting a subset of the plurality of unlabeled sensor data logs that have an importance score greater than a threshold, the importance score being assigned to each of the plurality of unlabeled sensor data logs in the subset using a function determined using the one or more trends; and

using the subset of the plurality of unlabeled sensor data logs for updating the machine learning model to generate an updated model.

Assignments (2)

Continuity (2)

References Cited (45)