IP Library › Granted Patent US 12705495
Granted Patent B2
US 12705495 · App. 17/230,760 · Granted Aug 11, 2026

Data labeling for training artificial intelligence systems

Inventors: Tarek Aziz Lahlou (McLean, VA); Megan Lynn DeLaunay (Silver Spring, MD); Corey Jonathan Fyock (Manassas, VA); Erin Babinsky (Arlington, VA)
Assignee: Capital One Services, LLC
G06N3/09G06N20/00G06N3/084G06N5/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12705495
App. No.
17/230,760
Granted
Aug 11, 2026
Kind
B2
Abstract

Systems, apparatuses, and methods are described for data labeling for training artificial intelligence systems. A candidate dataset comprising data samples and corresponding labels may be used to update an incumbent dataset comprise data samples and corresponding labels. The integrity of a data sample-label pair in the candidate dataset may be determined before the data sample-label pair is added to the incumbent dataset. For determining labeling integrity, a plurality of machine classifiers may be trained based on the incumbent dataset and portions of the candidate dataset. The plurality of machine classifiers as trained may be used to generate predicted labels for data samples in the candidate dataset. The integrity of the data sample-label pair in the candidate dataset may be measured based on the predicted labels for the data sample.

Claims (116)

1 . A method comprising:

determining, by a computing device, an incumbent dataset comprising a first plurality of data samples and a first plurality of labels corresponding to the first plurality of data samples;

determining a candidate dataset for updating the incumbent dataset, wherein the candidate dataset comprises a second plurality of data samples and a second plurality of labels corresponding to the second plurality of data samples;

testing the candidate dataset by a plurality of machine classifiers, wherein each machine classifier of the plurality of machine classifiers comprises a plurality of model parameters, wherein the plurality of model parameters are assigned at random for each machine classifier, and wherein testing the candidate dataset by a given machine classifier, of the plurality of machine classifiers, comprises:

determining, for the given machine classifier, a training subset of the candidate dataset and a remaining subset of the candidate dataset;

training the given machine classifier, based on the incumbent dataset and the training subset, to refine the plurality of model parameters of the given machine classifier; and

generating, based on the trained given machine classifier, a first plurality of predicted labels corresponding to a plurality of data samples of the remaining subset;

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregating a second plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the aggregated second plurality of predicted labels corresponding to a candidate data sample of the second plurality of data samples;

determining a degree of consistency of the aggregated second plurality of predicted labels;

based on the degree of consistency of the aggregated second plurality of predicted labels not satisfying a threshold, marking the candidate data sample for additional review;

grouping, based on the marking of the candidate data sample, a plurality of corresponding vector representations of the second plurality of data samples, and degrees of similarity among the second plurality of data samples, the second plurality of data samples into clusters; and

updating, based on receiving a single assigned label assigned to a cluster of the clusters of the second plurality of data samples, the candidate dataset to include the single assigned label for all second plurality of data samples in the cluster.

2 . The method of claim 1 , further comprising:

distributing the second plurality of data samples to a set of annotator devices for manual labeling; and

for each data sample of the second plurality of data samples:

receiving a plurality of labels determined via the set of annotator devices; and

determining, based on the plurality of labels determined via the set of annotator devices, a consensus label,

wherein the second plurality of labels comprises the consensus labels determined for the second plurality of data samples.

3 . The method of claim 1 , further comprising:

determining, based on the second plurality of data samples, the plurality of corresponding vector representations;

determining, based on the plurality of corresponding vector representations, the degrees of similarity among the second plurality of data samples;

wherein the grouping comprises:

causing display, via a set of annotator devices for manual labeling, of the clusters of data samples with indications of spatial relationships, among the second plurality of data samples, corresponding to the degrees of similarity among the second plurality of data samples; and

receiving an indication of the single assigned label assigned to the cluster of the clusters of data samples.

4 . The method of claim 1 , wherein:

the training subset comprises a first selection of data samples from the candidate dataset; and

the remaining subset comprises a second selection of data samples from the candidate dataset, the second selection of data samples being distinct from the first selection of data samples.

5 . The method of claim 1 , wherein the training subset determined for the given machine classifier of the plurality of machine classifiers is different from a training subset determined for another machine classifier of the plurality of machine classifiers.

6 . The method of claim 1 , further comprising:

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregating a third plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the third plurality of predicted labels corresponding to a second candidate data sample of the second plurality of data samples;

determining a degree of consistency of the third plurality of predicted labels; and

based on the degree of consistency of the third plurality of predicted labels satisfying the threshold, determining a machine classifier consensus label, corresponding to the second candidate data sample, based on the third plurality of predicted labels.

7 . The method of claim 6 , further comprising:

determining an annotator device consensus label, corresponding to the second candidate data sample, of the second plurality of labels; and

based on the machine classifier consensus label corresponding to the annotator device consensus label, adding the second candidate data sample to the incumbent dataset.

8 . The method of claim 6 , further comprising:

determining an annotator device consensus label, corresponding to the second candidate data sample, of the second plurality of labels; and

based on the machine classifier consensus label not corresponding to the annotator device consensus label:

associating the second candidate data sample with the machine classifier consensus label; and

marking the second candidate data sample for additional review for removing an association of the second candidate data sample with the annotator device consensus label.

9 . The method of claim 1 , further comprising:

updating the incumbent dataset based on at least a portion of the candidate dataset; and

generating, based on the updated incumbent dataset, a predicted label corresponding to a received data sample.

10 . A method comprising:

determining, by a computing device, an incumbent dataset comprising a first plurality of data samples and a first plurality of labels corresponding to the first plurality of data samples;

determining a candidate dataset for updating the incumbent dataset, wherein the candidate dataset comprises a second plurality of data samples and a second plurality of labels corresponding to the second plurality of data samples;

testing the candidate dataset by a plurality of machine classifiers, wherein each machine classifier of the plurality of machine classifiers comprises a plurality of model parameters, and wherein testing the candidate dataset by a given machine classifier, of the plurality of machine classifiers, comprises:

determining, for the given machine classifier, a training subset of the candidate dataset and a remaining subset of the candidate dataset;

training the given machine classifier, based on the incumbent dataset and the training subset, to refine the plurality of model parameters of the given machine classifier; and

generating, based on the trained given machine classifier, a first plurality of predicted labels corresponding to a plurality of data samples of the remaining subset;

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregating a second plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the aggregated second plurality of predicted labels corresponding to a candidate data sample of the second plurality of data samples;

determining a degree of consistency of the aggregated second plurality of predicted labels;

based on the degree of consistency of the aggregated second plurality of predicted labels satisfying a threshold, determining a machine classifier consensus label, corresponding to the candidate data sample, based on the aggregated second plurality of predicted labels;

determining an annotator device consensus label, corresponding to the candidate data sample, of the second plurality of labels;

based on the machine classifier consensus label corresponding to the annotator device consensus label, adding the candidate data sample to the incumbent dataset;

grouping, based on the marking of the candidate data sample, a plurality of corresponding vector representations of the second plurality of data samples, and degrees of similarity among the second plurality of data samples, the second plurality of data samples into clusters; and

updating, based on receiving a single assigned label assigned to a cluster of the clusters of the second plurality of data samples, the candidate dataset to include the single assigned label for all second plurality of data samples in the cluster.

11 . The method of claim 10 , further comprising:

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregating a third plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the third plurality of predicted labels corresponding to a second candidate data sample of the second plurality of data samples;

determining a degree of consistency of the third plurality of predicted labels;

based on the degree of consistency of the third plurality of predicted labels satisfying the threshold, determining a second machine classifier consensus label, corresponding to the second candidate data sample, based on the third plurality of predicted labels;

determining a second annotator device consensus label, corresponding to the second candidate data sample, of the second plurality of labels; and

based on the second machine classifier consensus label not corresponding to the second annotator device consensus label:

associating the second candidate data sample with the second machine classifier consensus label; and

marking the second candidate data sample for additional review for removing an association of the second candidate data sample with the second annotator device consensus label.

12 . The method of claim 10 , further comprising:

distributing the second plurality of data samples to a set of annotator devices for manual labeling; and

for each data sample of the second plurality of data samples:

receiving a plurality of labels determined via the set of annotator devices; and

determining, based on the plurality of labels determined via the set of annotator devices, a consensus label,

wherein the second plurality of labels comprises the consensus labels determined for the second plurality of data samples.

13 . The method of claim 10 , further comprising:

determining, based on the second plurality of data samples, the plurality of corresponding vector representations;

determining, based on the plurality of corresponding vector representations, the degrees of similarity among the second plurality of data samples;

wherein the grouping comprises:

causing display, via a set of annotator devices for manual labeling, of the clusters of data samples with indications of spatial relationships, among the second plurality of data samples, corresponding to the degrees of similarity among the second plurality of data samples; and

receiving an indication of the assigned label assigned to the cluster of the clusters of data samples.

14 . The method of claim 10 , wherein:

the training subset comprises a first selection of data samples from the candidate dataset; and

the remaining subset comprises a second selection of data samples from the candidate dataset, the second selection of data samples being distinct from the first selection of data samples.

15 . The method of claim 10 , wherein the training subset determined for the given machine classifier of the plurality of machine classifiers is different from a training subset determined for another machine classifier of the plurality of machine classifiers.

16 . The method of claim 10 , further comprising:

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregating a third plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the third plurality of predicted labels corresponding to a second candidate data sample of the second plurality of data samples;

determining a degree of consistency of the third plurality of predicted labels; and

based on the degree of consistency of the third plurality of predicted labels not satisfying the threshold, marking the second candidate data sample for additional review.

17 . The method of claim 10 , further comprising:

updating the incumbent dataset based on at least a portion of the candidate dataset; and

generating, based on the updated incumbent dataset, a predicted label corresponding to a received data sample.

18 . An apparatus comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

determine an incumbent dataset comprising a first plurality of data samples and a first plurality of labels corresponding to the first plurality of data samples;

determine a candidate dataset for updating the incumbent dataset, wherein the candidate dataset comprises a second plurality of data samples and a second plurality of labels corresponding to the second plurality of data samples;

test the candidate dataset by a plurality of machine classifiers, wherein each machine classifier of the plurality of machine classifiers comprises a plurality of model parameters, and wherein testing the candidate dataset by a given machine classifier, of the plurality of machine classifiers, comprises:

determining, for the given machine classifier, a training subset of the candidate dataset and a remaining subset of the candidate dataset;

training the given machine classifier, based on the incumbent dataset and the training subset, to refine the plurality of model parameters of the given machine classifier; and

generating, based on the trained given machine classifier, a first plurality of predicted labels corresponding to a plurality of data samples of the remaining subset;

based on the testing of the candidate dataset by the plurality of machine classifiers, aggregate a second plurality of predicted labels generated based on multiple machine classifiers of the plurality of machine classifiers, the aggregated second plurality of predicted labels corresponding to a candidate data sample of the second plurality of data samples;

determining a degree of consistency of the aggregated second plurality of predicted labels;

mark the candidate data sample for additional review based on the degree of consistency of the aggregated second plurality of predicted labels being below a threshold;

group, based on the marking of the candidate data sample, a plurality of corresponding vector representations of the second plurality of data samples, and degrees of similarity among the second plurality of data samples, the second plurality of data samples into clusters;

update, based on receiving a single assigned label assigned to a cluster of the clusters of the second plurality of data samples, the candidate dataset to include the single assigned label for all second plurality of data samples in the cluster; and

when the degree of consistency of the aggregated second plurality of predicted labels satisfies the threshold:

determine a machine classifier consensus label, corresponding to the candidate data sample, based on the aggregated second plurality of predicted labels;

determine an annotator device consensus label, corresponding to the candidate data sample, of the second plurality of labels; and

add the candidate data sample to the incumbent dataset based on the machine classifier consensus label corresponding to the annotator device consensus label.

19 . The apparatus of claim 18 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to, when the degree of consistency of the second plurality of predicted labels satisfies the threshold:

when the machine classifier consensus label does not correspond to the annotator device consensus label:

associate the candidate data sample with the machine classifier consensus label; and

mark the candidate data sample for additional review for removing an association of the candidate data sample with the annotator device consensus label.

20 . The apparatus of claim 18 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to:

determine, based on the second plurality of data samples, the plurality of corresponding vector representations;

determine, based on the plurality of corresponding vector representations, the degrees of similarity among the second plurality of data samples;

wherein the grouping comprises:

cause display, via a set of annotator devices for manual labeling, of the clusters of data samples with indications of spatial relationships, among the second plurality of data samples, corresponding to the degrees of similarity among the second plurality of data samples; and

receive an indication of the single assigned label assigned to the cluster of the clusters of data samples.