IP Library › Granted Patent US 11,645,386
Granted Patent B2
US 11,645,386 · App. 17/945,990 · Granted May 9, 2023

Systems and methods for automated labeling of subscriber digital event data in a machine learning-based digital threat mitigation platform

Inventors: Wei Liu (Seattle, WA); Ralf Gunter Correa Carvalho (Seattle, WA)
Assignee: Sift Science, Inc.
G06F21/554G06N5/022G06F2221/034
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,645,386
App. No.
17/945,990
Granted
May 9, 2023
Kind
B2
Abstract

A system and method for accelerating an automated labeling of a volume of unlabeled digital event data samples includes identifying a corpus characteristic of a digital event data corpus that includes a plurality of distinct unlabeled digital event data samples; selecting an automated bulk labeling algorithm based on the corpus characteristic associated with the digital event data corpus satisfying a bulk labeling criterion of the automated bulk labeling algorithm; evaluating a subset of the plurality of unlabeled digital event data samples, wherein evaluating the subset includes attributing a distinct classification label to each digital event data sample within the subset; and in response to the selection, executing the selected automated bulk labeling algorithm against the digital event data corpus, wherein the executing includes simultaneously assigning a classification label equivalent to the distinct classification label to a superset of the digital event data corpus that relates to the subset.

Claims (58)

1. A method for accelerating an automated labeling of unlabeled data samples, the method comprising:

selecting an automated bulk labeling algorithm of a plurality of distinct automated bulk labeling algorithms based on one or more characteristics or one or more metrics of a digital event data corpus satisfying a bulk labeling criterion of one of the plurality of distinct automated bulk labeling algorithms, wherein the digital event data corpus includes a plurality of unlabeled digital event data samples;

evaluating a first subset of the plurality of unlabeled digital event data samples, wherein evaluating the first subset of the plurality of unlabeled digital event data samples includes associating a distinct classification label to each unlabeled digital event data sample of the first subset;

executing, by one or more computers, the selected automated bulk labeling algorithm based on the evaluation of the first subset, wherein the executing includes automatically assigning a classification label equivalent to the distinct classification label to a second subset of the plurality of unlabeled digital event data samples;

training, by the one or more computers, a machine learning-based threat model based on the execution of the selected automated bulk labeling algorithm;

computing, using the machine learning-based threat model, a digital threat inference based on one or more features extracted from a target digital transaction event associated with a subscriber; and

computing, via the one or more computers, an automated disposal decision that enables a blocking or an allowance of the target digital transaction event based on the digital threat inference.

2. The method according to claim 1 , wherein

each of the plurality of distinct automated bulk labeling algorithms includes a set of computer-executable instructions that, when executed, executes an automated sequence of tasks that automatically assigns at least one classification label to a target corpus of digital event data samples.

3. The method according to claim 1 , wherein:

the one or more characteristics or the one or more metrics associated with the digital event data corpus satisfies the bulk labeling criterion of an automated connected component graph bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on identifying that a collection of the plurality of unlabeled digital event data samples comprise a plurality of distinct digital event attribute values of a first type that are digitally linked to a distinct digital event attribute value of a second type.

4. The method according to claim 1 , wherein:

the one or more characteristics or the one or more metrics associated with the digital event data corpus satisfies the bulk labeling criterion of an automated connected component graph bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on the plurality of unlabeled digital event data samples of the digital event data corpus corresponding to a suspected automated fraud attack.

5. The method according to claim 1 , wherein

the one or more characteristics or the one or more metrics satisfies the bulk labeling criterion of an automated digital event embeddings-based bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on the plurality of unlabeled digital event data samples corresponding to a probable digital abuse type.

6. The method according to claim 1 , wherein

the one or more characteristics or the one or more metrics associated with the digital event data corpus satisfies the bulk labeling criterion of an automated digital event embeddings-based bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on the digital event data corpus exceeding a predetermined corpus size threshold.

7. The method according to claim 1 , wherein

the digital event data corpus includes unlabeled digital event data samples corresponding to a first distinct subscriber of a machine learning-based digital threat mitigation service and labeled digital event data samples corresponding to a second distinct subscriber of the machine learning-based digital threat mitigation service.

8. The method according to claim 7 , wherein

the one or more characteristics or the one or more metrics satisfies the bulk labeling criterion of an automated context-based bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on the digital event data corpus comprising a plurality of labeled digital event data samples that exceeds a service-defined threshold value.

9. The method according to claim 1 , wherein

one of the one or more characteristics satisfying the bulk labeling criterion corresponds to a suspected digital fraud type characteristic.

10. A method for accelerating an automated labeling of unlabeled data samples:

computing one or more metrics associated with a volume of unlabeled digital event data samples, wherein at least one of the one or more metrics informs a probable degree of digital fraud risk associated with the volume;

identifying an automated bulk labeling algorithm of a plurality of distinct automated bulk labeling algorithms based on the one or more metrics satisfying a bulk labeling criterion associated with one of the plurality of distinct automated bulk labeling algorithms;

based on identifying the automated bulk labeling algorithm, executing, by one or more computers, the identified automated bulk labeling algorithm against the volume, wherein the executing includes assigning to each unlabeled digital event data sample of the volume of unlabeled digital event data samples, a classification label of one of (i) a first label based on the probable degree of digital fraud risk satisfying a predetermined threat threshold value and (ii) a second classification label based on the probable degree of digital fraud risk not satisfying the predetermined threat threshold value;

training, by the one or more computers, a machine learning-based model based on the execution of the identified automated bulk labeling algorithm;

computing, using the machine learning-based model, a digital threat inference based on one or more features extracted from a target digital transaction event associated with a subscriber; and

computing, via the one or more computers, an automated disposal decision that enables a blocking or an allowance of the target digital transaction event based on the digital threat inference.

11. The method according to claim 10 , further comprising:

constructing a labeled digital event data corpus that includes the volume of unlabeled digital event data samples and the classification label.

12. The method according to claim 10 , further comprising:

receiving a verification data signal from the subscriber confirming or disconfirming a bulk labeling action associated with the execution of the automated bulk labeling algorithm.

13. A method for accelerating an automated labeling of unlabeled data samples, the method comprising:

identifying one or more characteristics or one or more metrics associated with a volume of unlabeled data samples;

selecting an automated bulk labeling algorithm of a plurality of distinct automated bulk labeling algorithms based on the one or more characteristics or the one or more metrics satisfying a bulk labeling criterion associated with one of the plurality of distinct automated bulk labeling algorithms;

evaluating a subset of the volume of unlabeled data samples that includes associating a distinct classification label of a plurality of distinct classification labels to each unlabeled data sample of the subset; and

based on the selection, executing, by one or more computers, the selected automated bulk labeling algorithm against the volume of unlabeled data samples, wherein the executing includes assigning, in bulk, a classification label substantially equivalent to the distinct classification label to a set of unlabeled data samples of the volume of unlabeled data samples that is distinct from the subset

training, by the one or more computers, a machine learning-based threat model based on a training data corpus associated with the execution of the selected automated bulk labeling algorithm;

computing, using the machine learning-based threat model, a digital threat inference based on one or more features extracted from a target digital transaction event associated with a subscriber; and

computing, via the one or more computers, an automated disposal decision that enables a blocking or an allowance of the target digital transaction event based on the digital threat inference.

14. The method according to claim 13 , wherein each of the plurality of distinct automated bulk labeling algorithms includes a distinct automated sequence of actions that, when executed, executes the distinct automated sequence of actions that automatically assigns at least one classification label to each data sample of a target volume of data samples.

15. The method according to claim 13 , wherein:

the one or more characteristics or the one or more metrics satisfies the bulk labeling criterion of an automated connected component graph bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on a determination that a collection of the plurality of distinct unlabeled digital event data samples comprise a plurality of distinct attribute values of a first type that are digitally linked to a distinct attribute value of a second type.

16. The method according to claim 13 , wherein

the one or more characteristics or the one or more metrics satisfies the bulk labeling criterion of an automated digital event embeddings-based bulk labeling algorithm, and

the bulk labeling criterion is satisfied based on the volume of unlabeled data samples exceeding a predetermined corpus size threshold.

17. The method according to claim 13 , wherein:

the one or more characteristics or the one or more metrics satisfies the bulk labeling criterion of an automated connected component graph bulk labeling algorithm.

18. The method according to claim 13 , wherein

the one or more characteristics or the one or more metrics associated satisfies the bulk labeling criterion of an automated digital event embeddings-based bulk labeling algorithm.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 16, 2022
From: LIU, WEI; CARVALHO, RALF GUNTER CORREA
To: SIFT SCIENCE, INC.
Reel/Frame 061120/0377 →
Continuity (3)
Continuation 17693851 · Mar 14, 2022
Provisional Application 63170427 · Apr 2, 2021
Related Publication 20230012656A1 · Jan 19, 2023
Cited By (1)
US 12,737,767