Systems and methods for block rate matching when updating a machine learning model based fraud detection system

1 . A system, comprising:

a transaction system comprising a transaction data store, a transaction manager, a fraud detection computer system comprising a first machine learning (ML) model, and a first processor and a first memory storing executable instructions that when executed by the first processor causes the first processor to perform operations, comprising:

continuously receiving a first set of transactions from a network,

receiving from a computer system of a merchant a threshold,

storing the threshold and a first block rate,

storing the first set of transactions for the merchant in the transaction data store,

receiving, via the transaction manager, a first transaction associated with the merchant from the network, and

sending first parameters associated with the first transaction to the fraud detection computer system, and adding the received first transaction to the first set of transactions stored in the transaction data store;

the fraud detection computer system comprising a second processor and second memory storing executable instructions that when executed by the second processor causes the second processor to perform operations, comprising:

generating a score for the received first transaction utilizing the received first parameters and the first ML model and denying or approving the first transaction by comparing the score to the threshold;

a communication interface connected to the transaction system, the fraud detection computer system and a block rate matching system;

the block rate matching system comprising a fraud model testing manager, a block rate testing manager, a third processor and a third memory storing executable instructions that when executed by the third processor causes the third processor to perform operations, comprising:

storing a second ML model, and, via the fraud model testing manager:

accessing, via the fraud model testing manager and over the communication interface, the first set of transactions in the transaction data store, generating, via the fraud model testing manager, a plurality of candidate thresholds and scoring, via the fraud model testing manager for each candidate threshold in the plurality of candidate thresholds, each transaction in the set of transactions, utilizing, via the fraud model testing manager, the second ML model and determining, via the fraud model testing manager, whether each transaction in the set of transactions would be approved or denied,

accessing, via the fraud model testing manager over the communication interface, transactions of the received first set of transactions received after the plurality of candidate thresholds were generated, applying, via the fraud model testing manager, each of the generated candidate thresholds to transactions received after the plurality of candidate thresholds were generated, and determining, via the fraud model testing manager for each of the plurality of candidate thresholds, a second block rate, by dividing a number of transactions that were blocked by a number of received transactions,

determining, via the fraud model testing manager, an adjusted threshold from the plurality of candidate thresholds based on the first block rate and the second block rate, and

communicating, via the block rate testing manager over the communication interface, the second model and the adjusted threshold to the fraud detection computer system;

wherein the first memory stores further executable instructions that when executed by the first processor causes the first processor to perform further operations, comprising:

receiving, via the transaction manager, a second transaction associated with the merchant from the network, sending, via the transaction manager, second parameters associated with the second transaction to the fraud detection computer system, and storing, via the transaction manager, the received second transaction in the transaction data store; and

wherein the second memory stores further executable instructions that when executed by the second processor causes the second processor to perform further operations, comprising:

replacing, the first ML model with the communicated second ML model and the threshold with the communicated adjusted threshold; and

generating a score for the received second transaction utilizing the received second parameters and the communicated second ML model, and denying or approving the second transaction by comparing the score generated utilizing the second parameters and the communicated second ML model to the communicated adjusted threshold.

2 . The system of claim 1 , wherein the third memory stores additional executable instructions that when executed by the third processor causes the third processor to perform further operations, comprising, via the fraud model testing manager,

accessing, in the transaction data store, a set of historical transaction data;

performing fraud detection scoring of a sampling of transactions from the set of historical transaction data using the first ML model; and

determining the first block rate of the first ML model based on a value of the threshold.

3 . The system of claim 2 , wherein the determining via the fraud model testing manager the value of the adjusted threshold further comprises:

performing fraud detection scoring of at least a portion of the sampling of transactions using the second ML model;

generating the plurality of candidate thresholds;

determining a candidate block rate of each of the plurality of candidate thresholds against a subset of the sampling of transactions;

selecting, as the value of the adjusted threshold, one of the plurality of candidate thresholds that provides a closest block rate to the first block rate; and

deploying the second ML model with the value of the adjusted threshold for performing fraud detection on future transactions processed by system.

4 . The system of claim 3 , wherein the generating via the fraud model testing manager the plurality of candidate thresholds further comprises:

sweeping first potential threshold values for fraud detection against scores generated by the second ML model for a first subset of the sampling of transactions;

detecting a first potential threshold value from among the first potential threshold values that provides a first candidate block rate within a predetermined amount of the first block rate; and

selecting the detected first potential threshold value as a first candidate threshold.

5 . The system of claim 4 , wherein the generating via the fraud model testing manager the plurality of candidate thresholds further comprises:

sweeping second potential threshold values for fraud detection against scores generated by the second ML model for a second subset of the sampling of transactions;

detecting a second potential threshold value from among the second potential threshold values that provides a second candidate block rate within the predetermined amount of the first block rate; and

selecting the detected second potential threshold value as a second candidate threshold.

6 . The system of claim 5 , wherein the generating via the fraud model testing manager the plurality of candidate thresholds further comprises:

sweeping third potential threshold values for fraud detection against weighted scores generated by the second ML model for the first or second subset of the sampling of transactions;

detecting a third potential threshold value from among the second potential threshold values that provides a third candidate block rate within the predetermined amount of the first block rate; and

selecting the detected third potential threshold value as a third candidate threshold.

7 . The system of claim 6 , wherein the weighted scores are generated by applying one or more weightings to scores generated by the second ML model.

8 . The system of claim 6 , wherein the third memory stores additional executable instructions that when executed by the third processor causes the third processor to perform operations, comprising:

determining a block rate of each of the first, second, and third candidate thresholds against the second subset of the sampling of transactions; and

selecting, as the adjusted threshold, one of the first, second, and third candidate thresholds that provides the closest block rate to the first block rate.

9 . The system of claim 6 , wherein the first subset of the sampling of transactions comprising transactions from one of a plurality of periods of time of the sampling of transactions, and the second subset of the sampling of transactions comprising transactions from a different one of the plurality of periods of time of the sampling of transactions.

10 . The system of claim 1 , wherein the third memory stores additional executable instructions that when executed by the third processor causes the third processor to perform operations, comprising:

wherein prior to communicating, via the block rate testing manager over the interface, the second model and the adjusted threshold to the fraud detection computer system and replacing, the first ML model with the communicated second ML model and the threshold with the communicated adjusted threshold,

determining whether the adjusted threshold when used with the second ML model to detect transaction fraud provides a shadow testing block rate within a predetermined margin of the first block rate of the threshold used by the first ML model by performing shadow testing, by the fraud model testing manager, of a plurality of new transactions;

when the shadow testing block rate of the adjusted threshold used with the second ML model is within the predetermined margin, deploying the second ML model using the adjusted threshold; and

when the shadow testing block rate of the adjusted threshold used with the second ML model fails to be within the predetermined margin, initiating a modification of the threshold used by the second ML model.

11 . The system of claim 1 , wherein the second ML model is one of a new ML model, an updated ML model, or a retrained version of the first ML model.

12 . A method performed by a system, comprising:

continuously receiving, by a transaction system, a first set of transactions from a network,

receiving, by the transaction system, from a computer system of a merchant a threshold,

storing, by the transaction system, the threshold and a first block rate,

storing, by the transaction system, the first set of transactions for the merchant in a data store of the transaction system,

receiving, by a transaction manager of the transaction system, a first transaction associated with the merchant from the network, and

sending, by the transaction system, first parameters associated with the first transaction to a fraud detection computer system, and adding the received first transaction to the first set of transactions stored in the transaction data store;

generating, by the fraud detection computer system, a score for the received first transaction utilizing the received first parameters and a first machine learning (ML) model and denying or approving, by the fraud detection computer system, the first transaction by comparing the score to the threshold, wherein the fraud detection computer system includes the first ML model, and wherein the transaction system includes the fraud detection computer system;

storing, by a block rate matching system, a second ML model;

accessing, by a fraud model testing manager of the block rate matching system via a communication interface, the first set of transactions in the transaction data store, generating, by the fraud model testing manager, a plurality of candidate thresholds and scoring, by the fraud model testing manager for each candidate threshold in the plurality of candidate thresholds, each transaction in the set of transactions, utilizing, by the fraud model testing manager, the second ML model and determining whether each transaction in the set of transactions would be approved or denied, wherein the communication interface is connected to the transaction system, the fraud detection system and the block rate matching system,

accessing, by the fraud model testing manager via the communication interface transactions of the received first set of transactions received after the plurality of candidate thresholds were generated, applying, by the fraud model testing manager, each of the generated candidate thresholds to transactions received after the plurality of candidate thresholds were generated, and determining, by the fraud model testing manager for each of the plurality of candidate thresholds, a second block rate, by dividing a number of transactions that were blocked by a number of received transactions,

determining, by the fraud model testing manager, an adjusted threshold from the plurality of candidate thresholds based on the first block rate and the second block rate, and

communicating, by a block rate testing manager of the block rate matching system via the communication interface, the second model and the adjusted threshold to the fraud detection computer system;

receiving, by the transaction manager, a second transaction associated with the merchant from the network, sending, by the transaction manager, second parameters associated with the second transaction to the fraud detection computer system, and storing, by the transaction manager, the received second transaction in the data store;

replacing, by the fraud detection computer system, the first ML model with the communicated second ML model and the threshold with the communicated adjusted threshold; and

generating, by the fraud detection computer system, a score for the received second transaction utilizing the received second parameters and the communicated second ML model, and denying or approving, by the fraud detection computer system, the second transaction by comparing the score generated utilizing the second parameters and the communicated second ML model to the communicated adjusted threshold.

13 . The method of claim 12 , further comprising:

accessing, by the fraud model testing manager, in the transaction data store, a set of historical transaction data; performing fraud detection scoring of a sampling of transactions from the set of historical transaction data using the first ML model; and

determining, by the fraud model testing manager, the first block rate of the first ML model based on a value of the threshold.

14 . The method of claim 12 , wherein the second ML model is one of a new ML model, an updated ML model, or a retrained version of the first ML model.

15 . The method of claim 12 , wherein the threshold used by the first ML model is set by a user of the merchant.