Systems and methods for frequent machine learning model retraining and rule optimization
Systems and methods for frequent machine learning model retraining and rule optimization are disclosed. In accordance with aspects, a method may include generating a challenger machine learning model based on a production machine learning model; training the challenger machine learning model on a plurality of datasets; scoring historical data with the challenger machine learning model, wherein the scoring produces a respective score for each record of a plurality of records in the historical data; determining that the challenger model performs within predetermined thresholds based on the scoring; selecting an optimal scaler value for a rule based on execution of the rule with a range of scaler values applied to the respective score for each record of the plurality of records evaluated by the rule; determining that the optimal scaler value outperforms a production scaler value; and promoting the challenger model and the optimal scaler value to a production environment.
1 . A method for frequent machine learning model retraining and rule optimization in a payment transaction fraud detection system implemented by at least one computing device, comprising:
generating a challenger machine learning model based on a production machine learning model;
training the challenger machine learning model on a plurality of datasets, wherein each of the plurality of datasets includes data records collected on a date defined as a number of days previous to a current date, wherein the plurality of datasets include historical payment transaction data;
scoring the historical payment transaction data with the challenger machine learning model, wherein the scoring produces a respective score for each record of a plurality of records in the plurality of datasets, wherein the scoring produces respective fraud scores for said each record of the plurality of records;
determining that the challenger model performs within predetermined thresholds based on the scoring;
selecting an optimal scaler value for a rule based on execution of the rule with a range of scaler values applied to the respective score for said each record of the plurality of records evaluated by the rule;
determining that the optimal scaler value outperforms a production scaler value; and
promoting the challenger model and the optimal scaler value to a production environment to automatically process payment transaction data for fraud detection by (1) calculating a model score of the payment transaction data and assigning a return on investment value to the payment transaction data based on variables of the payment transaction data and (2) calculating a fraud score of the payment transaction data.
2 . The method of claim 1 , wherein the plurality of datasets includes a first dataset, and wherein the first dataset includes data records collected on a date defined as 1 day previous to the current day;
wherein the plurality of datasets includes a second dataset, and wherein the second dataset includes data records collected on a date defined as 14 days previous to the current day; and
wherein the plurality of datasets includes a third dataset and wherein the third dataset includes data records collected on a date defined as 90 days previous to the current day.
3 . The method of claim 1 , wherein the rule is a segment rule for a segment, and wherein the segment is defined by a transaction type.
4 . The method of claim 3 , wherein the segment is one of a plurality of segments, and where each segment of the plurality of segments is defined by a different rule type.
5 . The method of claim 4 , wherein said each segment of the plurality of segments includes a segment rule and wherein said each segment rule includes a scaler value of which can be set independently.
6 . The method of claim 1 , wherein the plurality of datasets includes data collected within a predetermined time window.
7 . The method of claim 6 , wherein the predetermined time window is from the date defined as 1 day previous to the current date to the date defined as 14 days previous to the current date.
8 . The method of claim 1 , wherein the challenger machine learning model is based on an extreme gradient boosting algorithm.
9 . A system for frequent machine learning model retraining and rule optimization in a payment transaction fraud detection system comprising at least one computing device including a processor, wherein the at least one computing device is configured to:
generate a challenger machine learning model based on a production machine learning model;
train the challenger machine learning model on a plurality of datasets, wherein each of the plurality of datasets includes data records collected on a date defined as a number of days previous to a current date, wherein the plurality of datasets include historical payment transaction data;
score the historical payment transaction data with the challenger machine learning model, wherein the scoring produces a respective score for each record of a plurality of records in the plurality of datasets, wherein the scoring produces respective fraud scores for said each record of the plurality of records;
determine that the challenger model performs within predetermined thresholds based on the scoring;
select an optimal scaler value for a rule based on execution of the rule with a range of scaler values applied to the respective score for said each record of the plurality of records evaluated by the rule;
determine that the optimal scaler value outperforms a production scaler value; and
promote the challenger model and the optimal scaler value to a production environment to automatically process payment transaction data for fraud detection by (1) calculating a model score of the payment transaction data and assigning a return on investment value to the payment transaction data based on variables of the payment transaction data and (2) calculating a fraud score of the payment transaction data.
10 . The system of claim 9 , wherein the plurality of datasets includes a first dataset, and wherein the first dataset includes data records collected on a date defined as 1 day previous to the current day;
wherein the plurality of datasets includes a second dataset, and wherein the second dataset includes data records collected on a date defined as 14 days previous to the current day; and
wherein the plurality of datasets includes a third dataset and wherein the third dataset includes data records collected on a date defined as 90 days previous to the current day.
11 . The method of claim 9 , wherein the rule is a segment rule for a segment, and wherein the segment is defined by a transaction type.
12 . The system of claim 11 , wherein the segment is one of a plurality of segments, and where each segment of the plurality of segments is defined by a different rule type.
13 . The system of claim 12 , wherein said each segment of the plurality of segments includes a segment rule and wherein said each segment rule includes a scaler value of which can be set independently.
14 . The system of claim 9 , wherein the plurality of datasets includes historical data is data collected within a predetermined time window.
15 . The system of claim 14 , wherein the predetermined time window is from the date defined as 1 day previous to the current date to the date defined as 14 days previous to the current date.
16 . A non-transitory computer readable storage medium, including instructions stored thereon for frequent machine learning model retraining and rule optimization in a payment transaction fraud detection system, which when read and executed by one or more computers cause the one or more computers to perform steps comprising:
generating a challenger machine learning model based on a production machine learning model;
training the challenger machine learning model on a plurality of datasets, wherein each of the plurality of datasets includes data records collected on a date defined as a number of days previous to a current date, wherein the plurality of datasets include historical payment transaction data;
scoring the historical payment transaction data with the challenger machine learning model, wherein the scoring produces a respective score for each record of a plurality of records in the plurality of datasets, wherein the scoring produces respective fraud scores for said each record of the plurality of records;
determining that the challenger model performs within predetermined thresholds based on the scoring;
selecting an optimal scaler value for a rule based on execution of the rule with a range of scaler values applied to the respective score for said each record of the plurality of records evaluated by the rule;
determining that the optimal scaler value outperforms a production scaler value; and
promoting the challenger model and the optimal scaler value to a production environment to automatically process payment transaction data for fraud detection by (1) calculating a model score of the payment transaction data and assigning a return on investment value to the payment transaction data based on variables of the payment transaction data and (2) calculating a fraud score of the payment transaction data.