Random forest rule generator
In an example embodiment, a random forest machine learning algorithm is used to create and/or identify rules to apply to an individual entity in a computer system that has a plurality of entities, each with a number of rules. More precisely, rule predicates are used as features of a random forest model built to predict a particular outcome (e.g., a transaction that is fraudulent). Hyperparameters of the random forest model are varied and iterated. A classifier is used to calculate feature importance for all features in the training data. Feature importance may be calculated using permutation feature importance. The N “most important” features are then found from this set. The N “most important” features are then used to find rules above a certain precision and recall rate. These rules may then be backtested and the best rules can be used to generate additional rules.
1 . A method for using one or more rules to classify a prediction related to a first entity, wherein the one or more rules are used to classify a prediction related to a second entity, comprising:
accessing a training set of data, the training set of data including data having a plurality of features and pertaining to the first entity;
using the training set of data to train a random forest model to output the prediction related to the first entity;
calculating, by at least one hardware processor, using a random forest classifier, an importance score for each of the plurality of features, the importance score indicating a relevance of a corresponding feature to the prediction related to the first entity;
identifying, using the random forest classifier, a set of N features having highest importance scores;
generating a plurality of rules, the plurality of rules each including a combination of rule predicates, one or more features from the set of N features, and one or more thresholds, each of the rule predicates comparing a feature from the one or more features from the set of N features with a threshold from the one or more thresholds, wherein each of the generated plurality of rules includes no more than a first preset maximum number of rule predicates, wherein the first preset maximum number of rule predicates for the first entity is different than a second preset maximum number of rule predicates for the second entity;
testing each of the plurality of rules against historical data to measure performance of each of the plurality of rules;
selecting a subset of the plurality of rules based on the respective performance of each of the plurality of rules meeting a performance threshold; and
applying the subset of the plurality of rules using a rules engine, to classify the prediction related to the first entity, the subset of the plurality of rules including the one or more rules used to classify the prediction related to the second entity.
2 . The method of claim 1 , wherein the first entity is a first merchant and the prediction related to the first entity comprises a likelihood that a particular transaction with the first merchant is fraudulent.
3 . The method of claim 1 , wherein N is determined by passing at least some of the training set of data to a separate machine learning model trained to identify a number of important features.
4 . The method of claim 1 , wherein the one or more thresholds are generated by identifying, for each feature in the set of N features, a plurality of quantiles of actual values of the corresponding feature in the training set.
5 . The method of claim 1 , wherein the one or more thresholds are generated using a decision tree.
6 . The method of claim 1 , wherein the one or more thresholds are generated using a boosted tree algorithm.
7 . The method of claim 1 , wherein the one or more thresholds are generated using a recursively partitioned tree algorithm.
8 . The method of claim 1 , wherein the plurality of features includes features that are contained in training data that is kept private by the first entity.
9 . The method of claim 1 , wherein one or more of the plurality of rules includes two or more predicates to establish a multi-dimensional rule.
10 . A system for using one or more rules to classify a prediction related to a first entity, wherein the one or more rules are used to classify a prediction related to a second entity, comprising:
one or more processors; and
at least one memory storing instructions that, when executed by at least one processor among the one or more processors, cause the at least one processor to perform operations comprising:
accessing a training set of data, the training set including data having a plurality of features and pertaining to the first entity;
using the training set of data to train a random forest model to output the prediction related to the first entity;
calculating, by at least one hardware processor, using a random forest classifier, an importance score for each of the plurality of features, the importance score indicating a relevance of a corresponding feature to the prediction related to the first entity;
identifying, using the random forest classifier, a set of N features having highest importance scores;
generating a plurality of rules, the plurality of rules each including a combination of rule predicates, one or more features from the set of N features, and one or more thresholds, each of the rule predicates comparing a feature from the one or more features from the set of N features with a threshold from the one or more thresholds, wherein each of the generated plurality of rules includes no more than a first preset maximum number of rule predicates, wherein the first preset maximum number of rule predicates for the first entity is different than a second preset maximum number of rule predicates for the second entity;
testing each of the plurality of rules against historical data to measure performance of each of the plurality of rules;
selecting a subset of the plurality of rules based on the respective performance of each of the plurality of rules meeting a performance threshold; and
applying the subset of the plurality of rules using a rules engine, to classify the prediction related to the first entity, the subset of the plurality of rules including the one or more rules used to classify the prediction related to the second entity.
11 . The system of claim 10 , wherein the first entity is a first merchant and the prediction related to the first entity is a likelihood that a particular transaction with the first merchant is fraudulent.
12 . The system of claim 10 , wherein N is determined by passing at least some of the training set of data to a separate machine learning model trained to identify a number of important features.
13 . The system of claim 10 , wherein the one or more thresholds are generated by identifying, for each feature in the set of N features, a plurality of quantiles of actual values of the corresponding feature in the training set.
14 . The system of claim 10 , wherein the plurality of features includes features that are contained in training data that is kept private by the first entity.
15 . A non-transitory machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform operations for using one or more rules to classify a prediction related to a first entity, wherein the one or more rules are used to classify a prediction related to a second entity, the operations comprising:
accessing a training set of data, the training set of data including data having a plurality of features and pertaining to the first entity;
using the training set of data to train a random forest model to output the prediction related to the first entity;
calculating, using a random forest classifier, an importance score for each of the plurality of features, the importance score indicating a relevance of a corresponding feature to the prediction related to the first entity;
identifying, using the random forest classifier, a set of N features having highest importance scores;
generating a plurality of rules, the plurality of rules each including a combination of rule predicates, one or more features from the set of N features, and one or more thresholds, each of the rule predicates comparing a feature from the one or more features from the set of N features with a threshold from the one or more thresholds, wherein each of the generated plurality of rules includes no more than a first preset maximum number of rule predicates, wherein the first preset maximum number of rule predicates for the first entity is different than a second preset maximum number of rule predicates for the second entity;
testing each of the plurality of rules against historical data to measure performance of each of the plurality of rules;
selecting a subset of the plurality of rules based on the respective performance of each of the plurality of rules meeting a performance threshold; and
applying the subset of the plurality of rules using a rules engine, to classify the prediction related to the first entity, the subset of the plurality of rules including the one or more rules used to classify the prediction related to the second entity.
16 . The non-transitory machine-readable medium of claim 15 , wherein the first entity is a first merchant and the prediction related to the first entity comprises a likelihood that a particular transaction with the first merchant is fraudulent.
17 . The non-transitory machine-readable medium of claim 15 , wherein Nis determined by passing at least some of the training set of data to a separate machine learning model trained to identify a number of important features.
18 . The non-transitory machine-readable medium of claim 15 , wherein the one or more thresholds are generated by identifying, for each feature in the set of N features, a plurality of quantiles of actual values of the corresponding feature in the training set.
19 . The non-transitory machine-readable medium of claim 15 , wherein the plurality of features includes features that are contained in training data that is kept private by the first entity.
20 . The non-transitory machine-readable medium of claim 15 , wherein one or more of the plurality of rules includes two or more predicates to establish a multi-dimensional rule.