Model-agnostic system for automatic investigation of the impact of new features on performance of machine learning models
Provided are computing systems, methods, and platforms that automatically investigate and analyze the impact of new features or signals on the performance of a machine learning model by producing a ranked list of the most impactful features from input of a set of candidate features. In particular, one example computing system can import a training dataset associated with a user. The computing system can train a machine learning model for the training dataset and generate baseline metrics for the machine learning model. Correlations between features or signals in the training dataset can be identified and the features or signals can be grouped into clusters based on the correlations. The computing system can determine the importance of each cluster and each feature or signal. A ranked list of signals and their importances can be exported in decreasing order of machine learning model performance lift based on cluster importance and signal importance.
1 . A computing system for automatic analysis of signals for machine learning models, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
importing a training dataset associated with a user, wherein the training dataset comprises one or more candidate features, and the one or more candidate features are each associated with one of one or more candidate signals;
training a machine learning model using the training dataset;
generating baseline metrics for the machine learning model;
identifying one or more correlations between the one or more candidate signals in the training dataset;
grouping the one or more candidate signals into one or more clusters based on the one or more correlations;
for each respective cluster of the one or more clusters:
identifying an importance of the respective cluster, wherein the importance of the respective cluster is indicative of a first estimated performance lift to the machine learning model associated with the respective cluster;
for each respective candidate signal of the one or more candidate signals:
identifying the importance of the respective candidate signal, wherein the importance of the respective candidate signal is indicative of a second estimated performance lift to the machine learning model associated with adding the respective candidate signal as an input to the machine learning model; and
returning a list of the one or more candidate signals and the importance of the respective candidate signal of the one or more candidate signals in decreasing order of machine learning model performance lift based on the importance of the respective cluster associated with the one or more candidate signals and the importance of the respective candidate signal.
2 . The computing system of claim 1 , wherein:
the operations further comprise:
combining the signals which already exist in the machine learning model with the one or more candidate signals; and
dividing the training dataset into test datasets and train datasets.
3 . The computing system of claim 1 , wherein identifying correlations between the one or more candidate signals comprises determining correlations for contiguous-contiguous signal pairs and for categorical-categorical signal pairs.
4 . The computing system of claim 1 , wherein grouping the one or more candidate signals into one or more clusters based on their one or more correlations comprises:
creating a graph, wherein nodes of the graph are the one or more candidate signals and a weight of each edge between the nodes is the correlation between the one or more candidate signals; and
grouping the one or more candidate signals into the one or more clusters that maximize a number of agreements.
5 . The computing system of claim 1 , wherein identifying the importance of the respective cluster comprises:
for the respective cluster:
for the respective signal in the respective cluster:
replacing all values in a test dataset with Nan;
entering the test dataset into the trained machine learning model;
exporting metrics output by the trained machine learning model;
comparing the metrics output by the trained machine learning model with the baseline metrics to determine performance lift; and
for the respective signal in the respective cluster:
assigning the values in the test dataset with the respective signal's value before replacing all values in the test dataset with Nan.
6 . The computing system of claim 1 , wherein identifying the importance of the respective candidate signal of the one or more candidate signals comprises:
for each respective candidate signal:
replacing all values in a column of a test dataset with Nan;
entering the test dataset into the trained machine learning model;
exporting metrics output by the trained machine learning model;
comparing the metrics output by the trained machine learning model with the baseline metrics to determine performance lift; and
assigning the values in the test dataset with the signal's value before replacing all values in the test dataset with Nan.
7 . The computing system of claim 1 , wherein returning a list of the one or more candidate signals and the importance of the respective candidate signal of the one or more candidate signals in decreasing order of machine learning model performance lift based on the importance of the respective cluster associated with the one or more candidate signals and the importance of the respective candidate signal comprises:
sorting the importance of the respective cluster into a ranked list of cluster importances in descending order of importance;
for the respective cluster in the ranked list of cluster importance:
appending all new signals of the respective cluster to a list of unimportant signals when the respective cluster has a signal which already exists in the machine learning model;
selecting the signal with the highest individual signal importance from the respective cluster when the respective cluster does not have a signal which already exists in the machine learning model;
appending the signal with the highest individual signal importance and all signals with the same individual signal importances to a ranked list of signal importances when the respective cluster does not have the signal which already exists in the machine learning model; and
appending the all new signals of the cluster to the list of unimportant signals when the cluster has the signal which already exists in the machine learning model and when the new signal is not appended to the ranked list of signal importances;
exporting the top signals and the signal importance from the ranked list of signal importances.
8 . The computing system of claim 7 , wherein a user can specify a number of top signals and the importance of the respective candidate signal of the one or more candidate signals to be exported.
9 . A computer-implemented method for automatic analysis of signals for machine learning models, the method performed by one or more computing devices and comprising:
importing a training dataset associated with a user, wherein the training dataset comprises one or more candidate features, and the one or more candidate features are each associated with one of one or more candidate signals;
training a machine learning model using the training dataset;
generating baseline metrics for the machine learning model;
identifying one or more correlations between the one or more candidate signals in the training dataset;
grouping the one or more candidate signals into one or more clusters based on the one or more correlations;
for each respective cluster of the one or more clusters:
identifying an importance of the respective cluster, wherein the importance of the respective cluster is indicative of a first estimated performance lift to the machine learning model associated with the respective cluster;
for each respective candidate signal of the one or more candidate signals:
identifying the importance of the respective candidate signal, wherein the importance of the respective candidate signal is indicative of a second estimated performance lift to the machine learning model associated with adding the respective candidate signal as an input to the machine learning model; and
returning a list of the one or more candidate signals and the importance of the respective candidate signal of the one or more candidate signals in decreasing order of machine learning model performance lift based on the importance of the respective cluster associated with the one or more candidate signals and the importance of the respective candidate signal.
10 . The computing system of claim 1 , wherein the first estimated performance lift is associated with adding at least one of the one or more candidate signals as an input to the machine learning model from the respective cluster.
11 . The computing system of claim 1 , wherein the baseline metrics for the machine learning model are indicative of a precision and a recall of the machine learning model.
12 . The computer-implemented method of claim 9 , wherein:
the operations further comprise:
combining the signals which already exist in the machine learning model with the one or more candidate signals; and
dividing the training dataset into test datasets and train datasets.
13 . The computer-implemented method of claim 9 , wherein identifying correlations between the one or more candidate signals comprises determining correlations for contiguous-contiguous signal pairs and for categorical-categorical signal pairs.
14 . The computer-implemented method of claim 9 , wherein grouping the one or more candidate signals into one or more clusters based on their one or more correlations comprises:
creating a graph, wherein nodes of the graph are the one or more candidate signals and a weight of each edge between the nodes is the correlation between the one or more candidate signals; and
grouping the one or more candidate signals into the one or more clusters that maximize a number of agreements.
15 . The computer-implemented method of claim 14 , wherein a cluster can contain a single element.
16 . The computer-implemented method of claim 9 , wherein identifying the importance of the respective cluster comprises:
for the respective cluster:
for the respective signal in the respective cluster:
replacing all values in a test dataset with Nan;
entering the test dataset into the trained machine learning model;
exporting metrics output by the trained machine learning model;
comparing the metrics output by the trained machine learning model with the baseline metrics to determine performance lift; and
for the respective signal in the respective cluster:
assigning the values in the test dataset with the respective signal's value before replacing all values in the test dataset with Nan.
17 . The computer-implemented method of claim 9 , wherein identifying the importance of the respective candidate signal of the one or more candidate signals comprises:
for the respective signal:
replacing all values in a column of a test dataset with Nan;
entering the test dataset into the trained machine learning model;
exporting metrics output by the trained machine learning model;
comparing the metrics output by the trained machine learning model with the baseline metrics to determine performance lift; and
assigning the values in the test dataset with the signal's value before replacing all values in the test dataset with Nan.
18 . The computer-implemented method of claim 9 , wherein returning a list of the one or more candidate signals and the importance of each respective candidate signal of the one or more candidate signals in decreasing order of machine learning model performance lift based on the importance of the respective cluster associated with the one or more candidate signals and the importance of the respective candidate signal comprises:
sorting the importance of the respective cluster into a ranked list of cluster importances in descending order of importance;
for the respective cluster in the ranked list of cluster importance:
appending all new signals of the respective cluster to a list of unimportant signals when the respective cluster has a signal which already exists in the machine learning model;
selecting the signal with the highest individual signal importance from the respective cluster when the respective cluster does not have a signal which already exists in the machine learning model;
appending the signal with the highest individual signal importance and all signals with the same individual signal importances to a ranked list of signal importances when the respective cluster does not have the signal which already exists in the machine learning model; and
appending the all new signals of the cluster to the list of unimportant signals when the cluster has the signal which already exists in the machine learning model and when the new signal is not appended to the ranked list of signal importances;
exporting the top signals and the signal importance from the ranked list of signal importances.
19 . The computer-implemented method of claim 18 , wherein a user can specify a number of top signals and the importance of the respective candidate signal of the one or more candidate signals to be exported.
20 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising:
importing a training dataset associated with a user, wherein the training dataset comprises one or more candidate features, and the one or more candidate features are each associated with one of one or more candidate signals;
training a machine learning model using the training dataset;
generating baseline metrics for the machine learning model;
identifying one or more correlations between the one or more candidate signals in the training dataset;
grouping the one or more candidate signals into one or more clusters based on the one or more correlations;
for each respective cluster of the one or more clusters:
identifying an importance of the respective cluster, wherein the importance of the respective cluster is indicative of a first estimated performance lift to the machine learning model associated with the respective cluster;
for each respective candidate signal of the one or more candidate signals:
identifying the importance of the respective candidate signal, wherein the importance of the respective candidate signal is indicative of a second estimated performance lift to the machine learning model associated with adding the respective candidate signal as an input to the machine learning model; and
returning a list of the one or more candidate signals and the importance of the respective candidate signal of the one or more candidate signals in decreasing order of machine learning model performance lift based on the importance of the respective cluster associated with the one or more candidate signals and the importance of the respective candidate signal.