Machine Learning Feature Stability Alerts
A method for creating machine learning model performance alerts showing the drifting of functions is described herein. The method starts by creating the initial machine learning model using a training data set. This initial machine learning model is then used in production, and the model is updated to account for the production data. To assure the quality of the updated machine learning model, test data results from the initial machine learning model is compared to the results from the updated machine learning model. Each feature is checked to see if the difference is within a p-value and whether the confidence intervals overlap. If not, an alert is generated to take action on the model.
1 . An improved machine learning method comprising:
creating a first machine learning model with training data;
periodically adjusting the first machine learning model with production data to create a second machine learning model;
creating a training dataset by processing the training data through the first machine learning model;
creating a prediction dataset by processing the production data set through the second machine learning model; and
looping through each feature in the prediction dataset:
determining a p-value by comparing the feature in the prediction dataset to the feature in the training dataset; and
if the p-value is less than a constant and a confidence interval for the training dataset does not overlap the confidence interval for the prediction dataset, creating an alert.
2 . The improved machine learning method of claim 1 further comprising performing a T-test to determine the p-value.
3 . The improved machine learning method of claim 1 further comprising performing a binomial proportions test to determine the p-value.
4 . The improved machine learning method of claim 1 further comprising automatically adjusting the first machine learning model based on the alert.
5 . The improved machine learning method of claim 1 further comprising automatically adjusting the second machine learning model based on the alert.
6 . The improved machine learning method of claim 1 further comprising creating a plot of the feature in the prediction dataset.
7 . The improved machine learning method of claim 1 wherein the first machine learning model is created using a Densicube algorithm.
8 . The improved machine learning method of claim 1 wherein the first machine learning model is created using a K-means algorithm.
9 . The improved machine learning method of claim 1 wherein the first machine learning model is created using a Random Forest algorithm.
10 . The improved machine learning method of claim 1 wherein the overlap in the confidence interval uses a mean and a margin of error.
11 . A method for creating machine learning model performance alerts comprising:
creating a first machine learning model with training data;
adjusting the first machine learning model with production data to create a second machine learning model;
creating a training dataset by processing the training data through the first machine learning model;
creating a prediction dataset by processing the production data through the second machine learning model; and
looping through each feature in the prediction dataset:
determining a p-value by comparing the feature in the prediction dataset to the feature in the training dataset; and
if the p-value is less than a constant and a confidence interval for the training dataset does not overlap the confidence interval for the prediction dataset, creating the machine learning model performance alert.
12 . The method of claim 11 further comprising if the feature is numeric, performing a T-test to determine the p-value.
13 . The method of claim 11 further comprising if the feature is not numeric, performing a binomial proportions test to determine the p-value.
14 . The method of claim 11 further comprising automatically adjusting the first machine learning model based on the machine learning model performance alert.
15 . The method of claim 11 further comprising automatically adjusting the second machine learning model based on the machine learning model performance alert.
16 . The method of claim 11 further comprising creating a plot of the feature in the prediction dataset.
17 . The method of claim 11 wherein the first machine learning model is created using a Densicube algorithm.
18 . The method of claim 11 wherein the first machine learning model is created using a K-means algorithm.
19 . The method of claim 11 wherein the first machine learning model is created using a Random Forest algorithm.
20 . The method of claim 11 wherein the overlap in the confidence interval uses a mean and a margin of error.