Machine learning model correction
A method, in some implementations, may include obtaining output from a machine learning (ML) model responsive to input data, obtaining initial training data representing training data used to train the ML model, generating, based on the output from the ML model and the initial training data, correction training data that represents a desired alteration to the output from the ML model responsive to one or more particular subgroups in the input data, generating, based on the correction training data, a correction ML model configured to receive, as input, the input data and to output correction values which, when combined with the output from the ML model, perform the desired alteration, and generating corrected output as a combination of the output from the ML model and the output correction values from the correction ML model, and providing, for display, the corrected output.
1 . A computationally efficient method for correcting and improving predictive machine learning models for which one or more attributes are prohibited as model inputs, the method implemented by a computer system and comprising:
until a stopping criterion is reached, iteratively:
applying an ensemble model comprising a trained original machine learning (ML) model to first input data associated with subgroups to generate scores for each row in the first input data;
generating, based on the scores and initial training data representing training data used to train the original ML model, correction training data that comprises one or more difference values for the subgroups, wherein each of the difference values represents a difference in the scores between at least two of the subgroups;
training, based on the correction training data, a predictive correction ML regression model, wherein the correction training data includes corrections to make the scores between the subgroups more similar; and
incorporating the predictive correction ML regression model in the model ensemble;
deploying the model ensemble as a scoring web service hosted by the computer system;
receiving, by the scoring web service, from an external system, and via one or more networks, second input data;
applying the model ensemble to the second input data to generate a score; and
returning the score via the networks to the external system in response to the second input data, wherein the score is fairer and more accurate than the scores generated by the original ML model with respect to the one or more attributes prohibited as model inputs.
2 . The method of claim 1 , wherein the predictive correction ML regression model comprises a plurality of regressors each with a plurality of trees.
3 . The method of claim 1 , wherein the input data comprises the initial training data and includes data from a census income dataset.
4 . The method of claim 1 , wherein the predictive correction ML regression model comprises a linear regression model, neural network, deep neural network, random forest, extra tree forest, radial basis function, XGBoost regressor, CatBoost regressor, TabNet regressor, decision tree regressor, or combinations thereof.
5 . The method of claim 1 , further comprising:
identifying the subgroups using a Bayesian Improved Surname Geocoding method BISG), Zest Race Predictor method (ZRP), or combination thereof;
grouping the scores using the subgroups; and
determining one or more differences in the scores for the subgroups.
6 . The method of claim 1 , wherein the subgroups comprise race, ethnicity, religious beliefs, or a combination thereof.
7 . A non-transitory computer-readable medium for computationally efficient correction and improvement of predictive machine learning models for which one or more attributes are prohibited as model inputs and storing instructions executable by a computer system to:
until a stopping criterion is reached, iteratively:
apply an ensemble model comprising a trained original machine learning (ML) model to first input data associated with subgroups to generate scores for each row in the first input data;
generate, based on the scores and training data used to train the original ML model, correction training data that comprises one or more difference values for the subgroups, wherein each of the difference values represents a difference in the scores between at least two of the subgroups;
train, based on the correction training data, a predictive correction ML regression model, wherein the correction training data includes corrections to make the scores between the subgroups more similar; and
incorporate the predictive correction ML regression model in the model ensemble;
deploy the model ensemble as a scoring web service hosted by the computer system;
receive, by the scoring web service, from an external system, and via one or more networks, second input data;
apply the model ensemble to the second input data to generate a score; and
return the score via the networks to the external system in response to the second input data, wherein the score is fairer and more accurate than the scores generated by the original ML model with respect to the one or more attributes prohibited as model inputs.
8 . A computer system, comprising:
one or more processors; and
memory coupled to the one or more processors and storing instructions for computationally efficient correction and improvement of predictive machine learning models for which one or more attributes are prohibited as model inputs that, when executed by the processors, cause the computer system to:
until a stopping criterion is reached, iteratively:
apply an ensemble model comprising a trained original machine learning (ML) model to first input data associated with subgroups to generate scores for each row in the first input data;
generate, based on the scores and training data used to train the original ML model, correction training data that comprises one or more difference values for the subgroups, wherein each of the difference values represents a difference in the scores between at least two of the subgroups;
train, based on the correction training data, a predictive correction ML regression model, wherein the correction training data includes corrections to make the scores between the subgroups more similar; and
incorporate the predictive correction ML regression model in the model ensemble;
deploy the model ensemble as a scoring web service hosted by the computer system;
receive, by the scoring web service, from an external system, and via one or more networks, second input data;
apply the model ensemble to the second input data to generate a score; and
return the score via the networks to the external system in response to the second input data, wherein the score is fairer and more accurate than the scores generated by the original ML model with respect to the one or more attributes prohibited as model inputs.
9 . The computer system of claim 8 , wherein the instructions, when executed by the processors, further cause the computer system to:
identify the subgroups using a Bayesian Improved Surname Geocoding method (BISG) or a Zest Race Predictor method (ZRP);
group the scores using the subgroups; and
determine one or more differences in the scores for the subgroups.
10 . The computer system of claim 8 , wherein the input data comprises the initial training data and includes data from a census income dataset.
11 . The computer system of claim 8 , wherein the predictive correction ML regression model comprises a plurality of regressors each with a plurality of trees.
12 . The computer system of claim 8 , wherein the predictive correction ML regression model comprises a linear regression model, neural network, deep neural network, random forest, extra tree forest, radial basis function, XGBoost regressor, CatBoost regressor, TabNet regressor, decision tree regressor, or combinations thereof.
13 . The computer system of claim 8 , wherein the subgroups comprise race, ethnicity, religious beliefs, or a combination thereof.
14 . The non-transitory computer-readable medium of claim 7 , wherein the predictive correction ML regression model comprises a plurality of regressors each with a plurality of trees.
15 . The non-transitory computer-readable medium of claim 7 , wherein the input data comprises the initial training data and includes data from a census income dataset.
16 . The non-transitory computer-readable medium of claim 7 , wherein the predictive correction ML regression model comprises a linear regression model, neural network, deep neural network, random forest, extra tree forest, radial basis function, XGBoost regressor, CatBoost regressor, TabNet regressor, decision tree regressor, or combinations thereof.
17 . The non-transitory computer-readable medium of claim 7 , wherein the instructions are further executable by the computer system to:
identify the subgroups using a Bayesian Improved Surname Geocoding method (BISG) or a Zest Race Predictor method (ZRP);
group the scores using the subgroups; and
determine one or more differences in the scores for the subgroups.
18 . The non-transitory computer-readable medium of claim 7 , wherein the subgroups comprise race, ethnicity, religious beliefs, or a combination thereof.
19 . The method of claim 1 , wherein the model ensemble at least partially corrects for bias or inaccuracy in the original ML model, the model ensemble has a computational cost that is linear in a product of a number of different values of a fairness penalty and a size of a test set used to evaluate the model ensemble, and the fairness penalty controls an extent to which a fairness correction is applied to the original ML model.
20 . The computer system of claim 8 , wherein the model ensemble at least partially corrects for bias or inaccuracy in the original ML model, the model ensemble has a computational cost that is linear in a product of a number of different values of a fairness penalty and a size of a test set used to evaluate the model ensemble, and the fairness penalty controls an extent to which a fairness correction is applied to the original ML model.
21 . The non-transitory computer-readable medium of claim 7 , wherein the model ensemble at least partially corrects for bias or inaccuracy in the original ML model, the model ensemble has a computational cost that is linear in a product of a number of different values of a fairness penalty and a size of a test set used to evaluate the model ensemble, and the fairness penalty controls an extent to which a fairness correction is applied to the original ML model.