Systems and methods for automatically retraining machine learning models to remove systematic bias

1 . A computer-implemented method of increasing accuracy of a machine learning model, the method comprising:

generating a candidate machine learning model, wherein the candidate machine learning model is configured to generate probability scores for members;

training the candidate machine learning model using an initial data set, wherein the initial data set includes bias;

testing the candidate machine learning model to generate performance metrics, wherein the performance metrics indicate a bias value correlated to the candidate machine learning model;

determining whether the performance metrics are below a threshold; and

in response to determining that the performance metrics are below the threshold:

generating a proxy feature based on the bias value,

adding the proxy feature to the candidate machine learning model,

calculating weights based on the bias value,

updating the initial data set using the weights to generate an updated data set, wherein the updated data set mitigates bias, and

retraining the candidate machine learning model using the updated data set.

2 . The method of claim 1 further comprising, in response to determining that the performance metrics are not below the threshold, saving the candidate machine learning model as an accepted machine learning model.

3 . The method of claim 1 further comprising:

testing the retrained candidate machine learning model to generate updated performance metrics; and

in response to determining that the updated performance metrics are not below the threshold, saving the retrained candidate machine learning model as an accepted machine learning model.

4 . The method of claim 1 wherein the performance metrics are generated by:

calculating a percentage favorable outcomes metric for a protected class;

calculating a percentage favorable outcomes metric for a control class; and

calculating an impact ratio based on the percentage favorable outcomes metric for the protected class and the percentage favorable outcomes metric for the control class.

5 . The method of claim 4 wherein the percentage favorable outcomes metric for the protected class is calculated by:

determining a first value indicative of a number of members belonging to the protected class having generated probability scores above a second threshold;

determining a second value indicative of a number of members belonging to the protected class; and

calculating the percentage favorable outcomes metric for the protected class by dividing the first value by the second value.

6 . The method of claim 5 wherein the percentage favorable outcomes metric for the protected class is calculated by:

determining a third value indicative of a number of members belonging to the protected class having generated probability scores above a second threshold;

determining a fourth value indicative of a number of members belonging to the protected class; and

calculating the percentage favorable outcomes metric for the protected class by dividing the first value by the second value.

7 . The method of claim 6 wherein the impact ratio is calculated by dividing the percentage favorable outcomes metric for the protected class by the percentage favorable outcomes metric for the control class.

8 . The method of claim 7 wherein generating the proxy feature based on the bias values comprises:

determining a protected class associated with the bias value; and

generating the proxy feature to represent the protected class associated with the bias value.

9 . The method of claim 8 wherein:

the candidate machine learning model is a logistic regression model; and

adding the proxy feature to the machine learning model includes adding an input variable representing the proxy feature to the logistic regression model.

10 . The method of claim 9 wherein calculating weights based on the bias value includes:

dividing the initial data set into categories; and

for each category:

calculating a fifth value related to a number of observances of the protected class or the control class in the category,

calculating a sixth value related to a number of observances of a positive or a negative outcome in the category,

calculating a seventh value related to a total number of observances in the initial data,

calculating an eighth value related to a total number of observances in the category, and

dividing a product of the fifth value and the sixth value with a product of the seventh value and the eighth value.

11 . A system of increasing accuracy of a machine learning model, the system comprising:

memory hardware configured to store instructions; and

processing hardware configured to execute the instructions, wherein the instructions include:

generating a candidate machine learning model, wherein the candidate machine learning model is configured to generate probability scores for members;

training the candidate machine learning model using an initial data set, wherein the initial data set includes bias;

testing the candidate machine learning model to generate performance metrics, wherein the performance metrics indicate a bias value correlated to the candidate machine learning model;

determining whether the performance metrics are below a threshold; and

in response to determining that the performance metrics are below the threshold:

generating a proxy feature based on the bias value,

adding the proxy feature to the candidate machine learning model,

calculating weights based on the bias value,

updating the initial data set using the weights to generate an updated data set, wherein the updated data set mitigates bias, and

retraining the candidate machine learning model using the updated data set.

12 . The system of claim 11 wherein the instructions include, in response to determining that the performance metrics are not below the threshold, saving the candidate machine learning model as an accepted machine learning model.

13 . The system of claim 11 wherein the instructions include:

testing the retrained candidate machine learning model to generate updated performance metrics; and

in response to determining that the updated performance metrics are not below the threshold, saving the retrained candidate machine learning model as an accepted machine learning model.

14 . The system of claim 11 wherein the performance metrics are generated by:

calculating a percentage favorable outcomes metric for a protected class;

calculating a percentage favorable outcomes metric for a control class; and

calculating an impact ratio based on the percentage favorable outcomes metric for the protected class and the percentage favorable outcomes metric for the control class.

15 . The system of claim 14 wherein the percentage favorable outcomes metric for the protected class is calculated by:

determining a first value indicative of a number of members belonging to the protected class having generated probability scores above a second threshold;

determining a second value indicative of a number of members belonging to the protected class; and

calculating the percentage favorable outcomes metric for the protected class by dividing the first value by the second value.

16 . The system of claim 15 wherein the percentage favorable outcomes metric for the protected class is calculated by:

determining a third value indicative of a number of members belonging to the protected class having generated probability scores above a second threshold;

determining a fourth value indicative of a number of members belonging to the protected class; and

calculating the percentage favorable outcomes metric for the protected class by dividing the first value by the second value.

17 . The system of claim 16 wherein the impact ratio is calculated by dividing the percentage favorable outcomes metric for the protected class by the percentage favorable outcomes metric for the control class.

18 . The system of claim 17 wherein generating the proxy feature based on the bias values comprises:

determining a protected class associated with the bias value; and

generating the proxy feature to represent the protected class associated with the bias value.

19 . The system of claim 18 wherein:

the candidate machine learning model is a logistic regression model; and

adding the proxy feature to the machine learning model includes adding an input variable representing the proxy feature to the logistic regression model.

20 . The system of claim 19 wherein calculating weights based on the bias value includes:

dividing the initial data set into categories; and

for each category:

calculating a fifth value related to a number of observances of the protected class or the control class in the category,

calculating a sixth value related to a number of observances of a positive or a negative outcome in the category,

calculating a seventh value related to a total number of observances in the initial data,

calculating an eighth value related to a total number of observances in the category, and

dividing a product of the fifth value and the sixth value with a product of the seventh value and the eighth value.