Systems and methods for detecting prejudice bias in machine-learning models
Aspects of the present invention provide methods, apparatuses, systems, computing devices, computing entities, and/or the like for detecting prejudice bias in machine-learning models and/or data sets used in training, testing, and/or validating the models. In accordance various aspects, a method is provided comprising: receiving a data set used for training, testing, and/or validating a model that comprises data instances; generating, using a classification model, a prediction of applicability for each sub-category of a plurality of sub-categories for each bias category of a plurality of bias categories for each data instance; determining that a particular sub-category for a particular bias category is applicable to a proportion of the data set, wherein predictions of applicability for the particular sub-category generated for the proportion of the data set satisfies a threshold; and determining, based on the proportion, that the data set has a prejudice bias with respect to the particular bias category.
1. A method comprising:
generating a plurality of outputs by processing a known data set using a machine-learning model, wherein the known data set comprises a plurality of data instances associated with a plurality of sub-categories for each bias category in a plurality of bias categories in proportions to represent each sub-category of the plurality of sub-categories for each bias category in the plurality of bias categories;
generating a plurality of result instances comprising combinations of the plurality of data instances and the plurality of outputs, wherein a result instance of the plurality of result instances comprises a combination of a data instance of the plurality of data instances and a corresponding output of the plurality of outputs generated for the data instance utilizing the machine-learning model;
providing the result instance comprising the combination of the data instance and the corresponding output of the machine-learning model to a classification model;
generating, by computing hardware utilizing the classification model to process the result instance, a prediction of applicability for each sub-category of the plurality of sub-categories for each bias category of the plurality of bias categories for the combination of the data instance and the corresponding output of the machine-learning model in the result instance, wherein:
(a) the classification model comprises an ensemble comprising a multi-label classifier for each bias category of the plurality of bias categories, wherein the plurality of bias categories comprises one or more of religion, sexual orientation, age, ethnicity, gender, location, or political opinions, and
(b) each multi-label classifier is configured to generate the prediction of applicability by generating a probability that each sub-category of the plurality of sub-categories for a corresponding bias category of the plurality of bias categories applies to the combination of the data instance and the corresponding output of the machine-learning model;
determining, by the computing hardware and according to a plurality of predictions generated using the classification model for the plurality of sub-categories of the plurality of bias categories, that a particular sub-category of the plurality of sub-categories for a particular bias category of the plurality of bias categories is applicable to a proportion of the plurality of result instances, wherein a prediction value of the prediction of applicability for the particular sub-category for each applicable result instance found in the proportion of the plurality of result instances is at least a threshold prediction value;
comparing the proportion of the plurality of result instances in the particular sub-category of the particular bias category to a threshold percentage of a data set representing the particular bias category;
determining, by the computing hardware, that the machine-learning model has a prejudice bias with respect to the particular bias category in response to determining that the proportion of the plurality of result instances in the particular sub-category satisfies the threshold percentage of the data set representing the particular bias category; and
causing, by the computing hardware, a computing system to:
generate a modified data set by adding data instances for one or more of the plurality of bias categories based on the prejudice bias to the known data set; and
re-train the machine-learning model using the modified data set to operate in a less biased manner for an artificial intelligence application.
2. The method of claim 1 , further comprising generating a notification indicating the prejudice bias that is communicated to an individual.
3. The method of claim 1 , wherein generating the modified data set comprises:
determining applicable sub-categories of the plurality of bias categories for a pool of data instances;
adding data instances for an underrepresented sub-category of the plurality of bias categories based on the prejudice bias from the pool of data instances to the known data set; and
removing a plurality of data instances applicable to the particular sub-category.
4. The method of claim 3 , further comprising:
generating, by computing hardware and with the classification model, a prediction of applicability for each sub-category of the plurality of sub-categories for each bias category of the plurality of bias categories for each data instance of a plurality of data instances found in the modified data set; and
determining, by the computing hardware based on the prediction of applicability generated for each sub-category of the plurality of sub-categories for each bias category of the plurality of bias categories for each data instance, that the modified data set does not have a prejudice bias.
5. The method of claim 1 , further comprising performing an action to suspend the machine-learning model from being used in the artificial intelligence application.
6. The method of claim 1 , further comprising determining that the known data set has a prejudice bias by determining, using an additional classification model, that a proportion of data instances of the known data set corresponding to the particular sub-category of the plurality of bias categories is at least a threshold.
7. The method of claim 1 , wherein generating the modified data set comprises:
processing a pool of data instances to identify data instances corresponding to a sub-category that is underrepresented in the known data set; and
adding the data instances correspond to the sub-category that is underrepresented in the known data set to the known data set.
8. The method of claim 1 , wherein the proportion of the plurality of result instances is associated with a location and the method further comprises:
mapping, by the computing hardware based on the location, the prejudice bias to a factor influencing a risk associated with the machine-learning model having the prejudice bias; and
determining, by the computing hardware based on the factor, the risk associated with the machine-learning model having the prejudice bias.
9. A system comprising:
a non-transitory computer-readable medium storing instructions; and
a processing device communicatively coupled to the non-transitory computer-readable medium,
wherein, the processing device is configured to execute the instructions and thereby perform operations comprising:
generating a plurality of outputs by processing a known data set using a machine-learning model, wherein the known data set comprises a plurality of data instances associated with a plurality of sub-categories for each bias category in a plurality of bias categories in proportions to represent each sub-category of the plurality of sub-categories for each bias category in the plurality of bias categories;
generating a plurality of result instances comprising combinations of the plurality of data instances and the plurality of outputs, wherein a result instance of the plurality of result instances comprises a combination of a data instance of the plurality of data instances and a corresponding output of the plurality of outputs generated for the data instance utilizing the machine-learning model;
providing the result instance comprising the combination of the data instance and the corresponding output of the machine-learning model to a classification model;
generating, by utilizing the classification model to process the result instance, a prediction of applicability for each sub-category of the plurality of sub-categories for each bias category of the plurality of bias categories for the combination of the data instance and the corresponding output of the machine-learning model in the result instance, wherein the classification model comprises an ensemble comprising a multi-label classifier for each bias category of the plurality of bias categories, wherein the plurality of bias categories comprises one or more of religion, sexual orientation, age, ethnicity, gender, location, or political opinions, and wherein each multi-label classifier is configured to generate the prediction of applicability by generating a probability that each sub-category of the plurality of sub-categories for a corresponding bias category of the plurality of bias categories applies to the combination of the data instance and the corresponding output of the machine-learning model;
determining, according to a plurality of predictions generated using the classification model for the plurality of sub-categories of the plurality of bias categories, that a particular sub-category of the plurality of sub-categories for a particular bias category of the plurality of bias categories is applicable to a proportion of the plurality of result instances, wherein a prediction value of the prediction of applicability for the particular sub-category for each applicable result instance found in the proportion of the plurality of result instances is at least a threshold prediction value;
comparing the proportion of the plurality of result instances in the particular sub-category of the particular bias category to a threshold percentage of a data set representing the particular bias category;
determining that the machine-learning model has a prejudice bias with respect to the particular bias category in response to determining that the proportion of the plurality of result instances in the particular sub-category satisfies the threshold percentage of the data set representing the particular bias category; and
causing a computing system to:
generate a modified data set by adding data instances for one or more of the plurality of bias categories based on the prejudice bias to the known data set; and
re-train the machine-learning model using the modified data set to operate in a less biased manner for an artificial intelligence application.
10. The system of claim 9 , wherein the operations further comprise generating a notification indicating the prejudice bias that is communicated to an individual.
11. The system of claim 9 , wherein the operations further comprise providing an interface to an analytics tool used in identifying a component of the machine-learning model influencing the machine-learning model having the prejudice bias.
12. The system of claim 9 , wherein determining that the machine-learning model has the prejudice bias comprises at least one of determining the proportion of the plurality of result instances is less than the threshold percentage and the prejudice bias indicates the particular sub-category is underrepresented in the plurality of result instances or determining the proportion of the plurality of result instances is greater than the threshold percentage and the prejudice bias indicates the particular sub-category is overrepresented in the plurality of result instances.
13. The system of claim 9 , wherein determining that the machine-learning model has the prejudice bias involves determining that the proportion of the plurality of result instances comprises a number of result instances that are falsely applicable to the particular sub-category satisfies the threshold percentage.
14. The system of claim 9 , wherein the operations further comprise processing one or more data instances of a pool of data instances to identify applicable sub-categories, wherein generating the modified data set is based on processing the one or more data instances of the pool of data instances, and wherein the added data instances correspond to a sub-category that is underrepresented in the known data set.
15. The system of claim 9 , wherein the proportion of the plurality of result instances is associated with a location and the operations further comprise:
mapping, based on the location, the prejudice bias to a factor influencing a risk associated with the machine-learning model having the prejudice bias; and
determining, based on the factor, the risk associated with the machine-learning model having the prejudice bias.
16. A computing system comprising:
first computing hardware configured for:
generating a plurality of outputs by processing a known data set using a machine-learning model, wherein the known data set comprises a plurality of data instances associated with a plurality of sub-categories for each bias category in a plurality of bias categories in proportions to represent each sub-category of the plurality of sub-categories for each bias category in the plurality of bias categories,
generating a plurality of result instances comprising combinations of the plurality of data instances and the plurality of outputs, wherein a result instance of the plurality of result instances comprises a combination of a data instance of the plurality of data instances and a corresponding output of the plurality of outputs generated for the data instance utilizing the machine-learning model,
providing the result instance comprising the combination of the data instance and the corresponding output of the machine-learning model to a classification model,
generating, by utilizing the classification model to process the result instance, a prediction of applicability for a sub-category of the plurality of sub-categories for a bias category of the plurality of bias categories for the combination of the data instance and the corresponding output of the machine-learning model in the result instance, wherein the classification model comprises an ensemble comprising a multi-label classifier for each bias category of the plurality of bias categories, wherein the plurality of bias categories comprises one or more of religion, sexual orientation, age, ethnicity, gender, location, or political opinions, and wherein each multi-label classifier is configured to generate the prediction of applicability by generating a probability that each sub-category of the plurality of sub-categories for a corresponding bias category of the plurality of bias categories applies to the combination of the data instance and the corresponding output of the machine-learning model,
determining, according to a plurality of predictions generated using the classification model for the plurality of sub-categories of the plurality of bias categories, that a particular sub-category of the plurality of sub-categories for a particular bias category of the plurality of bias categories is applicable to a proportion of the plurality of result instances, wherein a prediction value of the prediction of applicability for the particular sub-category for each applicable result instance found in the proportion of the plurality of result instances is at least a threshold prediction value, and
comparing the proportion of the plurality of result instances in the particular sub-category of the particular bias category to a threshold percentage of a data set representing the particular bias category;
determining that the machine-learning model has a prejudice bias with respect to the particular bias category in response to determining that the proportion of the plurality of result instances in the particular sub-category satisfies the threshold percentage of the data set representing the particular bias category; and
second computing hardware communicatively coupled to the first computing hardware and configured for:
performing, based on the prejudice bias, an action to:
generate a modified data set by adding data instances for one or more of the plurality of bias categories to the known data set; and
re-train a machine-learning model using the modified data set to operate in a less biased manner for an artificial intelligence application.
17. The computing system of claim 16 , wherein the first computing hardware is further configured for generating a notification indicating the prejudice bias that is communicated to an individual and transmitting the notification to the second computing hardware.
18. The computing system of claim 16 , wherein performing the action further comprises suspending the machine-learning model from being used in the artificial intelligence application.
19. The computing system of claim 16 , wherein the first computing hardware is further configured for:
determining, using an additional classification model, that a plurality of data instances of the known data set are applicable to the particular sub-category;
identifying a recommendation comprising at least one of an addition or a removal of the plurality of data instances applicable to the particular sub-category to make to the known data set; and
providing the recommendation through a notification communicated to a computing device of an individual.
20. The computing system of claim 16 , wherein the proportion of the plurality of result instances is associated with a location and the first computing hardware is further configured for:
mapping, based on the location, the prejudice bias to a factor influencing a risk associated with the machine-learning model having the prejudice bias; and
determining, based on the factor, the risk associated with the machine-learning model having the prejudice bias.