Intelligent detection of bias within an artificial intelligence model
Systems and methods to receive a text sentence and an identification of a class, calculate a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, calculate a protected gradient score for each token in the text sentence, aggregate the protected gradient scores of the tokens to form a sentence-level protected gradient score for the sentence, determine a fairness indicator of the AI model based on the sentence-level protected gradient score, and responsive to the fairness indicator being below a threshold, prevent deployment of the AI model. Intelligent detection of bias of an AI model with respect to a protected class is determined via a protected gradient score for the protected class at a token level of a text sentence.
1 . A method comprising:
receiving a text sentence and an identification of a class;
determining a bias direction with respect to the class for an embedding model used by an artificial intelligence (AI) model to be analyzed;
for each token in the text sentence, determining a protected gradient score,
wherein determining the protected gradient score includes:
determining a directional derivative based on an output of the artificial intelligence model based on the token and with respect to the bias direction to generate a protected gradient; and
determining a dot product of the protected gradient and the bias direction, wherein the protected gradient score is based on the dot product;
aggregating the protected gradient scores of the tokens to form a sentence-level protected gradient score for the sentence;
determining a fairness indicator of the AI model based on the sentence-level protected gradient score;
responsive to the fairness indicator being below a threshold, remediating the AI model to form a remediated AI model; and
deploying the remediated AI model in production.
2 . The method of claim 1 , further comprising:
determining a second fairness indicator for the remediated AI model based on a second sentence-level protected gradient score, wherein the second fairness indicator satisfies the threshold.
3 . The method of claim 1 , wherein remediating the AI model includes forming a new AI model using new training data selected to reduce bias with respect to the class.
4 . The method of claim 1 , wherein the bias direction is or is defined by a set of two or more vectors expressing a bias subspace.
5 . The method of claim 1 , wherein the bias direction expresses continuum in multi-dimensional space such that movement along the bias direction modifies a contribution of bias with respect to the class.
6 . The method of claim 1 , wherein determining the bias direction includes:
obtaining a corpus of token pairs related to the class;
determining token embeddings for each token of the pairs of the corpus using the embedding model;
performing principal component analysis on the embeddings such that variance associated with the class comes to lie on a first principal component that expresses a biased portion of the meaning of the tokens; and
determining the bias direction using the first principal component.
7 . The method of claim 1 , further comprising:
generating the text sentence using a large language model.
8 . The method of claim 1 , further comprising:
obtaining the text sentence from a training or validation set.
9 . The method of claim 1 , wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable.
10 . The method of claim 1 , wherein determining the fairness indicator of the AI model based on the sentence-level protected gradient score includes determining the fairness indicator of the AI model based on multiple sentence-level protected gradient scores determined for respective different sentences.
11 . The method of claim 1 , wherein the class is a protected class under laws of a jurisdiction in which the AI model will operate.
12 . The method of claim 1 , wherein the protected gradient scores of the tokens are each a scalar value expressing an extent to which a prediction of the AI model depends on a semantic association of the prediction with a protected class.
13 . The method of claim 1 , wherein the protected gradient scores of the tokens each represent a numerical amount of undesired sensitivity of the AI model to a biased component of the meaning of a respective token.
14 . A non-transitory computer readable medium having instructions that, when executed by one or more processors, cause the one or more processors to:
determine an AI model to test, wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable;
receive a text sentence and an identification of a class;
determine a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, wherein to determine the bias direction includes to:
obtain a corpus of token pairs related to the class;
determine token embeddings for each token of the pairs of the corpus using the embedding model;
perform principal component analysis on the embeddings such that variance associated with the class comes to lie on a first principal component that expresses a biased portion of the meaning of the tokens; and
determine the bias direction using the first principal component;
for each token in the text sentence, determine a protected gradient score, wherein determining the protected gradient score includes to:
determine a directional derivative based on an output of the artificial intelligence model based on the token and with respect to the bias direction to generate a protected gradient; and
determine a dot product of the protected gradient and the bias direction, wherein the protected gradient score is based on the dot product;
aggregate the protected gradient scores of the tokens to form a sentence-level protected gradient score for the sentence;
determine a fairness indicator of the AI model based on the sentence-level protected gradient score, wherein determining the fairness indicator of the AI model based on the sentence-level protected gradient score includes determining the fairness indicator of the AI model based on multiple sentence-level protected gradient scores determined for respective different sentences; and
responsive to the fairness indicator being below a threshold, remediate the AI model to form a remediated AI model and deploy the remediated AI model in production.
15 . The non-transitory computer readable medium of claim 14 , wherein the instructions further cause the one or more processors to:
cause a large language model to generate the text sentence.
16 . The non-transitory computer readable medium of claim 14 , wherein the bias direction is or is defined by a set of two or more vectors expressing a bias subspace.
17 . A system comprising:
one or more processors; and
memory having instructions that, when executed by the one or more processors, cause the one or more processors to:
determine an AI model to test, wherein the AI model is a trained deep learning model such that a gradient of the trained deep learning model with respect to each token of the sentence is calculable;
receive a text sentence and an identification of a class;
determine a bias direction with respect to the class for an embedding model used by an artificial intelligence model to be analyzed, wherein to determine the bias direction includes to:
obtain a corpus of token pairs related to the class;
determine token embeddings for each token of the pairs of the corpus using the embedding model;
perform principal component analysis on the embeddings such that variance associated with the class comes to lie on a first principal component that expresses a biased portion of the meaning of the tokens; and
determine the bias direction using the first principal component;
for each token in the text sentence, determine a protected gradient score,
wherein determining the protected gradient score includes to:
determine a directional derivative based on an output of the artificial intelligence model based on the token and with respect to the bias direction to generate a protected gradient; and
determine a dot product of the protected gradient and the bias direction, wherein the protected gradient score is based on the dot product;
aggregate the protected gradient scores of the tokens to form a sentence-level protected gradient score for the sentence;
determine a fairness indicator of the AI model based on the sentence-level protected gradient score, wherein determining the fairness indicator of the AI model based on the sentence-level protected gradient score includes determining the fairness indicator of the AI model based on multiple sentence-level protected gradient scores determined for respective different sentences; and
responsive to the fairness indicator being below a threshold, remediate the AI model to form a remediated AI model and deploy the remediated AI model in production.