Named entity bias detection and mitigation techniques for sentence sentiment analysis
Techniques for named entity bias detection and mitigation for sentence sentiment analysis. In one particular aspect, a method is provided that includes obtaining a training set of labeled examples for training a machine learning model to classify sentiment, preparing a list of named entities using one or more data sources, for each example in the training set of labeled examples with a named entity, replacing the named entity with a corresponding entity type tag to generate a labeled template data set, executing a sampling process for each entity type t within the labeled template data set to generate a augmented invariance data set comprising one or more invariance groups having labeled examples for each entity type t, and training the machine learning model using labeled examples from the augmented invariance data set.
1 . A computer-implemented method comprising:
obtaining an original labeled data set for training a machine learning model to classify sentiment, wherein each example from the original labeled data set is labeled with at least a sentiment classification;
preparing a list of named entities using one or more data sources;
for each example in the original labeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate a labeled template data set;
executing a sampling process for each entity type/within the labeled template data set to generate a first augmented invariance data set comprising one or more invariance groups having labeled examples for each entity type 1 , wherein the sampling process comprises: (i) selecting an example from the labeled template data set comprising an entity type tag of entity type r; and (ii) generating an invariance group by iteratively replacing the entity type tag in the labeled example with a named entity selected from the list of named entities; and
training the machine learning model using the labeled examples from the first augmented invariance data set, wherein the training comprises:
batching the labeled examples from the first augmented invariance data set into data batches, wherein the first augmented invariance data set is assigned a sampling weight hypertuned based on performance of the machine learning model in named entity behavioral tests;
evaluating, using a first objective function, performance of the machine learning model on the labeled examples in each data batch, wherein the first objective function is modified by the sampling weight during the evaluating; and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
2 . The computer-implemented method of claim 1 , further comprising:
obtaining an original unlabeled data set for training the machine learning model to classify sentiment, wherein examples for the original unlabeled data set are obtained from a corpus of sentences and are unlabeled for the sentiment classification;
for each example in the original unlabeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate an unlabeled template data set; and
executing another sampling process for each entity type/within the unlabeled template data set to generate a second augmented invariance data set comprising one or more invariance groups having unlabeled examples for each entity type t, wherein the another sampling process comprises: (i) selecting an example from the unlabeled template data set comprising an entity type tag of entity type t; and (ii) generating an invariance group by iteratively replacing the entity type tag in the unlabeled example with a named entity selected from the list of named entities, wherein the training is performed using the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set.
3 . The computer-implemented method of claim 2 , wherein the training comprises:
batching the unlabeled examples from the second augmented invariance data set into data batches;
evaluating, using a second objective function, performance of the machine learning model on the unlabeled examples in each data batch; and
updating, using the optimization function, the model parameters of the machine learning model based on the evaluating of the performance of the machine learning model using the first objective function and the second objective function.
4 . The computer-implemented method of claim 3 , wherein the training is performed by batching the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set and sampling the data batches interleavingly between the first augmented invariance data set and the second augmented invariance data set.
5 . The computer-implemented method of claim 4 , wherein the second augmented invariance data set is assigned a sampling weight, and the second objective function is modified by the sampling weight during the evaluating of the performance of the machine learning model.
6 . The computer-implemented method of claim 5 , wherein the first objective function is cross-entropy loss and the second objective function is Kullback-Leibler divergence, sum of squared differences, or max of absolute differences.
7 . A system comprising:
one or more data processors; and
one or more non-transitory computer readable media storing instructions which, when executed by the one or more data processors, cause the one or more data processors to perform the following operations:
obtaining an original labeled data set for training a machine learning model to classify sentiment, wherein each example from the original labeled data set is labeled with at least a sentiment classification;
preparing a list of named entities using one or more data sources;
for each example in the original labeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate a labeled template data set;
executing a sampling process for each entity type/within the labeled template data set to generate a first augmented invariance data set comprising one or more invariance groups having labeled examples for each entity type/, wherein the sampling process comprises: (i) selecting an example from the labeled template data set comprising an entity type tag of entity type/; and (ii) generating an invariance group by iteratively replacing the entity type tag in the labeled example with a named entity selected from the list of named entities; and
training the machine learning model using the labeled examples from the first augmented invariance data set, wherein the training comprises:
batching the labeled examples from the first augmented invariance data set into data batches, wherein the first augmented invariance data set is assigned a sampling weight hypertuned based on performance of the machine learning model in named entity behavioral tests;
evaluating, using a first objective function, performance of the machine learning model on the labeled examples in each data batch, wherein the first objective function is modified by the sampling weight during the evaluating; and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
8 . The system of claim 7 , wherein the operations further comprise:
obtaining an original unlabeled data set for training the machine learning model to classify sentiment, wherein examples for the original unlabeled data set are obtained from a corpus of sentences and are unlabeled for the sentiment classification;
for each example in the original unlabeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate an unlabeled template data set; and
executing another sampling process for each entity type/within the unlabeled template data set to generate a second augmented invariance data set comprising one or more invariance groups having unlabeled examples for each entity type t, wherein the another sampling process comprises: (i) selecting an example from the unlabeled template data set comprising an entity type tag of entity type t; and (ii) generating an invariance group by iteratively replacing the entity type tag in the unlabeled example with a named entity selected from the list of named entities, wherein the training is performed using the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set.
9 . The system of claim 8 , wherein the training comprises:
batching the unlabeled examples from the second augmented invariance data set into data batches;
evaluating, using a second objective function, performance of the machine learning model on the unlabeled examples in each data batch; and
updating, using the optimization function, the model parameters of the machine learning model based on the evaluating of the performance of the machine learning model using the first objective function and the second objective function.
10 . The system of claim 9 , wherein the training is performed by batching the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set and sampling the data batches interleavingly between the first augmented invariance data set and the second augmented invariance data set.
11 . The system of claim 10 , wherein the second augmented invariance data set is assigned a sampling weight, and the second objective function is modified by the sampling weight during the evaluating of the performance of the machine learning model.
12 . The system of claim 11 , wherein the first objective function is cross-entropy loss and the second objective function is Kullback-Leibler divergence, sum of squared differences, or max of absolute differences.
13 . A computer-program product tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processors to perform the following operations:
obtaining an original labeled data set for training a machine learning model to classify sentiment, wherein each example from the original labeled data set is labeled with at least a sentiment classification;
preparing a list of named entities using one or more data sources;
for each example in the original labeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate a labeled template data set;
executing a sampling process for each entity type/within the labeled template data set to generate a first augmented invariance data set comprising one or more invariance groups having labeled examples for each entity type t, wherein the sampling process comprises: (i) selecting an example from the labeled template data set comprising an entity type tag of entity type t; and (ii) generating an invariance group by iteratively replacing the entity type tag in the labeled example with a named entity selected from the list of named entities; and
training the machine learning model using the labeled examples from the first augmented invariance data set, wherein the training comprises:
batching the labeled examples from the first augmented invariance data set into data batches, wherein the first augmented invariance data set is assigned a sampling weight hypertuned based on performance of the machine learning model in named entity behavioral tests;
evaluating, using a first objective function, performance of the machine learning model on the labeled examples in each data batch, wherein the first objective function is modified by the sampling weight during the evaluating; and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
14 . The computer-program product of claim 13 , wherein the operations further comprise:
obtaining an original unlabeled data set for training the machine learning model to classify sentiment, wherein examples for the original unlabeled data set are obtained from a corpus of sentences and are unlabeled for the sentiment classification;
for each example in the original unlabeled data set with one or more named entities, replacing each named entity with a corresponding entity type tag to generate an unlabeled template data set; and
executing another sampling process for each entity type/within the unlabeled template data set to generate a second augmented invariance data set comprising one or more invariance groups having unlabeled examples for each entity type t, wherein the another sampling process comprises: (i) selecting an example from the unlabeled template data set comprising an entity type tag of entity type t; and (ii) generating an invariance group by iteratively replacing the entity type tag in the unlabeled example with a named entity selected from the list of named entities, wherein the training is performed using the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set.
15 . The computer-program product of claim 14 , wherein the training comprises:
batching the unlabeled examples from the second augmented invariance data set into data batches;
evaluating, using a second objective function, performance of the machine learning model on the unlabeled examples in each data batch; and
updating, using the optimization function, the model parameters of the machine learning model based on the evaluating of the performance of the machine learning model using the first objective function and the second objective function.
16 . The computer-program product of claim 15 , wherein the training is performed by batching the labeled examples from the first augmented invariance data set and the unlabeled examples from the second augmented invariance data set and sampling the data batches interleavingly between the first augmented invariance data set and the second augmented invariance data set.
17 . The computer-program product of claim 16 , wherein the second augmented invariance data set is assigned a sampling weight, and the second objective function is modified by the sampling weight during the evaluating of the performance of the machine learning model.