Data augmentation and batch balancing methods to enhance negation and fairness
Techniques for augmentation and batch balancing of training data to enhance negation and fairness of a machine learning model. In one particular aspect, a method is provided that includes generating a list of demographic words associated with a demographic group, searching an unlabeled corpus of text to identify unlabeled examples in a target domain comprising at least one demographic word from the list of demographic words, rewriting the unlabeled examples to create one or more versions of each of the unlabeled examples and generate a fairness invariance data set, and training the machine learning model using unlabeled examples from the fairness invariance data set.
1. A method comprising:
generating a list of demographic words associated with a demographic group;
searching an unlabeled corpus of text to identify unlabeled examples in a target domain comprising at least one demographic word from the list of demographic words;
rewriting the unlabeled examples to create one or more versions of each of the unlabeled examples and generate a fairness invariance data set, wherein each version of an unlabeled example includes a demographic word that is different from an original demographic word in the unlabeled example, and wherein the one or more versions of each of the unlabeled examples are created by substituting the original demographic word with a demographic word from the list of demographics words or a subset thereof that is associated with the demographic group from which the original demographic word belongs; and
training the machine learning model using unlabeled examples from the fairness invariance data set, wherein the training comprises:
batching the unlabeled examples from the fairness invariance data set into batches, the batching comprises:
grouping the unlabeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the fairness invariance data set, wherein each of the data groups is assigned a sampling weight based on the sentiment class, the data source, or a combination thereof, and
sampling unlabeled examples for each of the batches from the data groups;
evaluating, using a first objective function, performance of the machine learning model on the unlabeled examples in each batch, wherein the first objective function is modified by the sampling weight during the evaluating, and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
2. The method of claim 1 , wherein:
the one or more versions of each of the unlabeled examples are multiple versions, and the multiple versions of each of the unlabeled examples and the unlabeled examples are maintained in groups;
each group comprises an unlabeled original example from the unlabeled corpus of text and the multiple versions thereof that are different from the unlabeled original example by a demographic word; and
the sampling the unlabeled examples comprises selection of the unlabeled original example and the multiple versions thereof such that each group of unlabeled examples is maintained together within a given batch.
3. The method of claim 1 , further comprising obtaining a negation pair data set comprising pairs of labeled examples,
wherein the batching further comprises:
grouping the labeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the negation pair data set; and
sampling labeled examples for each of the batches from the data groups; and
wherein the evaluating further comprises evaluating, using the first objective function, a second objective function, or a combination thereof, the performance of the machine learning model on the labeled examples and the unlabeled examples in each batch.
4. The method of claim 3 , wherein each pair of labeled examples comprises a labeled example and a labeled contradictory form thereof, and the sampling the labeled examples comprises selection of the labeled example and the labeled contradictory form thereof such that each pair of labeled examples is maintained together within a given batch.
5. The method of claim 4 , wherein the first objective function is Kullback-Leibler divergence and the second objective function is cross-entropy loss.
6. A system comprising:
one or more data processors; and
one or more non-transitory computer readable media storing instructions which, when executed by the one or more data processors, cause the one or more data processors to perform the following operations:
generating a list of demographic words associated with a demographic group;
searching an unlabeled corpus of text to identify unlabeled examples in a target domain comprising at least one demographic word from the list of demographic words;
rewriting the unlabeled examples to create one or more versions of each of the unlabeled examples and generate a fairness invariance data set, wherein each version of an unlabeled example includes a demographic word that is different from an original demographic word in the unlabeled example, and wherein the one or more versions of each of the unlabeled examples are created by substituting the original demographic word with a demographic word from the list of demographics words or a subset thereof that is associated with the demographic group from which the original demographic word belongs; and
training the machine learning model using unlabeled examples from the fairness invariance data set, wherein the training comprises:
batching the unlabeled examples from the fairness invariance data set into batches, the batching comprises:
grouping the unlabeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the fairness invariance data set, wherein each of the data groups is assigned a sampling weight based on the sentiment class, the data source, or a combination thereof, and
sampling unlabeled examples for each of the batches from the data groups;
evaluating, using a first objective function, performance of the machine learning model on the unlabeled examples in each batch, wherein the first objective function is modified by the sampling weight during the evaluating, and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
7. The system of claim 6 , wherein:
the one or more versions of each of the unlabeled examples are multiple versions, and the multiple versions of each of the unlabeled examples and the unlabeled examples are maintained in groups;
each group comprises an unlabeled original example from the unlabeled corpus of text and the multiple versions thereof that are different from the unlabeled original example by a demographic word; and
the sampling the unlabeled examples comprises selection of the unlabeled original example and the multiple versions thereof such that each group of unlabeled examples is maintained together within a given batch.
8. The system of claim 6 , wherein the operations further comprise: obtaining a negation pair data set comprising pairs of labeled examples,
wherein the batching further comprises:
grouping the labeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the negation pair data set; and
sampling labeled examples for each of the batches from the data groups; and
wherein the evaluating further comprises evaluating, using the first objective function, a second objective function, or a combination thereof, the performance of the machine learning model on the labeled examples and the unlabeled examples in each batch.
9. The system of claim 8 , wherein each pair of labeled examples comprises a labeled example and a labeled contradictory form thereof, and the sampling the labeled examples comprises selection of the labeled example and the labeled contradictory form thereof such that each pair of labeled examples is maintained together within a given batch.
10. The system of claim 9 , wherein the first objective function is Kullback-Leibler divergence and the second objective function is cross-entropy loss.
11. A computer-program product tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processors to perform the following operations:
generating a list of demographic words associated with a demographic group;
searching an unlabeled corpus of text to identify unlabeled examples in a target domain comprising at least one demographic word from the list of demographic words;
rewriting the unlabeled examples to create one or more versions of each of the unlabeled examples and generate a fairness invariance data set, wherein each version of an unlabeled example includes a demographic word that is different from an original demographic word in the unlabeled example, and wherein the one or more versions of each of the unlabeled examples are created by substituting the original demographic word with a demographic word from the list of demographics words or a subset thereof that is associated with the demographic group from which the original demographic word belongs; and
training the machine learning model using unlabeled examples from the fairness invariance data set, wherein the training comprises:
batching the unlabeled examples from the fairness invariance data set into batches, the batching comprises:
grouping the unlabeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the fairness invariance data set, wherein each of the data groups is assigned a sampling weight based on the sentiment class, the data source, or a combination thereof, and
sampling unlabeled examples for each of the batches from the data groups;
evaluating, using a first objective function, performance of the machine learning model on the unlabeled examples in each batch, wherein the first objective function is modified by the sampling weight during the evaluating, and
updating, using an optimization function, model parameters of the machine learning model based on the evaluating of the performance of the machine learning model.
12. The computer-program product of claim 11 , wherein:
the one or more versions of each of the unlabeled examples are multiple versions, and the multiple versions of each of the unlabeled examples and the unlabeled examples are maintained in groups;
each group comprises an unlabeled original example from the unlabeled corpus of text and the multiple versions thereof that are different from the unlabeled original example by a demographic word; and
the sampling the unlabeled examples comprises selection of the unlabeled original example and the multiple versions thereof such that each group of unlabeled examples is maintained together within a given batch.
13. The computer-program product of claim 11 , wherein the operations further comprise: obtaining a negation pair data set comprising pairs of labeled examples,
wherein the batching further comprises:
grouping the labeled examples into data groups based on sentiment class, data source, or a combination thereof, the data source being the negation pair data set; and
sampling labeled examples for each of the batches from the data groups; and
wherein the evaluating further comprises evaluating, using the first objective function, a second objective function, or a combination thereof, the performance of the machine learning model on the labeled examples and the unlabeled examples in each batch.
14. The computer-program product of claim 13 , wherein each pair of labeled examples comprises a labeled example and a labeled contradictory form thereof, and the sampling the labeled examples comprises selection of the labeled example and the labeled contradictory form thereof such that each pair of labeled examples is maintained together within a given batch.