Transfer learning and prediction consistency for detecting offensive spans of text

1. A method for natural language processing, comprising:

receiving a span of text comprising an offensive span and a non-offensive span;

generating a contextualized word embedding for each of a plurality of words of the span of text;

generating a refined vector representation for each of the plurality of words based on corresponding contextualized word embedding using a refinement network trained for offensive text recognition using a multi-task loss function including a prediction loss and an auxiliary loss, where the prediction loss is based on a classification network that is applied to an output of the refinement network, and wherein the auxiliary loss is based on a sentiment analysis network that is applied to the output of the refinement network after filtering the output using a filtering component;

and

filtering, using the filtering component, the offensive span based on the refined vector representation to obtain a filtered text.

2. The method of claim 1 , wherein:

the multi-task loss function includes a consistency loss based on a similarity score for each of documents in a training batch.

3. The method of claim 1 , further comprising:

generating label information for each of the plurality of words based on the corresponding refined vector representation, wherein the label information comprises a first value for a first label, a second value for a second label, and a third value for a third label, wherein the first label indicates a word is a first word of the offensive span, the second label indicates the word is within the offensive span, and the third label indicates the word is not within the offensive span.

4. The method of claim 3 , wherein:

the label information comprises a probability distribution over a plurality of labels related to the offensive span.

5. The method of claim 1 , further comprising:

identifying a word comprising a plurality of word pieces;

generating a vector representation for each of the plurality of word pieces; and

averaging the vector representation for each of the plurality of word pieces to produce the contextualized word embedding.

6. An apparatus comprising:

at least one processor; and

at least one memory including instructions executable by the at least one processor to:

receive a span of text comprising an offensive span and a non-offensive span;

generate a contextualized word embedding for each of a plurality of words of the span of text;

generate a refined vector representation for each of the plurality of words based on corresponding contextualized word embedding using a refinement network trained for offensive text recognition using a multi-task loss function including a prediction loss and an auxiliary loss, where the prediction loss is based on a classification network that is applied to an output of the refinement network, and wherein the auxiliary loss is based on a sentiment analysis network that is applied to the output of the refinement network after filtering the output using a filtering component;

and

filter, using the filtering component, the offensive span based on the refined vector representation to obtain a filtered text.

7. The apparatus of claim 6 , wherein:

the multi-task loss function includes a consistency loss based on a similarity score for each of documents in a training batch.

8. The apparatus of claim 6 , further comprising instructions executable by the at least on processor to:

generate label information for each of the plurality of words based on the corresponding refined vector representation, wherein the label information comprises a first value for a first label, a second value for a second label, and a third value for a third label, wherein the first label indicates a word is a first word of the offensive span, the second label indicates the word is within the offensive span, and the third label indicates the word is not within the offensive span.

9. The apparatus of claim 8 , wherein:

the label information comprises a probability distribution over a plurality of labels related to the offensive span.

10. The apparatus of claim 6 , further comprising instructions executable by the at least one processor to:

identify a word comprising a plurality of word pieces;

generate a vector representation for each of the plurality of word pieces; and

average the vector representation for each of the plurality of word pieces to produce the contextualized word embedding.

11. A non-transitory computer readable medium storing code for natural language processing, the code comprising instructions executable by at least one processor to:

receive a span of text comprising an offensive span and a non-offensive span;

generate a contextualized word embedding for each of a plurality of words of the span of text;

generate a refined vector representation for each of the plurality of words based on corresponding contextualized word embedding using a refinement network trained for offensive text recognition using a multi-task loss function including a prediction loss and an auxiliary loss, where the prediction loss is based on a classification network that is applied to an output of the refinement network, and wherein the auxiliary loss is based on a sentiment analysis network that is applied to the output of the refinement network after filtering the output using a filtering component;

and

filter, using the filtering component, the offensive span based on the refined vector representation to obtain a filtered text.

12. The non-transitory computer readable medium of claim 11 , wherein:

the multi-task loss function includes a consistency loss based on a similarity score for each of documents in a training batch.

13. The non-transitory computer readable medium of claim 11 , the code further comprising instructions executable by the at least one processor to:

generate label information for each of the plurality of words based on the corresponding refined vector representation, wherein the label information comprises a first value for a first label, a second value for a second label, and a third value for a third label, wherein the first label indicates a word is a first word of the offensive span, the second label indicates the word is within the offensive span, and the third label indicates the word is not within the offensive span.

14. The non-transitory computer readable medium of claim 13 , wherein:

the label information comprises a probability distribution over a plurality of labels related to the offensive span.

15. The non-transitory computer readable medium of claim 11 , the code further comprising instructions executable by the at least one processor to:

identify a word comprising a plurality of word pieces;

generate a vector representation for each of the plurality of word pieces; and

average the vector representation for each of the plurality of word pieces to produce the contextualized word embedding.