Method and apparatus for training text classification model
This disclosure relates to a method and an apparatus for training a text classification model. The method may include determining a semantic representation of the training sample using the text classification model and determining a predicted classification result of the training sample based on the semantic representation. The method may further include generating an adversarial sample corresponding to the training sample based on the training sample and perturbation information and determining a semantic representation of the adversarial sample corresponding to the training sample using the text classification model. The method may further include determining a classification loss of the text classification model based on the predicted classification result, determining a contrastive loss of the text classification model based on the semantic representation and the semantic representation of the adversarial sample, and training the text classification model based on the classification loss and the contrastive loss.
1. A method for training a text classification model, executed by a computer device, the method comprising:
obtaining a training sample of the text classification model, the training sample being a text;
determining a semantic representation of the training sample using the text classification model, wherein the semantic representation of the training sample represents a semantic of the training sample;
determining a predicted classification result of the training sample based on the semantic representation;
generating an adversarial sample corresponding to the training sample based on the training sample and perturbation information for the training sample;
determining a semantic representation of the adversarial sample corresponding to the training sample using the text classification model, wherein the semantic representation of the adversarial sample represents a semantic of the adversarial sample, the semantic of the adversarial sample is consistent with the semantic of the training sample;
determining a classification loss of the text classification model based on the predicted classification result of the training sample;
determining a contrastive loss of the text classification model based on the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample; and
training the text classification model based on the classification loss and the contrastive loss.
2. The method of claim 1 , wherein the determining the contrastive loss of the text classification model based on the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample comprises:
determining a first contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and a semantic representation of a different sample, the different sample being a sample belonging to a different category from the training sample;
determining a second contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and a semantic representation of an adversarial sample corresponding to the different sample; and
determining the contrastive loss of the text classification model based on the first contrastive loss and the second contrastive loss.
3. The method of claim 2 , wherein the determining the first contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and the semantic representation of the different sample comprises:
calculating a first similarity, the first similarity being a similarity between the semantic representation of the adversarial sample corresponding to the training sample and the semantic representation of the training sample;
calculating a second similarity, the second similarity being a similarity between the semantic representation of the adversarial sample corresponding to the training sample and the semantic representation of the different sample; and
calculating the first contrastive loss based on the first similarity and the second similarity.
4. The method of claim 2 , wherein the determining the second contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and the semantic representation of the adversarial sample corresponding to the different sample comprises:
calculating a third similarity, the third similarity being a similarity between the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample;
calculating a fourth similarity, the fourth similarity being a similarity between the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the different sample; and
calculating the second contrastive loss based on the third similarity and the fourth similarity.
5. The method of claim 4 , further comprising:
selecting m samples from samples belonging to different categories from the training sample, to obtain different samples, m being a positive integer.
6. The method of claim 1 , wherein the training the text classification model based on the classification loss and the contrastive loss comprises:
determining a total loss of the text classification model based on the classification loss and the contrastive loss; and
adjusting a parameter of the text classification model to minimize the total loss and obtain a trained text classification model.
7. The method of claim 1 , wherein the generating the adversarial sample corresponding to the training sample based on the training sample and the perturbation information comprises:
obtaining a word embedding of the training sample; and
adding the perturbation information to the word embedding of the training sample to obtain a processed word embedding, the processed word embedding being used as the adversarial sample corresponding to the training sample.
8. The method of claim 1 , wherein the generating the adversarial sample corresponding to the training sample based on the training sample and the perturbation information comprises:
adding the perturbation information to a text content of the training sample to obtain processed text information, the processed text information being used as the adversarial sample corresponding to the training sample.
9. The method of claim 1 , further comprising:
creating a first buffer and a second buffer, the first buffer being for storing the semantic representation of the training sample, and the second buffer being for storing the semantic representation of the adversarial sample corresponding to the training sample;
dynamically updating data stored in the first buffer; and
dynamically updating data stored in the second buffer.
10. The method of claim 1 , wherein the text classification model comprises a word embedding extraction unit, an encoder, a classifier, and a contrastive loss calculation unit, wherein,
the word embedding extraction unit is configured to extract a word embedding of an input text;
the encoder is configured to generate a semantic representation of the input text based on the word embedding of the input text;
the classifier is configured to determine a category of the input text based on the semantic representation of the input text; and
the contrastive loss calculation unit is configured to calculate the contrastive loss of the text classification model.
11. An apparatus for training a text classification model, comprising:
a memory operable to store computer-readable instructions; and
a processor circuitry operable to read the computer-readable instructions, the processor circuitry when executing the computer-readable instructions is configured to:
obtain a training sample of the text classification model, the training sample being a text;
determine a semantic representation of the training sample using the text classification model, wherein the semantic representation of the training sample represents a semantic of the training sample;
determine a predicted classification result of the training sample based on the semantic representation;
generate an adversarial sample corresponding to the training sample based on the training sample and perturbation information for the training sample;
determine a semantic representation of the adversarial sample corresponding to the training sample using the text classification model, wherein the semantic representation of the adversarial sample represents a semantic of the adversarial sample, the semantic of the adversarial sample is consistent with the semantic of the training sample;
determine a classification loss of the text classification model based on the predicted classification result of the training sample;
determine a contrastive loss of the text classification model based on the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample; and
train the text classification model based on the classification loss and the contrastive loss.
12. The apparatus of claim 11 , wherein the processor circuitry is configured to:
determine a first contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and a semantic representation of a different sample, the different sample being a sample belonging to a different category from the training sample;
determine a second contrastive loss based on the semantic representation of the training sample, the semantic representation of the adversarial sample corresponding to the training sample, and a semantic representation of an adversarial sample corresponding to the different sample; and
determine the contrastive loss of the text classification model based on the first contrastive loss and the second contrastive loss.
13. The apparatus of claim 12 , wherein the processor circuitry is configured to:
calculate a first similarity, the first similarity being a similarity between the semantic representation of the adversarial sample corresponding to the training sample and the semantic representation of the training sample;
calculate a second similarity, the second similarity being a similarity between the semantic representation of the adversarial sample corresponding to the training sample and the semantic representation of the different sample; and
calculate the first contrastive loss based on the first similarity and the second similarity.
14. The apparatus of claim 12 , wherein the processor circuitry is configured to:
calculate a third similarity, the third similarity being a similarity between the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample;
calculate a fourth similarity, the fourth similarity being a similarity between the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the different sample; and
calculate the second contrastive loss based on the third similarity and the fourth similarity.
15. The apparatus of claim 14 , wherein the processor circuitry is further configured to:
select m samples from samples belonging to different categories from the training sample, to obtain the different samples, m being a positive integer.
16. The apparatus of claim 11 , wherein the processor circuitry is configured to:
determine a total loss of the text classification model based on the classification loss and the contrastive loss; and
adjust a parameter of the text classification model to minimize the total loss and obtain a trained text classification model.
17. The apparatus of claim 11 , wherein the processor circuitry is configured to:
obtain a word embedding of the training sample; and
add the perturbation information to the word embedding of the training sample to obtain a processed word embedding, the processed word embedding being used as the adversarial sample corresponding to the training sample.
18. The apparatus of claim 11 , wherein the processor circuitry is configured to:
add the perturbation information to a text content of the training sample to obtain processed text information, the processed text information being used as the adversarial sample corresponding to the training sample.
19. The apparatus of claim 11 , wherein the processor circuitry is further configured to:
create a first buffer and a second buffer, the first buffer being for storing the semantic representation of the training sample, and the second buffer being for storing the semantic representation of the adversarial sample corresponding to the training sample;
dynamically update data stored in the first buffer; and
dynamically update data stored in the second buffer.
20. A non-transitory machine-readable media, having instructions stored on the machine-readable media, the instructions configured to, when executed, cause a machine to:
obtain a training sample of a text classification model, the training sample being a text;
determine a semantic representation of the training sample using the text classification model, wherein the semantic representation of the training sample represents a semantic of the training sample;
determine a predicted classification result of the training sample based on the semantic representation;
generate an adversarial sample corresponding to the training sample based on the training sample and perturbation information for the training sample;
determine a semantic representation of the adversarial sample corresponding to the training sample using the text classification model, wherein the semantic representation of the adversarial sample represents a semantic of the adversarial sample, the semantic of the adversarial sample is consistent with the semantic of the training sample;
determine a classification loss of the text classification model based on the predicted classification result of the training sample;
determine a contrastive loss of the text classification model based on the semantic representation of the training sample and the semantic representation of the adversarial sample corresponding to the training sample; and
train the text classification model based on the classification loss and the contrastive loss.