Apparatus and method for processing natural language
An apparatus for processing natural language includes a collection module for collecting a document having style information on text in the document, a preprocessing module for extracting style information from the text of the collected document, and labeling the extracted style information with the text, and a first machine learning module receiving the text labeled with the style information, and predicting a position of a word having the style information in the received text.
1. An apparatus for processing natural language, the apparatus comprising:
a processor;
a memory storing one or more programs configured to be executed by the processor; and
the one or more programs including instructions for:
a collection module configured for collecting a document having style information on text in the document, the style information indicating a style that has been applied to the text in the document;
a preprocessing module configured for extracting the style information from the text of the collected document, and labeling the text with a position of a portion of the text to which the extracted style information has been applied; and
a first machine learning module configured for receiving the text labeled with the position of the portion of the text to which the extracted style information has been applied, and being trained, by using the labeled text as learning data, to predict a position of a word having the style information in the received text.
2. The apparatus of claim 1 , wherein the style information comprises at least one of a text color, a size, a shape, and a font that has been applied to the text.
3. The apparatus of claim 1 , wherein the preprocessing module is configured to perform the labeling based on a start position and an end position of the portion of the text to which the extracted style information has been applied.
4. The apparatus of claim 3 , wherein the first machine learning module comprises:
a first tokenization unit configured for generating first tokens of a preset unit by performing tokenization on the received text;
a first encoding unit configured for converting each of the first tokens into a first embedding vector;
a first predicting unit configured for receiving the first embedding vector for each of the first tokens and predicting a probability of reaching the start position of the portion of the text to which the style information has been applied from the first embedding vector; and
a second predicting unit configured for receiving the first embedding vector for each of the first tokens and predicting a probability of reaching the end position of the portion of the text to which the style information has been applied from the first embedding vector.
5. The apparatus of claim 4 , wherein the first machine learning module is configured to be trained so that a difference between a value predicted by the first predicting unit and a correct value for the start position of the portion of the text to which the style information has been applied is minimized, and a difference between a value predicted by the second predicting unit and a correct value for the end position of the portion of the text to which the style information has been applied is minimized.
6. The apparatus of claim 5 , wherein a loss function of an artificial neural network constituting the first machine learning module is expressed by the following equation by applying Cross Entropy Loss:
Loss=0.5*CrossEntrophy(Start′,Start)+0.5*CrossEntropy(End′,End) [Equation]
where Start′ is the value predicted by the first predicting unit;
Start is the correct value for the start position of the portion of the text to which the style information has been applied;
End′ is the value predicted by the second predicting unit; and
End is the correct value for the end position of the portion of the text to which the style information has been applied.
7. The apparatus of claim 5 , wherein the first machine learning module further comprises:
a filter unit configured for filtering first embedding vectors based on the value predicted by the first predicting unit and the value predicted by the second predicting unit.
8. The apparatus of claim 7 , wherein the filter unit extracts an embedding vector, as a style-related embedding vector, equal to or greater than a preset threshold value, among the first embedding vectors, based on the value predicted by the first predicting unit and the value predicted by the second predicting unit.
9. The apparatus of claim 8 , further comprising:
a second machine learning module configured for summarizing the received text by using the style-related embedding vector.
10. The apparatus of claim 9 , wherein the second machine learning module comprises:
a second tokenization unit configured for generating second tokens of a preset unit by performing tokenization on the received text;
a second encoding unit configured for converting each of the second tokens into a second embedding vector;
a summing unit configured for adding the style-related embedding vector to a second embedding vector corresponding thereto among the second vectors output from the second encoding unit and outputting the second embedding vector to which the style-related embedding vector is added; and
a decoding unit configured for outputting a summary of the text based on the outputted second embedding vector to which the style-related embedding vector is added.
11. The apparatus of claim 10 , wherein the summing unit is configured to:
transmit the second embedding vector, not corresponding to the style-related vector, among second embedding vectors output from the second encoding unit, to the decoding unit;
add the second embedding vector, corresponding to the style-related vector, among the second embedding vectors output from the second encoding unit with the style-related embedding vector; and
then transmit the second embedding vector, to which the style-related embedding vector is added, to the decoding unit.
12. The apparatus of claim 11 , wherein the second encoding unit and the decoding unit of the second machine learning module are initialized to a weight value of the learned first encoding unit of the first machine learning module.
13. An apparatus for processing natural language, the apparatus comprising:
a processor;
a memory storing one or more programs configured to be executed by the processor; and
the one or more programs including instructions for:
a collection module configured for collecting a document having style information on text in the document, the style information indicating a style that has been applied to the text in the document;
a preprocessing module for extracting the style information from the text of the collected document, and labeling the text with a position of a portion of the text to which the extracted style information has been applied;
a first machine learning module configured receiving the text labeled with the position of the portion of the text to which the extracted style information has been applied, and being trained, by using the labeled text as learning data, to predict a position of a word having the style information in the received text; and
a second machine learning module configured for summarizing the received text in the document based on a result predicted by the first machine learning module.
14. A method for processing natural language, the method performed by an apparatus comprising a collection module, a preprocessing module and a first machine learning module, the method comprising:
in the collection module, an operation of collecting a document having style information on text in the document, the style information indicating a style that has been applied to the text in the document;
in the preprocessing module, an operation of extracting the style information from the text of the collected document, and labeling the text with a position of a portion of the text to which the extracted style information has been applied; and
in the first machine learning module, an operation of receiving the text labeled with the position of the portion of the text to which the extracted style information has been applied, and being trained, by using the labeled text as learning data, to predict a position of a word having the style information in the received text.
15. The method of claim 14 , wherein the style information comprises at least one of a text color, a size, a shape, and a font that has been applied to the text.
16. The method of claim 14 , wherein the labeling comprises labeling the extracted style information with the text, based on a start position and an end position of a portion of the text to which the style information has been applied.
17. The method of claim 16 , wherein the first machine learning module comprises a first tokenization unit, a first encoding unit, a first predicting unit and a second predicting unit; and
the operation of being trained comprises
an operation, in the first tokenization unit, of generating first tokens of a preset unit by performing tokenization on the received text;
an operation, in the first encoding unit, of converting each of the first tokens into a first embedding vector;
an operation, in the first predicting unit, of receiving the first embedding vector for each of the first tokens and predicting a probability of reaching the start position of the portion of the text to which the style information has been applied the first embedding vector; and
an operation, in the second predicting unit, of receiving the first embedding vector for each of the first tokens and predicting a probability of reaching the end position of the portion of the text to which the style information has been applied from the first embedding vector.
18. The method of claim 17 , wherein the operation of being trained further comprises an operation, in the first machine learning module, of being trained so that a difference between a value predicted by the first predicting unit and a correct value for the start position of the portion of the text to which the style information has been applied is minimized and a difference between a value predicted by the second predicting unit and a correct value for the end position of the portion of the text to which the style information has been applied is minimized.
19. The method of claim 18 , further comprising:
an operation, in a filtering unit, of filtering first embedding vectors based on the value predicted by the first predicting unit and the value predicted by the second predicting unit.
20. The method of claim 19 , wherein the operation of filtering comprises an operation, in the filtering unit, of extracting an embedding vector, as a style-related embedding vector, equal to or greater than a preset threshold value, among the embedding vectors.
21. The method of claim 20 , further comprising:
an operation, in a second machine learning module, of summarizing the received text by using the style-related embedding vector.
22. The method of claim 21 , wherein the second machine learning module comprises a second tokenization unit, a second encoding unit, a summing unit, and a decoding unit; and
the operation of summarizing the text, comprises:
an operation of, in the second tokenization unit, generating second tokens of a preset unit by performing tokenization on the received text;
an operation of, in the second encoding unit, converting each of the second tokens into a second embedding vector;
an operation of, in the summing unit, adding the style-related embedding vector to the second embedding vector corresponding thereto, among second embedding vectors output from the second encoding unit and outputting the second embedding vector to which the style-related embedding vector is added; and
an operation of, in the decoding unit, outputting a summary of the text based on the outputted second embedding vector to which the style-related embedding vector is added.
23. The method of claim 22 , further comprising:
an operation of, in the summing unit, transmitting the embedding vector, not corresponding to the style-related vector among the second embedding vectors output from the second encoding unit, to the decoding unit; and
an operation of, in the summing unit, adding the second embedding vector, corresponding to the style-related vector, among the second embedding vectors output from the second encoding unit, with the style-related embedding vector, and then transmitting the second embedding vector, to which the style-related embedding vector is added, to the decoding unit.