Apparatus and method of training machine learning model, and apparatus and method for summarizing document using the same
An apparatus of training a machine learning model includes a preprocessing module segmenting a document into each sentence and performing tokenization to generate a token sequence for the document, wherein a document representative token representing the document and representative sentence tokens representing each sentence are included in the token sequence for the document, a first training module training the machine learning model to predict an order of sentences in the document, based on the token sequence for the document, and a second training module training the machine learning model to perform document similarity maximization based on the token sequence for the document.
1 . An apparatus of training a machine learning model, the apparatus comprising:
a processor;
a memory storing one or more programs, wherein the processor is configured to execute the one or more programs to perform preprocessing, first training, and second training,
wherein the preprocessing comprises segmenting a document into sentences and perform tokenization to generate a token sequence for the document, the token sequence including a document representative token representing the document and representative sentence tokens representing the sentences, respectively;
wherein the first training comprises training the machine learning model to predict an order of the sentences in the document, based on the token sequence for the document; and
wherein the second training comprises training the machine learning model to perform document similarity maximization in which a similarity between two documents that differ only in the order of sentences is maximized, the two documents being generated by rearranging the order of the token sequence for the document.
2 . The apparatus of claim 1 , wherein the preprocessing comprises tokenizing the sentences to generate sentence token sequences; and
the first training comprises rearranging an order of the sentence token sequences and inputting the rearranged sentence token sequences to the machine learning model.
3 . The apparatus of claim 2 , wherein a representative sentence token for a corresponding sentence is located at the front of a sentence token sequence, and the machine learning model embeds the representative sentence tokens in the rearranged sentence token sequences to generate embedding vectors, respectively, and predicts the order of the sentences in the document, based on the generated embedding vectors.
4 . The apparatus of claim 3 , wherein the first training comprises calculating a first error by comparing a predicted sentence order output from the machine learning model with an original sentence order of the document and adjusting a weight of the machine learning model to minimize the first error.
5 . The apparatus of claim 1 , wherein the second training comprises inputting two documents that differ in the order of sentences to the machine learning model, and training the machine learning model so that a difference between the two documents is minimized.
6 . The apparatus of claim 5 , wherein the second training comprises performing first rearranging of the order of the sentence token sequences in the document according to a first order, and inputting the first rearranged sentence token sequences according to the first order and a first document representative token representing the first rearranged document to the machine learning model.
7 . The apparatus of claim 6 , wherein the second training comprises:
performing second rearranging of the sentence token sequences in the document according to a second order so that the order of the second rearranged sentence token sequences is different from the order of the first rearranged sentence token sequences; and
input the second rearranged sentence token sequences and a second document representative token representing the second rearranged document to the machine learning model.
8 . The apparatus of claim 7 , wherein the machine learning model is configured to embed the first document representative token to generate a first embedding vector, and embed the second document representative token to generate a second embedding vector.
9 . The apparatus of claim 8 , wherein the second training comprises calculating a second error through a difference between the first embedding vector and the second embedding vector, and adjusting a weight of the machine learning model to minimize the second error.
10 . The apparatus of claim 1 , wherein a loss function (Loss) of the machine learning model is expressed by the following equation:
Loss=Loss SOP +α·Loss DSM [Equation]
Loss SOP is a loss function for sentence order prediction;
Loss DSM is a loss function for document similarity maximization; and
α is a normalization parameter.
11 . A method of training a machine learning model, the method performed by an apparatus comprising a processor, and a memory storing one or more programs configured to be executed by the processor, the method comprising:
segmenting a document into sentences and performing tokenization to generate a token sequence for the document, the token sequence including a document representative token representing the document and representative sentence tokens representing the sentences, respectively;
training the machine learning model to predict an order of sentences in the document, based on the token sequence for the document; and
training the machine learning model to perform document similarity maximization in which a similarity between two documents that differ only in the order of sentences is maximized, the two documents being generated by rearranging the order of the token sequence for the document.
12 . The method of claim 11 , further comprising: tokenizing the sentences to generate sentence token sequences,
wherein the training of the machine learning model to predict the order of sentences includes:
rearranging the order of the sentence token sequences in the document and inputting the rearranged sentence token sequences into the machine learning model;
calculating a first error by comparing a predicted sentence order output from the machine learning model with an original sentence order of the document; and
adjusting a weight of the machine learning model to minimize the first error.
13 . The method of claim 12 , wherein representative sentence token for a corresponding sentence is located at the front of the sentence token sequence, and the machine learning model embeds the representative sentence tokens in the rearranged sentence token sequences to generate embedding vectors, respectively, and predicts the order of the sentences in the document, based on the generated embedding vectors.
14 . The method of claim 11 , wherein the training of the machine learning model to perform document similarity maximization includes:
inputting two documents that differ in the order of sentences to the machine learning model; and
training the machine learning model so that a difference between the two documents is minimized.
15 . The method of claim 14 , wherein the training of the machine learning model to perform document similarity maximization includes:
performing a first rearranging of the order of the sentence token sequences in the document according to a first order;
inputting the first rearranged sentence token sequences and a first document representative token representing the first rearranged document to the machine learning model;
performing a second rearranging of the sentence token sequences in the document according to a second order so that the order of the second rearranged sentence token sequences is different from the order of the first rearranged sentence token sequences; and
inputting the second rearranged sentence token sequences and a second document representative token representing the second rearranged document to the machine learning model.
16 . The method of claim 15 , wherein the machine learning model embeds the first document representative token to generate a first embedding vector, and embeds the second document representative token to generate a second embedding vector.
17 . The method of claim 16 , wherein the training of the machine learning model to perform document similarity maximization includes:
calculating a second error through a difference between the first embedding vector and the second embedding vector; and
adjusting a weight of the machine learning model to minimize the second error.
18 . The method of claim 11 , wherein a loss function (Loss) of the machine learning model is expressed by the following equation:
Loss=Loss SOP +α·Loss DSM [Equation]
Loss SOP is a loss function for sentence order prediction;
Loss DSM is a loss function for document similarity maximization; and
α is a normalization parameter.
19 . An apparatus for summarizing a document using a machine learning model that is trained by the method of claim 11 , the apparatus comprising:
a processor;
a memory storing one or more programs configured to be executed by the processor to implement processing, the machine learning model, and summary extracting,
wherein the preprocessing comprises segmenting a document into sentences and perform tokenization to generate a token sequence for the document, the token sequence including a document representative token representing the document and representative sentence tokens representing the sentences, respectively;
wherein the machine learning model is configured to receive the token sequence for the document and embed the document representative token and the representative sentence tokens to output a document representative embedding vector and representative sentence embedding vectors; and
wherein the summary extracting comprises calculating a similarity between the document representative embedding vector and each representative sentence embedding vector and summarize the document according to the calculated similarity.
20 . The apparatus of claim 19 , wherein the preprocessing comprises locating the document representative token at a front of the document, and locate each representative sentence token at a front of a corresponding sentence.
21 . The apparatus of claim 20 , wherein the machine learning model is configured to be trained to predict an order of the sentences in the document based on the token sequence for the document, and to be trained to perform document similarity maximization, in which a similarity between two documents that differ only in the order of sentences is maximized, the two documents being generated by rearranging the order of the token sequence for the document.
22 . The apparatus of claim 21 , wherein the summary extracting comprises summarizing the document by extracting a sentence having the similarity greater than or equal to a preset threshold value.
23 . A method for summarizing a document using a machine learning model that is trained by the method of claim 11 , the method performed by an apparatus comprising a processor, and a memory storing one or more programs configured to be executed by the processor, the method comprising:
segmenting a document into sentences and performing tokenization to generate a token sequence for the document, the token sentence including a document representative token representing the document and representative sentence tokens representing the sentences, respectively;
receiving, using the machine learning model, the token sequence for the document and embedding the document representative token and the representative sentence tokens to output a document representative embedding vector and representative sentence embedding vectors; and
calculating a similarity between the document representative embedding vector and each representative sentence embedding vector and summarizing the document according to the calculated similarity.