Method and system for detecting duplicate document using vector quantization
Disclosed is a method and system for detecting a duplicate document using vector quantization. A duplicate document detection method may include acquiring, by processing circuitry, a respective vector expression for each of a plurality of documents using a similarity model, the similarity model being trained to output similar vector expressions for semantically similar documents, generating a key by performing a vector quantization on the respective vector expression, the key including a binary character string, and detecting a duplicate document from among the plurality of documents using the key.
1. A duplicate document detection method of a computer apparatus including processing circuitry, the method comprising:
inputting, by the processing circuitry, a plurality of documents into a trained similarity model to obtain a respective vector expression for each of the plurality of documents, the trained similarity model being trained using a plurality of reference document pairs according to a training process, the plurality of reference document pairs including a similar document pair set and a dissimilar document pair set, the similar document pair set including a plurality of similar document pairs having a common attribute, the dissimilar document pair set including a plurality of dissimilar document pairs having different attributes or being randomly paired, and the training process including
calculating a mathematical similarity for each of the plurality of similar document pairs and each of the plurality of dissimilar document pairs using a mathematical measure to obtain a first plurality of mathematical similarities based on the plurality of similar document pairs and a second plurality of mathematical similarities based on the plurality of dissimilar document pairs,
calculating a respective semantic similarity of each of the plurality of similar document pairs by increasing the first plurality of mathematical similarities to obtain a first plurality of semantic similarities,
calculating a respective semantic similarity of each of the plurality of dissimilar document pairs by decreasing the second plurality of mathematical similarities to obtain a second plurality of semantic similarities, and
training a similarity model based on the plurality of similar document pairs, the plurality of dissimilar document pairs, the first plurality of semantic similarities and the second plurality of semantic similarities to obtain the trained similarity model;
generating a key by performing a vector quantization on the respective vector expression, the key including a binary character string; and
detecting a duplicate document from among the plurality of documents using the key.
2. The method of claim 1 , wherein the respective vector expression is an N dimensional real vector, N denoting a natural number of 2 or more.
3. The method of claim 1 , wherein the generating the key comprises generating the binary character string by:
replacing a value in the respective vector expression of 0 or more with 1, or
replacing a negative value in the respective vector expression with 0.
4. The method of claim 1 , wherein the detecting the duplicate document comprises detecting two among the plurality of documents associated with the key.
5. The method of claim 1 , further comprising:
training the similarity model using a loss function adjusted based on a weight, the weight corresponding to a difference between an output value of the similarity model and a calculated value.
6. The method of claim 5 , further comprising:
adjusting an average distance between a plurality of vector expressions by adjusting a value of the weight, the plurality of vector expressions including the respective vector expression.
7. The method of claim 1 , further comprising:
extracting, by the processing circuitry, the plurality of reference document pairs from a document database; and
training, by the processing circuitry, the similarity model according to the training process.
8. The method of claim 1 , wherein the common attribute comprises at least one of an author of a document, a post section of the document, or a registration time range of the document.
9. The method of claim 1 , wherein
the calculating the respective semantic similarity of each of the plurality of similar document pairs includes calculating the first plurality of semantic similarities by inputting the first plurality of mathematical similarities to a first nonlinear function;
the calculating a respective semantic similarity of each of the plurality of dissimilar document pairs includes calculating the second plurality of semantic similarities by inputting the second plurality of mathematical similarities to a second nonlinear function; and
the first nonlinear function outputs a value greater than a value output by the second nonlinear function based on any value input to both the first nonlinear function and the second nonlinear function.
10. A non-transitory computer-readable record medium storing instructions that, when executed by processing circuitry, cause the processing circuitry to perform the method of claim 1 .
11. The method of claim 1 , further comprising:
displaying a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) in response to the detecting; and
blocking registration of the duplicate document in response to the detecting.
12. The method of claim 5 , wherein the training includes inputting vector expressions for a candidate document pair and a calculated semantic similarity for the candidate document pair into the loss function.
13. The method of claim 1 , wherein the detecting the duplicate document comprises:
comparing the key to a plurality of other keys stored in association with the plurality of documents in a table; and
detecting a particular document associated with the respective vector expression to be the duplicate document based on determining the key to be the same as one of the plurality of other keys.
14. The method of claim 1 , wherein the plurality of dissimilar documents pairs are randomly paired.
15. The method of claim 9 , wherein the plurality of dissimilar documents pairs are randomly paired.
16. A computer apparatus comprising:
processing circuitry configured to cause the computer apparatus to
input a plurality of documents into a trained similarity model to obtain a respective vector expression for each of the plurality of documents, the trained similarity model being trained using a plurality of reference document pairs according to a training process, the plurality of reference document pairs including a similar document pair set and a dissimilar document pair set, the similar document pair set including a plurality of similar document pairs having a common attribute, the dissimilar document pair set including a plurality of dissimilar document pairs having different attributes or being randomly paired, and the training process including
calculating a mathematical similarity for each of the plurality of similar document pairs and each of the plurality of dissimilar document pairs using a mathematical measure to obtain a first plurality of mathematical similarities based on the plurality of similar document pairs and a second plurality of mathematical similarities based on the plurality of dissimilar document pairs,
calculating a respective semantic similarity of each of the plurality of similarity document pairs by increasing the first plurality of mathematical similarities to obtain a first plurality of semantic similarities,
calculating a respective semantic similarity of each of the plurality of dissimilar document pairs by decreasing the second plurality of mathematical similarities to obtain a second plurality of semantic similarities, and
training a similarity model based on the plurality of similar document pairs, the plurality of dissimilar document pairs, the first plurality of semantic similarities and the second plurality of semantic similarities to obtain the trained similarity model,
generate a key by performing a vector quantization on the respective vector expression, the key including a binary character string, and
detect a duplicate document from among the plurality of documents using the key.
17. The computer apparatus of claim 16 , wherein the respective vector expression is an N dimensional real vector, N denoting a natural number of 2 or more.
18. The computer apparatus of claim 16 , wherein the processing circuitry is configured to cause the computer apparatus to generate the binary character string by:
replacing a value in the respective vector expression of 0 or more with 1; or
replacing a negative value in the respective vector expression with 0.
19. The computer apparatus of claim 16 , wherein the processing circuitry is configured to cause the computer apparatus to detect the duplicate document by detecting two among the plurality of documents associated with the key.
20. The computer apparatus of claim 16 , wherein the processing circuitry is configured to cause the computer apparatus to:
train the similarity model using a loss function adjusted based on a weight, the weight corresponding to a difference between an output value of the similarity model and a calculated value.
21. The computer apparatus of claim 20 , wherein the processing circuitry is configured to cause the computer apparatus to:
adjust an average distance between a plurality of vector expressions by adjusting a value of the weight, the plurality of vector expressions including the respective vector expression.
22. The computer apparatus of claim 16 , wherein the processing circuitry is configured to cause the computer apparatus to:
display a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) in response to detecting the duplicate document; and
block registration of the duplicate document in response to detecting the duplicate document.
23. The computer apparatus of claim 20 , wherein the processing circuitry is configured to cause the computer apparatus to:
train the similarity model including inputting vector expressions for a candidate document pair and a calculated semantic similarity for the candidate document pair into the loss function.