Apparatus and method for decoding using joint tokenization and translation
View Patent ↗Disclosed are a joint decoding apparatus and a joint decoding method that joins a tokenization process and a translation process. More particularly, the present disclosure can generate all available candidate tokens, reduce translation errors, and obtain an optimal translation result by jointly conducting a decoding by simultaneously conducting the tokenization process and the translation process for an input character sequence while decoding.
1. A joint decoding apparatus that joins a tokenization process and a translation process, the joint decoding apparatus comprising:
a candidate token generation unit configured to generate a plurality of candidate tokens by applying a maximum entropy model to an input character sequence;
a probability calculation unit configured to calculate a tokenization probability of each of the generated candidate tokens by using a language model;
an out of vocabulary (OOV) processing unit configured to process an OOV by using a word count and discount information for the OOV among the generated candidate tokens; and
a translation unit configured to generate a corresponding phrase corresponding to the input character sequence according to a translation model by using calculated probability values of the candidate tokens and an OOV processing result.
2. The joint decoding apparatus of claim 1 , further comprising:
a translation model database configured to store a translation model trained from tokenized data of a parallel corpus; and
a language model database configured to store a language model trained from a monolingual corpus.
3. The joint decoding apparatus of claim 2 , wherein the translation model database stores a character string-based translation model.
4. The joint decoding apparatus of claim 1 , wherein the candidate token generation unit tags at least one of a character corresponding to a beginning of a word, a character located in a middle position of the word, a character corresponding to an end of the word in the input character sequence, and a word consisting of a single character in terms of a word generation.
5. The joint decoding apparatus of claim 1 , wherein the probability calculation unit calculates a probability by combining the tokenization probability of each of the generated candidate tokens and an N-gram language model.
6. The joint decoding apparatus of claim 1 , wherein the OOV processing unit controls a word count for the OOV among the generated candidate tokens.
7. The joint decoding apparatus of claim 1 , wherein the translation unit generates a translated sentence corresponding to the input character sequence by applying the calculated probability values of the candidate tokens and the OOV processing result to a log-linear model.
8. A joint decoding method of joining a tokenization process and a translation process, the joint decoding method comprising:
generating a candidate token by generating a plurality of candidate tokens by applying a maximum entropy model to an input character sequence;
calculating a probability by combining a tokenization probability of each of the generated candidate tokens and a language model;
processing out of vocabularly (OOV) by using a word count and discount information for an OOV among the generated candidate tokens; and
conducting a translation that generates a corresponding phrase corresponding to the input character sequence by using calculated probability values of the candidate tokens and an OOV processing result.
9. The joint decoding method of claim 8 , wherein the translation model is a character string based translation model.
10. The joint decoding method of claim 8 , wherein the generating of the candidate token comprises tagging at least one of a character corresponding to a beginning of a word, a character located in a middle position of the word, a character corresponding to an end of the word in the input character sequence, and a word consisting of a single character in terms of a word generation.
11. The joint decoding method of claim 8 , wherein the calculating of the probability calculates a probability by combining the tokenization probability of each of the generated candidate tokens and an N-gram language model.
12. The joint decoding method of claim 8 , wherein the processing of the OOV comprises controlling a word count for the OOV among the generated candidate tokens.
13. The joint decoding method of claim 8 , wherein the conducting of the translation generates a translated sentence corresponding to the input character sequence by applying the calculated probability values of the candidate tokens and the 00 V processing result to a log-linear model.
14. A non-transitory computer readable recording medium for recording a program for executing the method according to claims 8 .
15. A non-transitory computer readable recording medium for recording a program for executing the method according to claim 9 .
16. A non-transitory computer readable recording medium for recording a program for executing the method according to claim 10 .
17. A non-transitory computer readable recording medium for recording a program for executing the method according to claim 11 .
18. A non-transitory computer readable recording medium for recording a program for executing the method according to claim 12 .
19. A non-transitory computer readable recording medium for recording a program for executing the method according to claim 13 .