Modeling ambiguity in neural machine translation
The technology addresses ambiguity in neural machine translation. An encoder module receives a given text exemplar and generates an encoded representation of it. A decoder module receives the encoded representation and a set of translation prefixes. The decoder module outputs an unbounded function corresponding to a set of tokens associated with each pair of the given text exemplar and translation prefix from the set of translation prefixes. Each token is assigned a probability between 0 and 1 in a vocabulary of the exemplar at each time step. A logits module generates, based on the unbounded function, a corresponding bounded conditional probability for each token, wherein the probabilities are not normalized over the vocabulary at each time step. A loss function module having a positive loss component and a scaled negative loss component identifies whether each target text of a set of target texts is a valid translation of the exemplar.
1. A system configured for a machine translation model, the system comprising:
memory configured to store a set of text exemplars in a source language and a set of rewritten texts in one or more languages different from the source language; and
one or more processing elements operatively coupled to the memory, the one or more processing elements implementing the machine translation model as a neural network having:
an encoder module comprising an encoder neural network configured to receive a given text exemplar (x=<x 1 , . . . x |x| >) and to generate an encoded representation of the given text exemplar:
a decoder module comprising a decoder neural network configured to receive the encoded representation and a set of translation prefixes (y <i =<y 1 , . . . , y 1-1 >) and to output an unbounded function ƒ(x, y <a ) corresponding to a set of tokens associated with each pair of the given text exemplar x and translation prefix from the set of translation prefixes ye, wherein each token is assigned a probability between 0 and 1 in a vocabulary of the given text exemplar at each time step:
a logits module configured to act on the unbounded function to generate a corresponding bounded conditional probability for each token, wherein the conditional probabilities are not normalized over the vocabulary at each time step; and
a loss function module having a positive loss component and a scaled negative loss component, in which the loss function module is configured to identify whether each target text of a set of target texts is a valid translation of the given text exemplar.
2. The system of claim 1 , wherein the set of text exemplars comprises a set of input sentences, and the system is configured to learn binary classifiers for each sentence pair (x,y) that indicate whether or not y is a valid translation of x.
3. The system of claim 2 , wherein intrinsic uncertainty is represented by setting the conditional probabilities of at least two correct translations y 1 and y 2 to a maximum probability simultaneously.
4. The system of claim 2 , wherein the logits module is configured to generate probabilities for each translation using separate binary classifiers.
5. The system of claim 1 , wherein a probability of a complete translation of the given text exemplar is decomposed into a product of token-level probabilities.
6. The system of claim 1 , wherein the logits module is configured to apply sigmoid activations to the unbounded function at each time step.
7. The system of claim 1 , wherein the positive loss component applies a log function to the bounded conditional probability for each reference token, and the scaled negative loss component applies a log function to the bounded conditional probability for each non-reference token.
8. The system of claim 1 , wherein during inference the system is configured to search for a translation that has a highest probability of being a translation of a text segment to be translated.
9. The system of claim 1 , wherein the system is configured to express intrinsic uncertainty associated with the machine translation model.
10. The system of claim 1 , wherein the loss function module is configured to adjust scaling of the negative loss component to maximize translation performance.
11. The system of claim 1 , wherein the encoder module and the decoder module comprise a self-attention neural network encoder-decoder architecture.
12. The system of claim 1 , wherein the encoder module and the decoder module comprise a sequence to sequence model architecture.
13. A machine translation method employing a neural network, the method comprising:
storing, in a memory, a set of text exemplars:
receiving, by an encoder module comprising an encoder neural network, a given text exemplar (x=<x 1 , . . . , x x| >):
generating, by the encoder module, an encoded representation of the given text exemplar:
receiving, by a decoder module, the encoded representation and a set of translation prefixes (y <i =<y 1 , . . . , y i-1 >):
outputting, by the decoder module, an unbounded function ƒ(x, y <i ) corresponding to a set of tokens associated with each pair of the given text exemplar x and translation prefix from the set of translation prefixes y <i , wherein each token is assigned a probability between 0 and 1 in a vocabulary of the given text exemplar at each time step:
generating, by a logits module based on the unbounded function, a corresponding bounded conditional probability for each token, wherein the conditional probabilities are not normalized over the vocabulary at each time step; and
identifying, by a loss function module having a positive loss component and a scaled negative loss component, whether each target text of a set of target texts is a valid translation of the given text exemplar.
14. The method of claim 13 , wherein the set of text exemplars comprises a set of input sentences, and the method includes learning binary classifiers for each sentence pair (x,y) that indicate whether or not y is a valid translation of x.
15. The method of claim 14 , wherein intrinsic uncertainty is represented by setting the conditional probabilities of at least two correct translations y 1 and y 2 to a maximum probability simultaneously.
16. The method of claim 14 , wherein generating the corresponding bounded conditional probability for each token comprises generating probabilities for each translation using separate binary classifiers.
17. The method of claim 13 , wherein a probability of a complete translation of the given text exemplar is decomposed into a product of token-level probabilities.
18. The method of claim 13 , wherein generating the corresponding bounded conditional probability for each token comprises applying sigmoid activations to the unbounded function at each time step.
19. The method of claim 13 , wherein the positive loss component applies a log function to the bounded conditional probability for each token, and the scaled negative loss component applies a log function to the bounded conditional probability for each token.
20. The method of claim 13 , further comprising, during inference searching for a translation that has a highest probability of being a translation of a text segment to be translated.
21. The method of claim 13 , further comprising adjusting scaling of the negative loss component to maximize translation performance.