Neural machine translation with latent tree attention
An attentional neural machine translation model is provided for the task of machine translation that, according to some embodiments, leverages the hierarchical structure of language to perform natural language processing without a priori annotation. Other features are also provided.
1. A machine translation system comprising:
a memory storing an encoder and an attention-based decoder, and a plurality of processor-executed instructions for operating the encoder and the attention-based decoder; and
a processor executing the plurality of processor-executable instructions to:
encode tokens of a source sequence in a first language over a plurality of encoding time steps, wherein each of the tokens of the source sequence corresponds to a character from the source sequence, wherein at each encoding time step the encoder:
pushes a non-terminal token to an encoder stack, the non-terminal token corresponding to a phrase type;
pushes a terminal token to the encoder stack;
pops tokens off the encoder stack to a last pushed non-terminal token,
generates a new token using a composition function applied to the popped tokens, and pushes the new token onto the encoder stack,
wherein non-terminal tokens are vectorized into a common embedding for different phrase types across encoding time steps without differentiating among the different phrase types; and
generate, by the attention-based decoder, non-terminal tokens based on attention to the tokens on the encoder stack, and outputs tokens of a target sequence in a second language based on the attention.
2. The system of claim 1 , wherein each of the tokens of the source sequence corresponds to a character from the source sequence.
3. The system of claim 1 , wherein the tokens of the source sequence are densely encoded using real-valued vectors.
4. The system of claim 1 , wherein the tokens of the source sequence are sparsely encoded using one-hot vectors.
5. The system of claim 1 , wherein a phrase tree structure of both the source and target sequences includes one or more character-based token constituents and a phrase type constituent.
6. The system of claim 5 , wherein a constant vector is used as the common embedding for different phrase types.
7. The system of claim 5 , wherein an encoder compositional embedding encodes one or more character-based token constituents, without encoding a phrase type constituent.
8. The system of claim 1 , further configured to:
use an ultimate encoder compositional embedding of the source sequence as a decoder embedding for an initially predicted phrase type of the target sequence.
9. The system of claim 1 , further configured to:
use policy gradient reinforcement learning to induce unsupervised phrase tree structures of both the source and target sequences.
10. The system of claim 1 , wherein the encoder and the attention-based decoder are long short-term memory (LSTM) networks.
11. The system of claim 10 , wherein each of the encoder and the attention-based decoder includes a bi-directional LSTM (Bi-LSTM) that calculates encoder and decoder compositional embeddings.
12. The system of claim 1 , wherein the encoder and the attention-based decoder are stack-only recurrent neural network grammar (s-RNNG) networks.
13. A method for machine translation comprising:
encoding tokens of a source sequence in a first language over a plurality of encoding time steps, wherein each of the tokens of the source sequence corresponds to a character from the source sequence;
at each encoding time step:
pushing a non-terminal token to an encoder stack, the non-terminal token corresponding to a phrase type;
pushing a terminal token to the encoder stack;
popping tokens off the encoder stack to a last pushed non-terminal token, generating a new token using a composition function applied to the popped tokens, and pushing the new token onto the encoder stack; and
generating non-terminal tokens based on attention to the tokens on the encoder stack, wherein non-terminal tokens are vectorized into a common embedding for different phrase types across encoding time steps without differentiating among the different phrase types; and
outputting tokens of a target sequence in a second language based on the attention.
14. The method of claim 13 , wherein each of the tokens of the source sequence corresponds to a character from the source sequence.
15. The method of claim 13 , comprising densely encoding the tokens of the source sequence using real-valued vectors.
16. The method of claim 13 , comprising sparsely encoding the tokens of the source sequence using one-hot vectors.
17. The method of claim 13 , wherein a phrase tree structure of both the source and target sequences includes one or more character-based token constituents and a phrase type constituent.
18. The method of claim 17 , wherein a constant vector is used as the common embedding for different phrase types.
19. The method of claim 17 , comprising encoding one or more character-based token constituents, without encoding a phrase type constituent.
20. The method of claim 13 , comprising using an ultimate encoder compositional embedding of the source sequence as a decoder embedding for an initially predicted phrase type of the target sequence.
21. The method of claim 13 , comprising using policy gradient reinforcement learning to induce unsupervised phrase tree structures of both the source and target sequences.