Pointer sentinel mixture architecture
The technology disclosed provides a so-called “pointer sentinel mixture architecture” for neural network sequence models that has the ability to either reproduce a token from a recent context or produce a token from a predefined vocabulary. In one implementation, a pointer sentinel-LSTM architecture achieves state of the art language modeling performance of 70.9 perplexity on the Penn Treebank dataset, while using far fewer parameters than a standard softmax LSTM.
1. A system for predicting an output token given a context text and a vocabulary text for a natural language processing (NLP) task, the system comprising:
a network interface receiving the vocabulary text and the context text;
a memory storing a plurality of processor-executable instructions for operating a pointer sentinel mixture architecture;
one or more processors reading and executing the plurality of processor-executable instructions to perform operations comprising:
generating, via a pointer network of the pointer sentinel mixture architecture, a pointer distribution, the pointer distribution corresponding to a distribution of attention probability masses over a window of tokens in the context text;
generating, via a vocabulary network of the pointer sentinel mixture architecture, a vocabulary distribution, the vocabulary distribution corresponding to a distribution of vocabulary probability masses over tokens in the vocabulary text;
generating, by a recurrent neural network (RNN), a final output state vector for each position in the window of tokens in the context text;
computing a gate probability mass based on a trained sentinel gate vector and a query vector formulated from the final output state vector;
mixing, at a pointer vocabulary mixer of the pointer sentinel mixer architecture, the pointer distribution and the vocabulary distribution, as governed by a gate probability mass; and
outputting the output token according to a mixed distribution in response to the NLP task.
2. The system of claim 1 , wherein the pointer distribution is generated using an unnormalized gate value.
3. The system of claim 2 , wherein the gate probability mass results from exponentially normalizing the unnormalized gate value.
4. The system of claim 1 , wherein a sum of a vector of the attention probability masses and the gate probability mass being a predetermined constant.
5. The system of claim 1 , wherein a sum of the distribution of vocabulary probability masses over the tokens in the vocabulary text being a predetermined constant.
6. The system of claim 1 , wherein the trained sentinel gate vector controls accumulation of information from the vocabulary network and the pointer network.
7. The system of claim 6 , wherein the gate probability mass being unity results in accumulation of information from the vocabulary network.
8. The system of claim 6 , wherein the gate probability mass being zero results in accumulation of information from the pointer network.
9. The system of claim 1 , wherein the query vector is formulated from the final output state vector of the RNN by processing the final output state vector through a linear layer to generate a non-linear projection.
10. The system of claim 1 , wherein the trained sentinel gate vector is trained a priori.
11. The system of claim 10 , wherein a dynamic sentinel gate vector is formulated from a final output state vector of the RNN by concatenating the final output state vector with the trained sentinel gate vector and processing the concatenation through a linear layer to generate a non-linear projection.
12. The system of claim 1 , wherein the output token comprises one of a word, a character, and a phrase.
13. A method for predicting an output token given a context text and a vocabulary text for a natural language processing (NLP) task, the method comprising:
receiving, via a network interface, the vocabulary text and the context text;
generating, via a pointer network of the pointer sentinel mixture architecture implemented on one or more hardware processors, a pointer distribution, the pointer distribution corresponding to a distribution of attention probability masses over a window of tokens in the context text;
generating via a vocabulary network of the pointer sentinel mixture architecture, a vocabulary distribution, the vocabulary distribution corresponding to a distribution of vocabulary probability masses over tokens in the vocabulary text;
generating, by a recurrent neural network (RNN), a final output state vector for each position in the window of tokens in the context text;
computing a gate probability mass based on a trained sentinel gate vector and a query vector formulated from the final output state vector;
mixing the pointer distribution and the vocabulary distribution, as governed by a gate probability mass; and
outputting the output token according to a mixed distribution in response to the NLP task.
14. The method of claim 13 , wherein the pointer distribution is generated using an unnormalized gate value.
15. The method of claim 14 , wherein the gate probability mass results from exponentially normalizing the unnormalized gate value.
16. The method of claim 13 , wherein a sum of a vector of the attention probability masses and the gate probability mass being a predetermined constant.
17. The method of claim 13 , wherein a sum of the distribution of vocabulary probability masses over the tokens in the vocabulary text being a predetermined constant.
18. The method of claim 13 , wherein the trained sentinel gate vector controls accumulation of information from the vocabulary network and the pointer network.
19. The method of claim 18 , wherein the gate probability mass being unity results in accumulation of information from the vocabulary network.
20. The method of claim 18 , wherein the gate probability mass being zero results in accumulation of information from the pointer network.
21. The method of claim 13 , wherein the query vector is formulated from the final output state vector of the RNN by processing the final output state vector through a linear layer to generate a non-linear projection.
22. The method of claim 13 , wherein the trained sentinel gate vector is trained a priori.
23. The method of claim 22 , wherein a dynamic sentinel gate vector is formulated from the final output state vector of the RNN by concatenating the final output state vector with the trained sentinel gate vector and processing the concatenation through a linear layer to generate a non-linear projection.
24. The method of claim 13 , wherein the output token comprises one of a word, a character, and a phrase.