Pointer sentinel mixture architecture
The technology disclosed provides a so-called “pointer sentinel mixture architecture” for neural network sequence models that has the ability to either reproduce a token from a recent context or produce a token from a predefined vocabulary. In one implementation, a pointer sentinel-LSTM architecture achieves state of the art language modeling performance of 70.9 perplexity on the Penn Treebank dataset, while using far fewer parameters than a standard softmax LSTM.
1. A method of predicting a token given context text and vocabulary text, including circumstances in which the token is in the vocabulary text and not in the context text, the method including:
mixing results of a pointer network and a vocabulary network to predict the token, with the mixing governed by a gate probability mass;
calculating the results of the pointer network over a window of tokens in the context text and the gate probability mass by
using
a trained sentinel gate vector,
output state vectors of a recurrent neural network (RNN) for each position in the window of tokens,
a query vector formulated from a final output state vector of the RNN,
an unnormalized gate value calculated as a distance measure product between the query vector and the trained sentinel gate vector, and
unnormalized attention values for each of the positions in the window of tokens, each calculated as a distance measure product between the query vector and each of the output state vectors,
concatenating the unnormalized attention values and the unnormalized gate value, and
exponentially normalizing the concatenated attention and gate values to produce a vector of attention probability masses and the gate probability mass;
calculating the results of the vocabulary network by computing, from the final output state vector of the RNN, a distribution of vocabulary probability masses over tokens in the vocabulary text;
producing a distribution of output probability masses over the tokens in the context text and the vocabulary text, including:
if a token is in the context text and not in the vocabulary text, then producing an output probability mass for the token as a corresponding attention probability mass,
if a token is in the vocabulary text is and not in the context text, then producing an output probability mass for the token by multiplying the gate probability mass with a corresponding vocabulary probability mass, and
if a token is in both the context text and the vocabulary text, then producing an output probability mass for the token by multiplying the gate probability mass with a corresponding vocabulary probability mass and adding a corresponding attention probability mass; and
selecting a token with a maximum output probability mass as a response to the query vector.
2. The method of claim 1 , wherein the gate probability mass results from exponentially normalizing the unnormalized gate value.
3. The method of claim 1 , wherein a sum of the vector of the attention probability masses and the gate probability mass being a predetermined constant.
4. The method of claim 1 , wherein a sum of the distribution of the vocabulary probability masses over the tokens in the vocabulary text being a predetermined constant.
5. The method of claim 1 , wherein a sum of the distribution of the output probability masses over the tokens in the context text and the vocabulary text being a predetermined constant.
6. The method of claim 1 , wherein a token appears at multiple positions in the context text, further including:
adding attention probability masses corresponding to each of the positions of the token in the context text to calculate a final attention probability mass for the token; and
using the final attention probability mass to calculate an output probability mass for the token.
7. The method of claim 1 , wherein the trained sentinel gate vector controls accumulation of information from the vocabulary network and the pointer network.
8. The method of claim 7 , wherein the gate probability mass being unity results in accumulation of information from the vocabulary network.
9. The method of claim 7 , wherein the gate probability mass being zero results in accumulation of information from the pointer network.
10. The method of claim 1 , wherein the query vector is formulated from the final output state vector of the RNN by processing the final output state vector through a linear layer to generate a non-linear projection.
11. The method of claim 1 , wherein the trained sentinel gate vector is trained a priori.
12. The method of claim 11 , wherein a dynamic sentinel gate vector is formulated from the final output state vector of the RNN by concatenating the final output state vector with the trained sentinel gate vector and processing the concatenation through a linear layer to generate a non-linear projection.
13. The method of claim 1 , wherein the token is a word.
14. The method of claim 1 , wherein the token is a character.
15. The method of claim 1 , wherein the token is a phrase.
16. A method of predicting a token given context text and vocabulary text, the method including:
mixing results of a pointer network and a vocabulary network to predict the token, with the mixing governed by a gate probability mass;
calculating the results of the pointer network over a window of tokens in the context text and the gate probability mass by
using
a trained sentinel gate vector,
output state vectors of a recurrent neural network (RNN) for each position in the window of tokens,
a query vector formulated from a final output state vector of the RNN,
an unnormalized gate value calculated as a distance measure product between the query vector and the trained sentinel gate vector, and
unnormalized attention values for each of the positions in the window of tokens, each calculated as a distance measure product between the query vector and each of the output state vectors,
concatenating the unnormalized attention values and the unnormalized gate value, and
exponentially normalizing the concatenated attention and gate values to produce a vector of attention probability masses and the gate probability mass;
calculating the results of the vocabulary network by computing, from the final output state vector of the RNN, a distribution of vocabulary probability masses over tokens in the vocabulary text;
producing a distribution of output probability masses over the tokens in the context text and the vocabulary text; and
selecting a token with a maximum output probability mass as a response to the query vector.
17. The method of claim 16 , further including repeatedly applying the method to context texts that present instances in which:
if a selected token that completes the context text is in the context text and not in the vocabulary text, then producing an output probability mass for the token as a corresponding attention probability mass,
if a token is in the vocabulary text is and not in the context text, then producing an output probability mass for the token by multiplying the gate probability mass with a corresponding vocabulary probability mass, and
if a token is in both the context text and the vocabulary text, then producing an output probability mass for the token by multiplying the gate probability mass with a corresponding vocabulary probability mass and adding a corresponding attention probability mass.
18. A method of predicting a token given context text and vocabulary text, the method including:
mixing results of a pointer network and a vocabulary network to predict the token, with the mixing governed by a gate probability mass;
calculating the results of the pointer network over a window of tokens in the context text and the gate probability mass by
using
a trained sentinel gate vector,
output state vectors of a recurrent neural network (RNN) for each position in the window of tokens,
a query vector formulated from a final output state vector of the RNN,
an unnormalized gate value calculated as a distance measure product between the query vector and the trained sentinel gate vector, and
unnormalized attention values for each of the positions in the window of tokens, each calculated as a distance measure product between the query vector and each of the output state vectors,
concatenating the unnormalized attention values and the unnormalized gate value, and
exponentially normalizing the concatenated attention and gate values to produce a vector of attention probability masses and the gate probability mass;
calculating the results of the vocabulary network by computing, from the final output state vector of the RNN, a distribution of vocabulary probability masses over tokens in the vocabulary text;
combining results after applying the gate probability mass to at least one of results of the pointer network and the vocabulary network; and
selecting the token responsive to the query vector from tokens in the context text or in the vocabulary text, based on the combined results.
19. A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the system of claim 1 .
20. A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the system of claim 16 .
21. A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the system of claim 18 .
22. A system including one or more processors coupled to memory, the memory loaded with computer instructions, the instructions, when executed on the processors, implement actions of claim 1 .
23. A system including one or more processors coupled to memory, the memory loaded with computer instructions, the instructions, when executed on the processors, implement actions of claim 16 .
24. A system including one or more processors coupled to memory, the memory loaded with computer instructions, the instructions, when executed on the processors, implement actions of claim 18 .