System and method for feature-rich continuous space language models
View Patent ↗Disclosed herein are systems, methods, and non-transitory computer-readable storage media for predicting probabilities of words for a language model. An exemplary system configured to practice the method receives a sequence of words and external data associated with the sequence of words and maps the sequence of words to an X-dimensional vector, corresponding to a vocabulary size. Then the system processes each X-dimensional vector, based on the external data, to generate respective Y-dimensional vectors, wherein each Y-dimensional vector represents a dense continuous space, and outputs at least one next word predicted to follow the sequence of words based on the respective Y-dimensional vectors. The X-dimensional vector, which is a binary sparse representation, can be higher dimensional than the Y-dimensional vector, which is a dense continuous space. The external data can include part-of-speech tags, topic information, word similarity, word relationships, a particular topic, and succeeding parts of speech in a given history.
1. A method comprising:
receiving a sequence of words and external data associated with the sequence of words;
mapping each word of the sequence of words to a respective X-dimensional vector based on interactions of the each word with previous words in the sequence of words, to yield a linear-based probability function comprising the respective X-dimensional vector of each word in the sequence of words, where a X-dimensional size vector corresponds to a size of a vocabulary;
processing, via a processor, each respective X-dimensional vector, based on the external data, to generate Y-dimensional vectors, wherein each Y-dimensional vector represents a non-linear based probability function using non-linear dependencies among the sequence of words based on the external data, and wherein the external data comprises word similarity based on a dot product of a probability of each word in the sequence of words and a probability of any other word in the vocabulary;
linearly mapping, for each word in the sequence of words, a discrete part-of-speech tag into the Y-dimensional vectors, to yield a modified non-linear based probability function;
weighting the linear-based probability function based on a rate of context variation, to yield a weighted linear-based probability function;
weighting the modified non-linear based probability function based on the rate of context variation, to yield a weighted modified non-linear based probability function; and
outputting a predicted next word to follow the sequence of words, the predicted next word predicted using a sum of the weighted linear-based probability function and the weighted modified non-linear based probability function.
2. The method of claim 1 , wherein the X-dimensional size vector is a higher dimensional representation than a Y-dimensional size vector.
3. The method of claim 1 , wherein the external data further comprises one of topic information, a first distribution of word relationships, a second distribution based on a particular topic, and a third distribution of possible succeeding parts of speech in a given history.
4. The method of claim 1 , wherein the predicted next word is used for one of spelling correction, language generation, and speech recognition.
5. The method of claim 1 , wherein the size of the vocabulary is at least 10,000 words, and the Y-dimensional mapping is less than 1,000 dimensions.
6. The method of claim 1 , wherein the external data further comprises a long-range dependency of a current topic according to one of a paragraph level and a document level.
7. A system comprising:
a processor; and
a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
receiving a sequence of words and external data associated with the sequence of words;
mapping each word of the sequence of words to a respective X-dimensional vector based on interactions of the each word with previous words in the sequence of words, to yield a linear-based probability function comprising the respective X-dimensional vector of each word in the sequence of words, where a X-dimensional size vector corresponds to a size of a vocabulary;
processing each respective X-dimensional vector, based on the external data, to generate Y-dimensional vectors, wherein each Y-dimensional vector represents a non-linear based probability function using non-linear dependencies among the sequence of words based on the external data, and wherein the external data comprises word similarity based on a dot product of a probability of each word in the sequence of words and a probability of any other word in the vocabulary; and
linearly mapping, for each word in the sequence of words, a discrete part-of-speech tag into the Y-dimensional vectors, to yield a modified non-linear based probability function;
weighting the linear-based probability function based on a rate of context variation, to yield a weighted linear-based probability function;
weighting the modified non-linear based probability function based on the rate of context variation, to yield a weighted modified non-linear based probability function; and
outputting a predicted next word to follow the sequence of words, the predicted next word predicted using a sum of the weighted linear-based probability function and the weighted modified non-linear based probability function.
8. The system of claim 7 , wherein the X-dimensional size vector is a higher dimensional representation than a Y-dimensional size vector.
9. The system of claim 7 , wherein the external data further comprises one of topic information, a first distribution of word relationships, a second distribution based on a particular topic, and a third distribution of possible succeeding parts of speech in a given history.
10. The system of claim 7 , wherein the predicted next word is used for one of spelling correction, language generation, and speech recognition.
11. The system of claim 7 , wherein the size of the vocabulary is at least 10,000 words, and the Y-dimensional mapping is less than 1,000 dimensions.
12. The system of claim 7 , wherein the external data further comprises a long-range dependency of a current topic according to one of a paragraph level and a document level.
13. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:
receiving a sequence of words and external data associated with the sequence of words;
mapping each word of the sequence of words to a respective X-dimensional vector based on interactions of the each word with previous words in the sequence of words, to yield a linear-based probability function comprising the respective X-dimensional vector of each word in the sequence of words, where a X-dimensional size vector corresponds to a size of a vocabulary;
processing each respective X-dimensional vector, based on the external data, to generate Y-dimensional vectors, wherein each Y-dimensional vector represents a non-linear based probability function using non-linear dependencies among the sequence of words based on the external data, and wherein the external data comprises word similarity based on a dot product of a probability of each word in the sequence of words and a probability of any other word in the vocabulary; and
linearly mapping, for each word in the sequence of words, a discrete part-of-speech tag into the Y-dimensional vectors, to yield a modified non-linear based probability function;
weighting the linear-based probability function based on a rate of context variation, to yield a weighted linear-based probability function;
weighting the modified non-linear based probability function based on the rate of context variation, to yield a weighted modified non-linear based probability function; and
outputting a predicted next word to follow the sequence of words, the predicted next word predicted using a sum of the weighted linear-based probability function and the weighted modified non-linear based probability function.
14. The computer-readable storage device of claim 13 , wherein the X-dimensional size vector is a higher dimensional representation than a Y-dimensional size vector.
15. The computer-readable storage device of claim 13 , wherein the external data further comprises one of topic information, a first distribution of word relationships, a second distribution based on a particular topic, and a third distribution of possible succeeding parts of speech in a given history.
16. The computer-readable storage device of claim 13 , wherein the size of the vocabulary is at least 10,000 words, and the Y-dimensional mapping is less than 1,000 dimensions.
17. The computer-readable storage device of claim 13 , wherein the external data further comprises a long-range dependency of a current topic according to one of a paragraph level and a document level.