Predicting probability of occurrence of a string using sequence of vectors
View Patent ↗Systems and methods are disclosed to receive a plurality of strings where each string of the plurality of strings comprises a plurality of symbols. For each string of the plurality of strings, a first sequence of vectors is generated based at least on a maximum length of word for each symbol in the string. The first sequence of vectors is provided to a machine learning unit for each string of the plurality of strings. A probability of occurrence of each string of the plurality of strings is obtained from the machine learning unit.
1. A method comprising:
receiving a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generating, by a processing device, a first sequence of vectors based at least on a maximum length of word for each symbol in the string, wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising one or more of a word: 1) with a commonplace meaning, or 2) found in a dictionary;
providing to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtaining from the machine learning unit a probability of occurrence of each string of the plurality of strings.
2. The method of claim 1 , wherein each vector of the first sequence of vectors corresponds to each symbol in the string.
3. The method of claim 2 , wherein each vector of the first sequence of vectors is derived by joining together a first vector comprising the maximum length of word for a given symbol in the string and a second vector comprising a symbol vector for the given symbol in the string.
4. The method of claim 3 , wherein the symbol vector for each symbol in the string is based on one or more of:
a symbol embedding;
a unified vector for symbols of an alphabet that is different from the alphabet used in the string;
a vector for frequently occurring punctuation marks;
a vector for rare punctuation marks; or
a vector for numerals.
5. The method of claim 1 , wherein each string of the plurality of strings differs from each other string of the plurality of strings by one symbol, the one symbol being located on a same position of each string.
6. The method of claim 1 , wherein the machine learning unit comprises:
a first fully connected layer and a second fully connected layer to apply matrix transformation on the first sequence of vectors for each string; and
a third fully connected layer to be used as an output layer.
7. The method of claim 6 , wherein a batch normalization function and a rectifier linear unit activation function are applied on a first output of the first fully connected layer and on a second output of the second fully connected layer, and wherein a sigmoid activation function is applied on a third output of the third fully connected layer.
8. The method of claim 1 , wherein the probability of occurrence for a given string having a value nearest to ‘one’ relative to the probability of occurrence values of the remaining strings indicates that the given string has the highest probability of occurrence.
9. A system comprising:
a memory device storing instructions;
a processing device coupled to the memory device, the processing device to execute the instructions to:
receive a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generate a first sequence of vectors based at least on a maximum length of word for each symbol in the string, wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising a word with a commonplace meaning;
provide to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtain from the machine learning unit a probability of occurrence of each string of the plurality of strings.
10. The system of claim 9 , wherein each vector of the first sequence of vectors corresponds to each symbol in the string.
11. The system of claim 10 , wherein each vector of the first sequence of vectors is derived by joining together a first vector comprising the maximum length of word for a given symbol in the string and a second vector comprising a symbol vector for a given symbol in the string.
12. The system of claim 11 , wherein the symbol vector for each symbol in the string is based on one or more of:
a symbol embedding;
a unified vector for symbols of an alphabet that is different from the alphabet used in the string;
a vector for frequently occurring punctuation marks;
a vector for rare punctuation marks; or
a vector for numerals.
13. The system of claim 9 , wherein the longest possible word is found in a dictionary.
14. The system of claim 9 , wherein each string of the plurality of strings differs from each other string of the plurality of strings by one symbol, the one symbol being located on a same position of each string.
15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to:
receive a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generate a first sequence of vectors based at least on a maximum length of word for each symbol in the string, wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising a word found in a dictionary;
provide to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtain from the machine learning unit a probability of occurrence of each string of the plurality of strings.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the machine learning unit comprises:
a first fully connected layer and a second fully connected layer to apply matrix transformation on the first sequence of vectors for each string; and
a third fully connected layer to be used as an output layer.
17. The non-transitory computer-readable storage medium of claim 16 , wherein a batch normalization function and a rectifier linear unit activation function are applied on a first output of the first fully connected layer and on a second output of the second fully connected layer, and wherein a sigmoid activation function is applied on a third output of the third fully connected layer.