System and method for identifying phrases in text
View Patent ↗A method includes accessing text that includes a plurality of words, tagging each of the plurality of words with one of a plurality of parts of speech (POS) tags, and creating a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag. The method further includes clustering one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens, and forming a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.
1. A system, comprising:
one or more memory units; and
one or more processing units operable to:
access text comprising a plurality of words;
tag each of the plurality of words with one of a plurality of parts of speech (POS) tags;
create a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;
cluster one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and
form a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.
2. The system of claim 1 , wherein the plurality of POS tags comprises:
a POS tag for a proper noun that is a geographic name;
a POS tag for a proper noun that is a name of a person; and
a POS tag for a proper noun that is can be both a geographic name and a name of a person.
3. The system of claim 1 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises utilizing a lexical analyzer to determine whether each of the plurality of words is certain type of entity, the certain type of entity chose from a plurality of entities comprising:
an acronym;
a plurality of initials;
a uniform resource locator (URL); and
an email address.
4. The system of claim 1 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.
5. The system of claim 1 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.
6. The system of claim 1 , the one or more processing units further operable to provide a count of the number of the words in the text, the count based on the plurality of created tokens.
7. The system of claim 1 , wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.
8. A computer-implemented method, comprising:
accessing text by a processing system, the text comprising a plurality of words;
tagging, by the processing system, each of the plurality of words with one of a plurality of parts of speech (POS) tags;
creating, by the processing system, a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;
clustering, by the processing system, one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and
forming, by the processing system, a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.
9. The computer-implemented method of claim 8 , wherein the plurality of POS tags comprises:
a POS tag for a proper noun that is a geographic name;
a POS tag for a proper noun that is a name of a person; and
a POS tag for a proper noun that is can be both a geographic name and a name of a person.
10. The computer-implemented method of claim 8 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises utilizing a lexical analyzer to determine whether each of the plurality of words is certain type of entity, the certain type of entity chose from a plurality of entities comprising:
an acronym;
a plurality of initials;
a uniform resource locator (URL); and
an email address.
11. The computer-implemented method of claim 8 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.
12. The computer-implemented method of claim 8 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.
13. The computer-implemented method of claim 8 , further comprising providing a count of the number of the words in the text, the count based on the plurality of created tokens.
14. The computer-implemented method of claim 8 , wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.
15. A non-transitory computer-readable medium comprising software, the software when executed by one or more processing units operable to perform operations comprising:
accessing text comprising a plurality of words;
tagging each of the plurality of words with one of a plurality of parts of speech (POS) tags;
creating a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;
clustering one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and
forming a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.
16. The non-transitory computer-readable medium of claim 15 , wherein the plurality of POS tags comprises:
a POS tag for a proper noun that is a geographic name;
a POS tag for a proper noun that is a name of a person; and
a POS tag for a proper noun that is can be both a geographic name and a name of a person.
17. The non-transitory computer-readable medium of claim 15 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.
18. The non-transitory computer-readable medium of claim 15 , wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.
19. The non-transitory computer-readable medium of claim 15 , further comprising providing a count of the number of the words in the text, the count based on the plurality of created tokens.
20. The non-transitory computer-readable medium of claim 15 , wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.