Named entity recognition from short unstructured text

1. A computer-implemented method for recognizing named entities in a message, comprising:

receiving an electronic message;

tokenizing the electronic message;

probabilistically identifying, using either a short-phrase classifier or a regular text classifier, whether each token constitutes a named entity, thereby identifying one or more named entities within the electronic message, wherein each token is a continuous sequence of characters grouped together, and wherein the probabilistically identifying comprises accessing the short-term classifier or the regular text classifier depending on whether the number of tokens being less than or equal to a predefined number or a threshold number;

comparing each token with one or more databases comprising known named entities, thereby identifying the one or more name entities within the electronic message, wherein

the probabilistically identifying and the comparing of each token is performed simultaneously; and

returning a response message to the user when the one or more named entities are found in the electronic message, wherein the response message identifies the one or more named entities from the comparison, or

returning a null response message when the comparison fails to identify named entities in the electronic message.

2. The computer-implemented method of claim 1 , wherein the named entities comprise a name, a location, and/or an organization.

3. The computer-implemented method of claim 1 , wherein the tokenizing of the electronic message comprises converting each sub-word, word, or punctuation within the electronic message into tokens for recognizing the one or more named entities.

4. The computer-implemented method of claim 1 , wherein the probabilistically identifying comprises

accessing a short-phrase classifier to probabilistically identify whether one or more tokens or a sequence of tokens constitutes the one or more named entities, when a number of tokens is less than or equal to a threshold.

5. The computer-implemented method of claim 1 , wherein a process for probabilistically identifying comprises

projecting the one or more tokens or the sequence of tokens in a d-dimension vector space, where d is 100, 200, 300, or 600; and

further projecting a token vector for each of the one or more tokens or the sequence of tokens to a dimensional space where linear separation is possible, providing a maximum likelihood of distinguishing the one or more tokens or the sequences of tokens representing the one or more named entities from other tokens or sequences that do not represent named entities.

6. The computer-implemented method of claim 1 , further comprising

accessing a regular text classifier to probabilistically identify whether one or more tokens or a sequence of tokens constitutes the one or more named entities, when a number of tokens is greater than a threshold.

7. The computer-implemented method of claim 1 , further comprising

accessing one or more gazetteer lookup databases to scan for a recognized named entity; and

filtering each token that fails to match with a named entity to quickly identify a token that contains the named entity.

8. The computer-implemented method of claim 1 , further comprising:

removing duplicates for the one or more named entities by comparing a result from a short-phrase classifier, a regular text classifier, a gazetteer lookup database, or any combination thereof.

9. The computer-implemented method of claim 1 , further comprising:

combining one or more tokens to form the one or more named entities, wherein the one or more tokens are results from a short-phrase classifier, a regular text classifier, a gazetteer lookup database, or any combination thereof.

10. The computer-implemented method of claim 1 , further comprising:

selecting a superset of the one or more named entities when a short-phrase classifier, a regular text classifier, a gazetteer lookup database, or any combination thereof returns the superset of the one or more named entities and a subset of the superset of the one or more named entities.

11. An apparatus for recognizing named entities in an electronic communication, comprising:

at least one processor; and

memory comprising a set of instructions, wherein

the set of instructions are configured to cause the at least one processor to

receive an electronic message;

tokenize the electronic message;

probabilistically identify, using either a short-phrase classifier or a regular text classifier, whether each token constitutes a named entity, thereby identifying one or more named entities within the electronic message, wherein each token is a continuous sequence of characters grouped together, and wherein the probabilistically identifying comprises access the short-term classifier or the regular text classifier depending on whether the number of tokens being less than or equal to a predefined number or a threshold number;

compare each token, with one or more databases comprising known named entities, thereby identified the one or more name entities within the electronic message, wherein

the probabilistically identifying and the comparing of each token is performed simultaneously; and

return a response message to the user when the one or more named entities are found in the electronic message, wherein the response message identifies the one or more named entities from the comparison, or

return a null response message when the comparison fails to identify named entities in the electronic message.

12. The apparatus of claim 11 , wherein the named entities comprise a name, a location, and/or an organization.

13. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to convert each sub-word, word, or punctuation within the electronic message into tokens for recognizing the one or more named entities.

14. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

access a short-phrase classifier to probabilistically identify whether one or more tokens or a sequence of tokens constitutes the one or more named entities when a number of tokens is less than or equal to a threshold.

15. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

project the one or more tokens or the sequence of tokens in a d-dimension vector space, where d is 100, 200, 300, or 600; and

further project a token vector for each of the one or more tokens or the sequence of tokens to a dimensional space where linear separation is possible, providing a maximum likelihood of distinguishing the one or more tokens or the sequences of tokens representing the one or more named entities from other tokens or sequences that do not represent named entities.

16. The apparatus of claim 12 , wherein the set of instructions are further configured to cause the at least one processor to

access a regular text classifier to probabilistically identify whether one or more tokens or a sequence of tokens constitutes the one or more named entities when a number of tokens is greater than a threshold.

17. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

access one or more gazetteer lookup databases to scan for a recognized named entity; and

filter each token that fails to match with a named entity to quickly identify a token that contains the named entity.

18. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

remove duplicates for the one or more named entities by comparing a result from a short-phrase classifier, a regular text classifier, a gazetteer lockup database, or any combination thereof.

19. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

combine one or more tokens to form the one or more named entities, the one or more tokens are results from a short-phrase classifier, a regular text classifier, a gazetteer lockup database, or any combination thereof.

20. The apparatus of claim 11 , wherein the set of instructions are further configured to cause the at least one processor to

select a superset of the one or more named entities when a short-phrase classifier, a regular text classifier, a gazetteer lockup database, or any combination thereof returns the superset of the one or more named entities and a subset of the superset of the one or more named entities.

21. A computer-implemented process, comprising:

tokenizing an electronic message received from another computing device;

probabilistically identifying, using either a short-phrase classifier or a regular text classifier, whether each token constitutes a named entity, thereby identifying one or more named entities within the electronic message, wherein each token is a continuous sequence of characters grouped together and wherein the probabilistically identifying comprises accessing the short-term classifier or the regular text classifier depending on whether the number of tokens being less than or equal to a predefined number or a threshold number; and

returning a response message to the user when the one or more named entities are found in the electronic message, or

returning a null response message when the comparison fails to identify named entities in the electronic message.