Real time parsing and suggestions from pre-generated corpus with hypernyms
Systems and methods of natural language processing in an environment with no existing corpus are disclosed. The method includes defining an input grammar specific to a chosen domain, the input grammar having a domain specific knowledge and general grammatical knowledge. Groups of tokens are identified within the input grammar having syntactic and semantic equivalence. The identified groups are assembled into hypernyms, wherein the hypernyms include a semantic output for each token in the hypernyms. A list of fields is then combined with the hypernyms for combination with the hypernyms. A corpus of possible combinations of hypernyms and fields is created. A data structure mapping each possible combination to a partial semantic output is generated and the data structure is saved for use in later processing.
1. A non-transitory computer readable medium storing computer executable instructions that when executed by a processor cause the processor to perform acts comprising:
receiving an input query string from a remote device via a network connection;
assembling each of one or more semantic groups into a hypernym to obtain a plurality of hypernyms, wherein each hypernym comprises a hypernym data structure including a hypernym identifier combining a corresponding semantic identifier and syntactic identifier of at least one of the one or more semantic groups, and tokens of at least one of the one or more semantic groups mapped to semantic outputs, wherein at least one of the semantic outputs corresponding to at least one of the tokens includes a semantic identifier linking to another hypernym;
providing a list of fields for combination with the plurality of hypernyms, wherein the list of fields comprises text fields for input of at least one of names or keywords and custom tokens for input of at least one of dates or locations;
generating a corpus of valid combinations of hypernyms and fields based on at least some of the plurality of hypernyms and at least some of the list of fields;
determining semantics of the input query string based on a tokenization of the input query string and the corpus; and
transmitting, to the remote device, a communication associated with a meaning of the input query string based on the semantics.
2. The computer readable medium of claim 1 , wherein the corpus of valid combinations includes valid combinations less than a given length.
3. The computer readable medium of claim 1 , the acts further comprising generating a mapping data structure mapping partial tokens to each valid combination whose first defined number of tokens match the partial tokens.
4. The computer readable medium of claim 3 , wherein determining semantics of the input query string is based on the mapping data structure.
5. The computer readable medium of claim 1 , the acts further comprising filtering the corpus to remove pragmatically irrelevant items.
6. The computer readable medium of claim 1 , wherein determining semantics of the input query string comprises:
tokenizing the input query string to obtain the tokenization of the input query string.
7. The computer readable medium of claim 6 , wherein determining semantics of the input query string comprises:
identifying a combination of hypernyms and fields in the corpus that matches the tokenization; and
determining the semantics of the input query string based on the combination.
8. A method, comprising:
receiving an input query string from a remote device via a network connection;
assembling each of one or more semantic groups into a hypernym to obtain a plurality of hypernyms, wherein each hypernym comprises a hypernym data structure including a hypernym identifier combining a corresponding semantic identifier and syntactic identifier of at least one of the one or more semantic groups, and tokens of at least one of the one or more semantic groups mapped to semantic outputs, wherein at least one of the semantic outputs corresponding to at least one of the tokens includes a semantic identifier linking to another hypernym;
providing a list of fields for combination with the plurality of hypernyms, wherein the list of fields comprises text fields for input of at least one of names or keywords and custom tokens for input of at least one of dates or locations;
generating a corpus of valid combinations of hypernyms and fields based on at least some of the plurality of hypernyms and at least some of the list of fields;
determining semantics of the input query string based on a tokenization of the input query string and the corpus; and
transmitting, to the remote device, a communication associated with a meaning of the input query string based on the semantics.
9. The method of claim 8 , wherein the corpus of valid combinations includes valid combinations less than a given length.
10. The method of claim 8 , further comprising generating a mapping data structure mapping partial tokens to each valid combination whose first defined number of tokens match the partial tokens.
11. The method of claim 10 , wherein determining semantics of the input query string is based on the mapping data structure.
12. The method of claim 8 , further comprising filtering the corpus to remove pragmatically irrelevant items.
13. The method of claim 8 , wherein determining semantics of the input query string comprises:
tokenizing the input query string to obtain the tokenization of the input query string.
14. The method of claim 13 , wherein determining semantics of the input query string comprises:
identifying a combination of hypernyms and fields in the corpus that matches the tokenization; and
determining the semantics of the input query string based on the combination.
15. A system, comprising:
a processor; and
memory storing computer executable instructions that cause the processor to perform acts comprising:
receiving an input query string from a remote device via a network connection;
assembling each of one or more semantic groups into a hypernym to obtain a plurality of hypernyms, wherein each hypernym comprises a hypernym data structure including a hypernym identifier combining a corresponding semantic identifier and syntactic identifier of at least one of the one or more semantic groups, and tokens of at least one of the one or more semantic groups mapped to semantic outputs, wherein at least one of the semantic outputs corresponding to at least one of the tokens includes a semantic identifier linking to another hypernym;
providing a list of fields for combination with the plurality of hypernyms, wherein the list of fields comprises text fields for input of at least one of names or keywords and custom tokens for input of at least one of dates or locations;
generating a corpus of valid combinations of hypernyms and fields based on at least some of the plurality of hypernyms and at least some of the list of fields;
determining semantics of the input query string based on a tokenization of the input query string and the corpus; and
transmitting, to the remote device, a communication associated with a meaning of the input query string based on the semantics.
16. The system of claim 15 , wherein the corpus of valid combinations includes valid combinations less than a given length.
17. The system of claim 15 , the acts further comprising generating a mapping data structure mapping partial tokens to each valid combination whose first defined number of tokens match the partial tokens.
18. The system of claim 17 , wherein determining semantics of the input query string is based on the mapping data structure.
19. The system of claim 15 , the acts further comprising filtering the corpus to remove pragmatically irrelevant items.
20. The system of claim 15 , wherein determining semantics of the input query string comprises:
tokenizing the input query string to obtain the tokenization of the input query string;
identifying a combination of hypernyms and fields in the corpus that matches the tokenization; and
determining the semantics of the input query string based on the combination.