Multi-stage recognition of named entities in natural language text based on morphological and semantic features
Systems and methods for multi-stage recognition of named entities based on morphological and semantic features of natural language texts. An example method comprises: performing a lexico-morphological analysis of a natural language text comprising a plurality of tokens, each token comprising at least one natural language word; determining, based on the lexico-morphological analysis, one or more lexical meanings and grammatical meanings associated with each token of the plurality of tokens; for each token the plurality of tokens, evaluating one or more classifier functions using the lexical and grammatical meanings associated with the tokens, wherein a value of each classifier function is indicative of a degree of association of the token with a category of named entities; performing a syntactico-semantic analysis of at least part of the natural language text to produce a plurality of semantic structures representing the part of the natural language text; and interpreting the semantic structures using a set of production rules to determine, for one or more tokens comprised by the part of the natural language text, a degree of association of the token with a category of named entities.
1. A method, comprising:
performing, by a computer system, a lexico-morphological analysis of a natural language text comprising a plurality of tokens, each token comprising at least one natural language word;
determining, based on the lexico-morphological analysis, one or more lexical meanings and grammatical meanings associated with each token of the plurality of tokens;
for each token of the plurality of tokens, evaluating one or more classifier functions using the lexical and grammatical meanings associated with the tokens, wherein a value of each classifier function is indicative of a degree of association of the token with a category of named entities;
performing a syntactico-semantic analysis of at least part of the natural language text to produce a plurality of semantic structures representing the part of the natural language text; and
interpreting the semantic structures using a set of production rules to determine, for one or more tokens comprised by the part of the natural language text, a degree of association of the token with a category of named entities.
2. The method of claim 1 , wherein the syntactico-semantic analysis of the part of the natural language text is performed responsive to determining that a metric indicative of a quality of associating the tokens with respective categories falls below a threshold value.
3. The method of claim 1 , wherein the syntactico-semantic analysis of the part of the natural language text is performed responsive to receiving a user interface input.
4. The method of claim 1 , further comprising:
identifying the part of the natural language text as comprising at least one sentence in which no tokens have been associated with categories.
5. The method of claim 1 , further comprising:
identifying the part of the natural language text as comprising at least one sentence in which a metric indicative of a quality of associating the tokens with respective categories falls below a threshold value.
6. The method of claim 5 , wherein the metric is provided by a ratio of a number of tokens that have been correctly associated with categories to a number of tokens representing named entities that have been identified in the natural language text.
7. The method of claim 5 , wherein the metric is provided by a ratio of a number of tokens that have been correctly associated with categories to a total number of tokens representing named entities in the natural language text.
8. The method of claim 1 , wherein the syntactico-semantic analysis of the part of the natural language text is performed responsive to receiving a graphical user interface input.
9. The method of claim 1 , further comprising:
determining, using a training data set, at least one parameter of the classifier function, wherein the training data set comprises a training natural language text comprising a plurality of tokens, wherein each token is associated with a certain category.
10. The method of claim 1 , wherein each semantic structure of the plurality of semantic structures is represented by a graph comprising a plurality of nodes corresponding to a plurality of semantic classes and a plurality of edges corresponding to a plurality of semantic relationships.
11. The method of claim 1 , wherein a production rule comprises one or more logical expressions defined on one or more semantic structure templates.
12. A system, comprising:
a memory;
a processor, coupled to the memory, the processor configured to:
perform a lexico-morphological analysis of a natural language text comprising a plurality of tokens, each token comprising at least one natural language word;
determine, based on the lexico-morphological analysis, one or more lexical meanings and grammatical meanings associated with each token of the plurality of tokens;
for each token of the plurality of tokens, evaluate one or more classifier functions using the lexical and grammatical meanings associated with the tokens, wherein a value of each classifier function is indicative of a degree of association of the token with a category of named entities;
perform a syntactico-semantic analysis of at least part of the natural language text to produce a plurality of semantic structures representing the part of the natural language text; and
interpret the semantic structures using a set of production rules to determine, for one or more tokens comprised by the part of the natural language text, a degree of association of the token with a category of named entities.
13. The system of claim 12 , wherein the syntactico-semantic analysis of the part of the natural language text is performed responsive to determining that a metric indicative of a quality of associating the tokens with respective categories falls below a threshold value.
14. The system of claim 12 , wherein the syntactico-semantic analysis of the part of the natural language text is performed responsive to receiving a user interface input.
15. The system of claim 12 , wherein the processor is further configured to:
identify the part of the natural language text as comprising at least one sentence in which no tokens have been associated with categories.
16. The system of claim 12 , wherein the processor is further configured to:
identify the part of the natural language text as comprising at least one sentence in which a metric indicative of a quality of associating the tokens with respective categories falls below a threshold value.
17. The system of claim 16 , wherein the metric is provided by a ratio of a number of tokens that have been correctly associated with categories to a number of tokens representing named entities that have been identified in the natural language text.
18. The system of claim 16 , wherein the metric is provided by a ratio of a number of tokens that have been correctly associated with categories to a total number of tokens representing named entities in the natural language text.
19. The system of claim 12 , wherein the processor is further configured to:
determine, using a training data set, at least one parameter of the classifier function, wherein the training data set comprises a training natural language text comprising a plurality of tokens, wherein each token is associated with a certain category.
20. A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to:
perform a lexico-morphological analysis of a natural language text comprising a plurality of tokens, each token comprising at least one natural language word;
determine, based on the lexico-morphological analysis, one or more lexical meanings and grammatical meanings associated with each token of the plurality of tokens;
for each token of the plurality of tokens, evaluate one or more classifier functions using the lexical and grammatical meanings associated with the tokens, wherein a value of each classifier function is indicative of a degree of association of the token with a category of named entities;
perform a syntactico-semantic analysis of at least part of the natural language text to produce a plurality of semantic structures representing the part of the natural language text; and
interpret the semantic structures using a set of production rules to determine, for one or more tokens comprised by the part of the natural language text, a degree of association of the token with a category of named entities.