Identifying word collocations in natural language texts
View Patent ↗Systems and methods for identifying word collocations in natural language texts. An example method comprises: performing, by a computing device, semantico-syntactic analysis of a natural language text to produce a plurality of semantic structures; generating, in view of relationships defined by the semantic structures, a raw list of word combinations; producing a list of collocations by applying a heuristic filter to the raw list of word combinations; and using the list of collocations to perform a natural language processing operation.
1. A method, comprising:
performing, by a computing device, semantico-syntactic analysis of a natural language text to produce a plurality of semantic structures;
generating, in view of relationships defined by the semantic structures, a raw list of word combinations;
determining an expected frequency of a word combination as a product of frequencies, within the plurality of semantic structures, of semantic classes representing individual words of the word combination and frequencies of relationships between words comprised by the word combination;
responsive to determining that a quality metric represented by a difference between an actual frequency and the expected frequency of the word combination falls below a quality metric threshold, removing the word combination from the raw list of word combinations to produce a list of collocations; and
using the list of collocations to perform a natural language processing operation.
2. The method of claim 1 , wherein generating the raw list of word combinations comprises traversing a semantic structure of the plurality of semantic structures.
3. The method of claim 1 , wherein generating the raw list of word combinations comprises determining values of a pre-defined statistic for each word combination comprised by the raw list of word combinations.
4. The method of claim 1 , wherein producing the list of collocations further comprises evaluating an attribute of each word combination comprised by the raw list of word combinations.
5. The method of claim 1 , wherein producing the list of collocations further comprises removing a word combination associated with a red-flagged word.
6. The method of claim 1 , wherein producing the list of collocations further comprises replacing a word combination with a corresponding abstract collocation pattern.
7. The method of claim 1 , wherein producing the list of collocations further comprises removing a word combination that is contained by another word combination referenced by the raw list of word combinations.
8. The method of claim 1 , wherein producing the list of collocations further comprises removing a word combination that contains another word combination referenced by the raw list of word combinations.
9. The method of claim 1 , wherein producing the list of collocations further comprises removing a word combination having a distance between words in a linear word order that exceeds a maximum distance threshold.
10. A system, comprising:
a memory;
a processor, coupled to the memory, the processor configured to:
perform semantico-syntactic analysis of a natural language text to produce a plurality of semantic structures;
generate, in view of relationships defined by the semantic structures, a raw list of word combinations;
produce a list of collocations by applying a heuristic filter to the raw list of word combinations, wherein applying the heuristic filter comprises removing a word combination having a quality metric value falling below a quality metric threshold, wherein the quality metric is represented by a difference between an actual frequency and an expected frequency of the word combination, and wherein the expected frequency of a word combination is determined as a product of frequencies, within the plurality of semantic structures, of semantic classes representing individual words of the word combination and frequencies of relationships between words comprised by the word combination; and
use the list of collocations to perform a natural language processing operation.
11. The system of claim 10 , wherein generating the raw list of word combinations comprises traversing a semantic structure of the plurality of semantic structures.
12. The system of claim 10 , wherein generating the raw list of word combinations comprises determining values of a pre-defined statistic for each word combination comprised by the raw list of word combinations.
13. The system of claim 10 , wherein applying the heuristic filter comprises replacing a word combination with a corresponding abstract collocation pattern.
14. The system of claim 10 , wherein applying the heuristic filter comprises removing a word combination that is contained by another word combination referenced by the raw list of word combinations.
15. The system of claim 10 , wherein applying the heuristic filter comprises removing a word combination that contains another word combination referenced by the raw list of word combinations.
16. The system of claim 10 , wherein applying the heuristic filter comprises removing a word combination having a distance between words in a linear word order that exceeds a maximum distance threshold.
17. A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
performing semantico-syntactic analysis of a natural language text to produce a plurality of semantic structures;
generating, in view of relationships defined by the semantic structures, a raw list of word combinations;
producing a list of collocations by applying a heuristic filter to the raw list of word combinations, wherein applying the heuristic filter comprises removing a word combination having a quality metric value falling below a quality metric threshold, wherein the quality metric is represented by a difference between an actual frequency and an expected frequency of the word combination, and wherein the expected frequency of a word combination is determined as a product of frequencies, within the plurality of semantic structures, of semantic classes representing individual words of the word combination and frequencies of relationships between words comprised by the word combination; and
using the list of collocations to perform a natural language processing operation.
18. The computer-readable non-transitory storage medium of claim 17 , wherein applying the heuristic filter comprises replacing a word combination with a corresponding abstract collocation pattern.
19. The computer-readable non-transitory storage medium of claim 17 , wherein applying the heuristic filter comprises removing a word combination that contains another word combination referenced by the raw list of word combinations.
20. The computer-readable non-transitory storage medium of claim 17 , wherein applying the heuristic filter comprises removing a word combination having a distance between words in a linear word order that exceeds a maximum distance threshold.