Automatic generation of N-grams and concept relations from linguistic input data
A method of automatically generating a lemma dictionary from a web resource may include extracting a plurality of tokens from text-based documents within the web resource, and generating a plurality of N-grams from the plurality of tokens. The method may additionally include receiving one or more filter definitions that identify valid N-grams, and filtering the plurality of N-grams using the one or more filter definitions to generate a lemma dictionary. The method may further include generating an ontology that comprises the lemma dictionary.
1. A method of automatically generating a lemma dictionary from a web resource, the method comprising:
extracting a plurality of tokens from text-based documents within the web resource;
generating a plurality of N-grams from the plurality of tokens;
receiving one or more filter definitions that identify valid N-grams;
filtering the plurality of N-grams using the one or more filter definitions to generate a lemma dictionary;
generating an ontology that comprises the lemma dictionary;
causing a user interface to display a network of nodes representing the lemma dictionary; and
receiving one or more inputs that assign ontology relationships to the network of nodes.
2. The method of claim 1 wherein extracting the plurality of tokens from the text-based documents comprises identifying and eliminating structural and formatting text that is thereby excluded from the plurality of tokens.
3. The method of claim 1 wherein the web resource comprises a web domain, and wherein the web domain comprises a plurality of HTML webpages.
4. The method of claim 1 wherein generating the plurality of N-grams comprises generating word combinations as they appear in the web resource.
5. The method of claim 1 further comprising:
causing a user interface to be displayed after filtering the plurality of N-grams;
receiving input that eliminates at least one of the N-grams in the lemma dictionary.
6. The method of claim 1 wherein the one or more filter definitions comprises a part-of-speech filter for individual tokens in an N-gram.
7. The method of claim 1 wherein the one or more filter definitions comprises a text pattern.
8. The method of claim 1 wherein the one or more filter definitions comprises a minimum frequency for an N-gram to appear in the web resource.
9. The method of claim 1 wherein the one or more filter definitions comprises a selection of a language.
10. A non-transitory, computer-readable medium comprising instructions which, when executed by one or more processors, causes the one or more processors to perform operations comprising:
extracting a plurality of tokens from text-based documents within the web resource;
generating a plurality of N-grams from the plurality of tokens;
receiving one or more filter definitions that identify valid N-grams;
filtering the plurality of N-grams using the one or more filter definitions to generate a lemma dictionary;
generating an ontology that comprises the lemma dictionary;
causing a user interface to display a network of nodes representing the lemma dictionary; and
receiving one or more inputs that assign ontology relationships to the network of nodes.
11. The non-transitory, computer-readable medium of claim 10 wherein generating the plurality of N-grams comprises generating word combinations as they appear in the web resource.
12. The non-transitory, computer-readable medium of claim 10 wherein the one or more filter definitions comprises a part-of-speech filter for individual tokens in an N-gram.
13. The non-transitory, computer-readable medium of claim 10 wherein the one or more filter definitions comprises a text pattern.
14. The non-transitory, computer-readable medium of claim 10 wherein the one or more filter definitions comprises a minimum frequency for an N-gram to appear in the web resource.
15. The non-transitory, computer-readable medium of claim 10 wherein the one or more filter definitions comprises a selection of a language.
16. A system comprising:
one or more processors; and
one or more memory devices comprising instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
extracting a plurality of tokens from text-based documents within the web resource;
generating a plurality of N-grams from the plurality of tokens;
receiving one or more filter definitions that identify valid N-grams;
filtering the plurality of N-grams using the one or more filter definitions to generate a lemma dictionary;
generating an ontology that comprises the lemma dictionary;
causing a user interface to display a network of nodes representing the lemma dictionary; and
receiving one or more inputs that assign ontology relationships to the network of nodes.
17. The system of claim 16 wherein generating the plurality of N-grams comprises generating word combinations as they appear in the web resource.
18. The system of claim 16 wherein the one or more filter definitions comprises a part-of-speech filter for individual tokens in an N-gram.
19. The system of claim 16 wherein the one or more filter definitions comprises a minimum frequency for an N-gram to appear in the web resource.
20. The system of claim 16 wherein the one or more filter definitions comprises a selection of a language.