Natural language processing using an ontology-based concept embedding model
View Patent ↗A computer system generates a vector space model based on an ontology of concepts. One or more training examples are extracted for one or more concepts of a hierarchical ontology, wherein the one or more training examples for the one or more concepts are based on neighboring concepts in the hierarchical ontology. A plurality of vectors, each including one or more features, are initialized, wherein each vector corresponds to a concept of the one or more concepts. A vector space model is generated by iteratively modifying one or more vectors of the plurality of vectors to optimize a loss function. Natural language processing is performed using the vector space model. Embodiments of the present invention further include a method and program product for generating a vector space model in substantially the same manner described above.
1. A computer-implemented method for natural language processing comprising:
extracting one or more training examples for one or more concepts of a hierarchical ontology, wherein the one or more training examples for the one or more concepts are based on neighboring concepts in the hierarchical ontology;
initializing a plurality of vectors each including one or more features, wherein each vector corresponds to a concept of the one or more concepts;
generating a vector space model by iteratively modifying one or more vectors of the plurality of vectors to optimize a loss function, wherein an output of the loss function for each concept is proportional to a first cosine similarity of a vector for the concept and a first mean vector, wherein the first mean vector is a mean of a plurality of vectors for child concepts of the concept; and
performing natural language processing using the vector space model.
2. The computer-implemented method of claim 1 , wherein the vector space model comprises a continuous bag of words model.
3. The computer-implemented method of claim 1 , wherein the one or more training examples for the one or more concepts include one or more positive training examples and one or more negative training examples, and wherein optimizing the loss function comprises:
modifying one or more vectors of the plurality of vectors to minimize the loss function for each of the one or more positive training examples; and
modifying one or more vectors of the plurality of vectors to maximize the loss function for each of the one or more negative training examples.
4. The computer-implemented method of claim 1 , wherein the mean of the first mean vector further includes one or more vectors for parent concepts of the concept.
5. The computer-implemented method of claim 4 , further comprising:
identifying one or more non-ambiguous text representations of a concept; and wherein the output of the loss function is further proportional to a second cosine similarity between the vector for the concept and a second mean vector, wherein the second mean vector is a mean of one or more word vectors based on the one or more non-ambiguous text representations.
6. The computer-implemented method of claim 5 , wherein the one or more non-ambiguous text representations are identified in one or more lexical corpora.
7. The computer-implemented method of claim 5 , further comprising:
identifying, using the vector space model, one or more words that are related to a queried concept based on a cosine similarity of word vectors of the one or more words to a concept vector of the queried concept.
8. The computer-implemented method of claim 5 , further comprising:
identifying, using the vector space model, one or more concepts that are related to a queried word based on a cosine similarity of concept vectors of the one or more concepts to a word vector of the queried word.
9. A computer system for natural language processing, the computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising instructions to:
extract one or more training examples for one or more concepts of a hierarchical ontology, wherein the one or more training examples for the one or more concepts are based on neighboring concepts in the hierarchical ontology;
initialize a plurality of vectors each including one or more features, wherein each vector corresponds to a concept of the one or more concepts;
generate a vector space model by iteratively modifying one or more vectors of the plurality of vectors to optimize a loss function, wherein an output of the loss function for each concept is proportional to a first cosine similarity of a vector for the concept and a first mean vector, wherein the first mean vector is a mean of a plurality of vectors for child concepts of the concept; and
perform natural language processing using the vector space model.
10. The computer system of claim 9 , wherein the vector space model comprises a continuous bag of words model.
11. The computer system of claim 9 , wherein the one or more training examples for the one or more concepts include one or more positive training examples and one or more negative training examples, and wherein the program instructions to optimize the loss function comprise instructions to:
modify one or more vectors of the plurality of vectors to minimize the loss function for each of the one or more positive training examples; and
modify one or more vectors of the plurality of vectors to maximize the loss function for each of the one or more negative training examples.
12. The computer system of claim 9 , wherein the mean of the first mean vector further includes one or more vectors for parent concepts of the concept.
13. The computer system of claim 12 , wherein the program instructions further comprise instructions to:
identify one or more non-ambiguous text representations of a concept; and wherein the output of the loss function is further proportional to a second cosine similarity between the vector for the concept and a second mean vector, wherein the second mean vector is a mean of one or more word vectors based on the one or more non-ambiguous text representations.
14. The computer system of claim 13 , wherein the one or more non-ambiguous text representations are identified in one or more lexical corpora.
15. The computer system of claim 13 , wherein the program instructions further comprise instructions to:
identify, using the vector space model, one or more words that are related to a queried concept based on a cosine similarity of word vectors of the one or more words to a concept vector of the queried concept.
16. The computer system of claim 13 , wherein the program instructions further comprise instructions to:
identify, using the vector space model, one or more concepts that are related to a queried word based on a cosine similarity of concept vectors of the one or more concepts to a word vector of the queried word.
17. A computer program product for natural language processing, the computer program product comprising one or more computer readable storage media collectively having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
extract one or more training examples for one or more concepts of a hierarchical ontology, wherein the one or more training examples for the one or more concepts are based on neighboring concepts in the hierarchical ontology;
initialize a plurality of vectors each including one or more features, wherein each vector corresponds to a concept of the one or more concepts;
generate a vector space model by iteratively modifying one or more vectors of the plurality of vectors to optimize a loss function, wherein an output of the loss function for each concept is proportional to a first cosine similarity of a vector for the concept and a first mean vector, wherein the first mean vector is a mean of a plurality of vectors for child concepts of the concept; and
perform natural language processing using the vector space model.
18. The computer program product of claim 17 , wherein the vector space model comprises a continuous bag of words model.
19. The computer program product of claim 17 , wherein the one or more training examples for the one or more concepts include one or more positive training examples and one or more negative training examples, and wherein the program instructions to optimize the loss function cause the computer to:
modify one or more vectors of the plurality of vectors to minimize the loss function for each of the one or more positive training examples; and
modify one or more vectors of the plurality of vectors to maximize the loss function for each of the one or more negative training examples.
20. The computer program product of claim 17 , wherein the mean of the first mean vector further includes one or more vectors for parent concepts of the concept.
21. The computer program product of claim 20 , wherein the program instructions further cause the computer to:
identify one or more non-ambiguous text representations of a concept; and wherein the output of the loss function is further proportional to a second cosine similarity between the vector for the concept and a second mean vector, wherein the second mean vector is a mean of one or more word vectors based on the one or more non-ambiguous text representations.
22. The computer program product of claim 21 , wherein the one or more non-ambiguous text representations are identified in one or more lexical corpora.
23. The computer program product of claim 21 , wherein the program instructions further cause the computer to: identify, using the vector space model, one or more words that are related to a queried concept based on a cosine similarity of word vectors of the one or more words to a concept vector of the queried concept.
24. The computer program product of claim 21 , wherein the program instructions further cause the computer to identify, using the vector space model, one or more concepts that are related to a queried word based on a cosine similarity of concept vectors of the one or more concepts to a word vector of the queried word.