Automatic detection and cleansing of erroneous concepts in an aggregated knowledge base
A mechanism is provided for automatically detecting and cleansing erroneous concepts in an aggregated knowledge base. A graph data structure representing the concept present in a portion of the natural language content is generated. The graph data structure is analyzed to determine whether or not the graph data structure comprises one or more concept conflicts in association with a set of nodes in the graph data structure, the one or more concept conflicts are associated with the set of nodes if two or more nodes represent separate and distinct concepts. Responsive to determining that there are one or more concept conflicts due to there being two or more nodes representing separate and distinct concepts, the two or more nodes are split into separate distinct concepts within the knowledge base.
1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement natural language processing (NLP) system, wherein the method comprises:
receiving, by the NLP system, a portion of natural language content related to a selected concept from a knowledge base;
generating, by the NLP system, a graph data structure representing the concept present in the portion of the natural language content, wherein nodes of the graph data structure comprise a first node representing a name of the concept and a one or more other nodes representing synonyms associated with the first node and wherein the graph data structure further indicates relationships between the first node and the one or more other nodes based on a similarity measure of the other nodes to the first node;
analyzing, by the NLP system, the graph data structure to determine whether or not the graph data structure comprises one or more concept conflicts in association with a set of nodes in the graph data structure, wherein the one or more concept conflicts are associated with the set of nodes responsive to two or more nodes representing separate and distinct concepts; and
responsive to determining that there are one or more concept conflicts due to there being two or more nodes representing separate and distinct concepts, splitting, by the NLP system, the two or more nodes into separate distinct concepts within the knowledge base.
2. The method of claim 1 , wherein analyzing the graph data structure comprises:
automatically generating, by the NLP system, a first cluster of nodes including the first node and one or more other nodes associated with the first node; and
storing, by the NLP system, in the knowledge base, the first cluster of nodes in association with a first separate distinct concept.
3. The method of claim 2 , wherein automatically generating the cluster comprises:
assigning, by the NLP system, other nodes, in the one or more other nodes, to the first cluster of nodes associated with the first node based on the similarity measure of the other nodes to the first node.
4. The method of claim 3 , wherein the similarity measure is a cosine similarity or a Euclidean distance.
5. The method of claim 1 , wherein analyzing the graph data structure comprises:
identifying, by the NLP system, a second node in the graph data structure, wherein the second node is a node that has one or more other nodes connected to the second node by an edge in the graph and has a relatively high visiting probability;
automatically generating, by the NLP system, a second cluster of nodes including the second node and one or more other nodes associated with the second node; and
storing, by the NLP system, in the knowledge base, the second cluster of nodes in association with a second separate distinct concept.
6. The method of claim 5 , wherein identifying the second canonical concept node in the graph comprises:
calculating, by the NLP system, for each node in the graph, a visiting probability indicating a probability that the concept associated with the node will be used by a cognitive system to perform a cognitive operation; and
selecting, by the NLP system, a node having a largest visiting probability as the first node.
7. The method of claim 5 , wherein automatically generating the cluster comprises:
assigning, by the NLP system, other nodes, in the one or more other nodes, to the second cluster of nodes associated with the second node based on a distance of the other nodes to the second node.
8. The method of claim 7 , wherein the similarity measure is a cosine similarity or a Euclidean distance.
9. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a natural language processing (NLP) system which operates to:
receive a portion of natural language content related to a selected concept from a knowledge base;
generate a graph data structure representing the concept present in the portion of the natural language content, wherein nodes of the graph data structure comprise a first node representing a name of the concept and a one or more other nodes representing synonyms associated with the first node and wherein the graph data structure further indicates relationships between the first node and the one or more other nodes based on a similarity measure of the other nodes to the first node;
analyze the graph data structure to determine whether or not the graph data structure comprises one or more concept conflicts in association with a set of nodes in the graph data structure, wherein the one or more concept conflicts are associated with the set of nodes responsive to two or more nodes representing separate and distinct concepts; and
responsive to determining that there are one or more concept conflicts due to there being two or more nodes representing separate and distinct concepts, split the two or more nodes into separate distinct concepts within the knowledge base.
10. The computer program product of claim 9 , wherein the computer readable program to analyze the graph data structure further causes the computing device to implement the NLP system which operates to:
automatically generate a first cluster of nodes including the first node and one or more other nodes associated with the first node; and
store, in the knowledge base, the first cluster of nodes in association with a first separate distinct concept.
11. The computer program product of claim 10 , wherein the computer readable program to automatically generate the cluster further causes the computing device to implement the NLP system which operates to:
assign other nodes, in the one or more other nodes, to the first cluster of nodes associated with the first node based on the similarity measure of the other nodes to the first node.
12. The computer program product of claim 9 , wherein the computer readable program to analyze the graph data structure further causes the computing device to implement the NLP system which operates to:
identify a second node in the graph data structure, wherein the second node is a node that has one or more other nodes connected to the second node by an edge in the graph and has a relatively high visiting probability;
automatically generate a second cluster of nodes including the second node and one or more other nodes associated with the second node; and
store in the knowledge base, the second cluster of nodes in association with a second separate distinct concept.
13. The computer program product of claim 12 , wherein the computer readable program to identify the second canonical concept node in the graph further causes the computing device to implement the NLP system which operates to:
calculate for each node in the graph, a visiting probability indicating a probability that the concept associated with the node will be used by a cognitive system to perform a cognitive operation; and
select a node having a largest visiting probability as the first node.
14. The computer program product of claim 12 , wherein the computer readable program to automatically generate the cluster further causes the computing device to implement the NLP system which operates to:
assign other nodes, in the one or more other nodes, to the second cluster of nodes associated with the second node based on a distance of the other nodes to the second node.
15. An apparatus comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a natural language processing (NLP) system which operates to:
receive a portion of natural language content related to a selected concept from a knowledge base;
generate a graph data structure representing the concept present in the portion of the natural language content, wherein nodes of the graph data structure comprise a first node representing a name of the concept and a one or more other nodes representing synonyms associated with the first node and wherein the graph data structure further indicates relationships between the first node and the one or more other nodes based on a similarity measure of the other nodes to the first node;
analyze the graph data structure to determine whether or not the graph data structure comprises one or more concept conflicts in association with a set of nodes in the graph data structure, wherein the one or more concept conflicts are associated with the set of nodes responsive to two or more nodes representing separate and distinct concepts; and
responsive to determining that there are one or more concept conflicts due to there being two or more nodes representing separate and distinct concepts, split the two or more nodes into separate distinct concepts within the knowledge base.
16. The apparatus of claim 15 , wherein the instructions to analyze the graph data structure further cause the processor to implement the NLP system which operates to:
automatically generate a first cluster of nodes including the first node and one or more other nodes associated with the first node; and
store, in the knowledge base, the first cluster of nodes in association with a first separate distinct concept.
17. The apparatus of claim 16 , wherein the instructions to automatically generate the cluster further cause the processor to implement the NLP system which operates to:
assign other nodes, in the one or more other nodes, to the first cluster of nodes associated with the first node based on the similarity measure of the other nodes to the first node.
18. The apparatus of claim 15 , wherein the instructions to analyze the graph data structure further cause the processor to implement the NLP system which operates to:
identify a second node in the graph data structure, wherein the second node is a node that has one or more other nodes connected to the second node by an edge in the graph and has a relatively high visiting probability;
automatically generate a second cluster of nodes including the second node and one or more other nodes associated with the second node; and
store in the knowledge base, the second cluster of nodes in association with a second separate distinct concept.
19. The apparatus of claim 18 , wherein the instructions to identify the second canonical concept node in the graph further cause the processor to implement the NLP system which operates to:
calculate for each node in the graph, a visiting probability indicating a probability that the concept associated with the node will be used by a cognitive system to perform a cognitive operation; and
select a node having a largest visiting probability as the first node.
20. The apparatus of claim 18 , wherein the instructions to automatically generate the cluster further cause the processor to implement the NLP system which operates to:
assign other nodes, in the one or more other nodes, to the second cluster of nodes associated with the second node based on a distance of the other nodes to the second node.