Distributed entity re-resolution based on streaming updates
Mechanisms are provided for dynamic re-resolution of entities in a knowledge graph (KG) based on streaming updates. The KG and corresponding initial clusters associated with first entities are received along with a dynamic data stream having second documents referencing second entities. Clustering on the second documents based on the set of initial clusters, and document features of the second documents, is performed to provide a set of second document clusters. For second document clusters that should be modified based on entities associated with the second document cluster, a cluster modification operation is performed. Updated clusters are generated based on the clustering and modification of clusters. Entity re-resolution is dynamically performed on the entities in the KG based on the second entities associated with the updated clusters to generate an updated knowledge graph data structure.
1. A method, in a data processing system, the method comprising the data processing system:
receiving a knowledge graph data structure comprising data representations of a plurality of first entities specified in a first set of documents, and a corresponding set of initial clusters associated with corresponding ones of the plurality of first entities;
receiving at least one dynamic data stream from at least one source computing system, the at least one dynamic data stream comprising second documents having data specifying second entities referenced by the second documents, wherein each second document is a collection of unstructured textual data; and
automatically, in response to receiving the at least one dynamic data stream:
executing a clustering operation on the second documents based on the set of initial clusters, and document features of the second documents, to provide a set of second document clusters comprising the second documents, wherein the clustering operation is a modified Dirichlet Hawkes Process (DHP) that performs distributed clustering, in parallel, on partitions of the at least one dynamic data stream, across a master compute node and a plurality of slave compute nodes of the data processing system;
determining, for each second document cluster in the one or more second document clusters, whether the second document cluster should be modified based on entities associated with the second document cluster;
executing, for each second document cluster that is determined should be modified, a cluster modification operation on the second document cluster, wherein updated clusters are generated comprising a combination of second document clusters that are modified and second document clusters that are not modified;
dynamically executing entity re-resolution on the plurality of first entities in the knowledge graph data structure based on the second entities associated with the updated clusters to generate an updated knowledge graph data structure;
inputting information associated with the updated knowledge graph data structure into an artificial intelligence computing system and analyzing patterns of the entity re-resolution; and
generating an identity-fraud alert based on analyzing the patterns of the entity re-resolution and based on a determination that an entity is re-resolved a plurality of times over a time period.
2. The method of claim 1 , further comprising:
providing the updated knowledge graph data structure to a downstream computer system to perform a downstream computer system operation based on the updated knowledge graph data structure.
3. The method of claim 1 , wherein the clustering operation performs clustering based on temporal characteristics associated with the second entities referenced in the second documents of the at least one dynamic data stream.
4. The method of claim 3 , wherein the clustering operation comprises:
performing distributed DHP clustering when no new clusters are needed as part of the clustering operation; and
performing non-distributed DHP clustering when a new cluster is determined to be needed as part of the clustering operation.
5. The method of claim 1 , wherein the cluster modification operation comprises merging entities that only occur in the same second document cluster so that a single entity represents the same second document cluster in the updated clusters.
6. The method of claim 1 , wherein the cluster modification operation comprises:
determining whether an entity is present in more than one second document cluster; and
in response to the entity being present in more than one second document cluster:
associating the entity with a first one of the second document clusters of the entity; and
generating one or more sub-entities corresponding to the entity, wherein each of the one or more sub-entities is associated with a second one of second document clusters or a newly generated cluster.
7. The method of claim 6 , wherein the one or more sub-entities corresponding to the entity are entities corresponding to a smallest sub-cluster of at least one of the more than one second document cluster.
8. The method of claim 1 , wherein the at least one source computing system comprises at least one of a social media website, a social networking computer system, a news feed computer system, a document aggregator computer system, a document segregator computer system, or a data streaming services computer system, and wherein the streaming data comprises metadata and textual content corresponding to submissions from users of the at least one source computing system.
9. The method of claim 1 , further comprising:
inputting the updated knowledge graph data structure into a graph neural network that generates embeddings of characteristics, for each node in the updated knowledge graph data structure, of a neighborhood of that node in the updated knowledge graph data structure; and
generating a visualization output that explains reasoning for entity re-resolution in the updated knowledge graph data structure at least by projecting the embeddings of the characteristics, wherein the visualization output represents proximity of re-resolved entities with regard to temporal characteristics.
10. The method of claim 1 , wherein executing the clustering operation comprises:
determining, based on the set of initial clusters, for each second entity, whether the clustering operation requires creation of a new cluster for the second entity;
in response to determining that none of the second entities require creation of a new cluster, executing the clustering operation on partitions of the second documents distributed across a plurality of first compute nodes; and
in response to a determination that at least one second entity requires creation of a new cluster for the at least one second entity, generating the new cluster for the at least one second entity and executing the clustering operation in a sequential clustering operation by a second compute node.
11. A non-transitory computer-readable medium storing a set of instructions for distributed data processing, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive a knowledge graph data structure comprising data representations of a plurality of first entities specified in a first set of documents, and a corresponding set of initial clusters associated with corresponding ones of the plurality of first entities;
receive at least one dynamic data stream from at least one source computing system, the at least one dynamic data stream comprising second documents having data specifying second entities referenced by the second documents, wherein each second document is a collection of unstructured textual data; and
automatically, in response to receiving the at least one dynamic data stream:
execute a clustering operation on the second documents based on the set of initial clusters, and document features of the second documents, to provide a set of second document clusters comprising the second documents, wherein the clustering operation is a modified Dirichlet Hawkes Process (DHP) that performs distributed clustering, in parallel, on partitions of the at least one dynamic data stream, across a master compute node and a plurality of slave compute nodes of the data processing system;
determine, for each second document cluster in the one or more second document clusters, whether the second document cluster should be modified based on entities associated with the second document cluster;
execute, for each second document cluster that is determined should be modified, a cluster modification operation on the second document cluster, wherein updated clusters are generated comprising a combination of second document clusters that are modified and second document clusters that are not modified;
dynamically execute entity re-resolution on the plurality of first entities in the knowledge graph data structure based on the second entities associated with the updated clusters to generate an updated knowledge graph data structure;
input information associated with the updated knowledge graph data structure into an artificial intelligence computing system and analyzing patterns of the entity re-resolution; and
generate an identity-fraud alert based on analyzing the patterns of the entity re-resolution and based on a determination that an entity is re-resolved a plurality of times over a time period.
12. The non-transitory computer-readable medium of claim 11 , wherein the one or more instructions, cause the device to:
provide the updated knowledge graph data structure to a downstream computer system to perform a downstream computer system operation based on the updated knowledge graph data structure.
13. The non-transitory computer-readable medium of claim 11 , wherein the clustering operation performs clustering based on temporal characteristics associated with the second entities referenced in the second documents of the at least one dynamic data stream.
14. The non-transitory computer-readable medium of claim 13 , wherein the one or more instructions, to cause the device to perform the clustering operation, cause the device to:
perform a distributed DHP clustering when no new clusters are needed as part of the clustering operation; and
perform a non-distributed DHP clustering when a new cluster is determined to be needed as part of the clustering operation.
15. The non-transitory computer-readable medium of claim 11 , wherein the one or more instructions, to cause the device to perform a cluster modification operation, cause the device to merge entities that only occur in a same second document cluster so that a single entity represents the same second document cluster in the updated clusters.
16. The non-transitory computer-readable medium of claim 11 , wherein the one or more instructions, to cause the device to perform cluster modification operation, cause the device to:
determine whether an entity is present in more than one second document cluster; and
in response to the entity being present in more than one second document cluster:
associate the entity with a first one of the second document clusters of the entity; and
generate one or more sub-entities corresponding to the entity, wherein each of the one or more sub-entities is associated with a second one of second document clusters or a newly generated cluster.
17. The non-transitory computer-readable medium of claim 16 , wherein the one or more sub-entities corresponding to the entity are entities corresponding to a smallest sub-cluster of at least one of the more than one second document cluster.
18. The non-transitory computer-readable medium of claim 11 , wherein the one or more instructions cause the device to:
input the updated knowledge graph data structure into a graph neural network that generates embeddings of characteristics, for each node in the updated knowledge graph data structure, of a neighborhood of that node in the updated knowledge graph data structure; and
generate a visualization output that explains reasoning for entity re-resolution in the updated knowledge graph data structure at least by projecting the embeddings of the characteristics, wherein the visualization output represents proximity of re-resolved entities with regard to temporal characteristics.
19. The non-transitory computer-readable medium of claim 11 , wherein the one or more instructions, to cause the device to execute the clustering operation, cause the device to:
determine, based on the set of initial clusters, for each second entity, whether the clustering operation requires creation of a new cluster for the second entity;
in response to determining that none of the second entities require creation of a new cluster, execute the clustering operation on partitions of the second documents distributed across a plurality of first compute nodes; and
in response to a determination that at least one second entity requires creation of a new cluster for the at least one second entity, generate the new cluster for the at least one second entity and execute the clustering operation in a sequential clustering operation by a second compute node.
20. A data processing system, comprising:
one or more processors; and
one or more memory devices coupled to the one or more processors, wherein the one or more processors are configured to:
receive a knowledge graph data structure comprising data representations of a plurality of first entities specified in a first set of documents, and a corresponding set of initial clusters associated with corresponding ones of the plurality of first entities;
receive at least one dynamic data stream from at least one source computing system, the at least one dynamic data stream comprising second documents having data specifying second entities referenced by the second documents, wherein each second document is a collection of unstructured textual data; and
automatically, in response to receiving the at least one dynamic data stream:
execute a clustering operation on the second documents based on the set of initial clusters, and document features of the second documents, to provide a set of second document clusters comprising the second documents, wherein the clustering operation is a modified Dirichlet Hawkes Process (DHP) that performs distributed clustering, in parallel, on partitions of the at least one dynamic data stream, across a master compute node and a plurality of slave compute nodes of the data processing system;
determine, for each second document cluster in the one or more second document clusters, whether the second document cluster should be modified based on entities associated with the second document cluster;
execute, for each second document cluster that is determined should be modified, a cluster modification operation on the second document cluster, wherein updated clusters are generated comprising a combination of second document clusters that are modified and second document clusters that are not modified;
dynamically execute entity re-resolution on the plurality of first entities in the knowledge graph data structure based on the second entities associated with the updated clusters to generate an updated knowledge graph data structure;
input information associated with the updated knowledge graph data structure into an artificial intelligence computing system and analyzing patterns of the entity re-resolution; and
generate an identity-fraud alert based on analyzing the patterns of the entity re-resolution and based on a determination that an entity is re-resolved a plurality of times over a time period.