APPARATUS, SYSTEMS, AND METHODS FOR GROUPING DATA RECORDS
The present application relates to apparatus, systems, and methods for grouping data records based on entities referenced by the data records. The disclosed grouping mechanism can include determining a pair-wise similarity between a large number of data records, and clustering a subset of the data records based on their pair-wise similarity.
1 . An apparatus comprising:
a processor configured to run one or more modules stored in memory, wherein the one or more modules are configured to:
identify at least one pair of data records for which to determine a similarity value;
determine the similarity value for the at least one pair of data records based, at least in part, on a plurality of attributes associated with the at least one pair of data records; and
associate the at least one pair of data records with one or more clusters, each associated with a unique entity, based on the similarity value for the at least one pair of data records.
2 . The apparatus of claim 1 , wherein the one or more modules are further configured to identify one or more pairs of data records for which a similarity value need not be determined based on a predetermined set of attributes that are likely to be shared by related data records.
3 . The apparatus of claim 2 , wherein the one or more modules are configured to adjust the predetermined set of attributes based on association of data records to clusters from a previous iteration.
4 . The apparatus of claim 1 , wherein the one or more modules are configured to determine the similarity value based on a similarity function learned from training data records.
5 . The apparatus of claim 4 , wherein the similarity function is designed to infer an importance of a particular component associated with a particular attribute of a data record, wherein the similarity function is learned by:
determining differences between components associated with the particular attribute of the training data records, wherein the training data records are known belong to the same cluster; and
determining the importance of the particular component based on a number of times the particular component appears in the differences.
6 . The apparatus of claim 4 , wherein the similarity function is designed to infer a likelihood of interchanging a first component in a particular attribute of a data record with a second component, wherein the similarity function is learned by:
determining differences between components associated with the particular attribute of the training data records, wherein the training data records are known belong to the same cluster; and
determining the likelihood of interchanging the first component with the second component based on a number of times the first component and the second component appears in the differences at the same time.
7 . The apparatus of claim 4 , wherein the similarity function is designed to determine a conditional likelihood that a missing attribute of a data record has a particular component, wherein the conditional likelihood is determined by:
determining a combination of known attributes corresponding to a particular entity; determining all variations of a missing attribute amongst data records of the particular entity having the combination of known attributes; and
determining a conditional probability, based on the variations of the missing attribute, that the missing attribute has a particular component given that the data record has the particular combination of known attributes.
8 . The apparatus of claim 1 , wherein the one or more modules are configured to:
represent the plurality of data records as a plurality of nodes in a graph;
represent the similarity value for the at least one pair of data records as at least one edge between nodes, in the graph, corresponding to the at least one pair of data records; and
determine the one or more clusters from based on the graph.
9 . The apparatus of claim 8 , wherein the one or more modules are configured to determine the one or more clusters based on the graph using a graph clustering technique.
10 . The apparatus of claim 1 , wherein the one or more modules are configured to receive a clustering directive requiring the one or more modules to associate two data records with the same cluster.
11 . The apparatus of claim 1 , wherein the one or more modules are configured to
associate at least one of the plurality of data records to one or more clusters using a clustering technique; and
adjust a parameter for the clustering technique for each of the one or more clusters independently, based on data records in the one or more clusters.
12 . The apparatus of claim 1 , wherein the one or more modules are configured to determine the similarity value for the at least one pair of data records by receiving the similarity value for the at least one pair of data records from another computing device.
13 . The apparatus of claim 1 , wherein the one or more modules are configured to:
receive, from a plurality of computing devices, a plurality of sub-clusters independently identified at the plurality of computing devices; and
perform a union-find operation on the plurality of sub-clusters to identify the one or more clusters.
14 . A method for clustering a plurality of data records into at least one cluster, the method comprising:
identifying, at a candidate reduction module in a computing device, at least one pair of the plurality of data records for which to determine a similarity value;
determining, at a similarity computation module residing in the computing device, in communication with the candidate reduction module, the similarity value for the at least one pair based, at least in part, on a plurality of attributes associated with the at least one pair of data records; and
associating, at a clustering computation module residing in the computing device, in communication with the similarity computation module, the at least one pair of data records with one or more clusters, each associated with a unique entity, based on the similarity value for the at least one pair of data records.
15 . The method of claim 14 , further comprising identifying, at the candidate reduction module, one or more pairs of the plurality of data records for which a similarity value need not be determined based on a predetermined set of attributes that are likely to be shared by related data records.
16 . The method of claim 15 , further comprising adjusting, at the candidate reduction module, the predetermined set of attributes based on association of data records to clusters from a previous iteration.
17 . The method of claim 14 , further comprising determining, at the similarity computation module, the similarity value based on a similarity function learned from training data records.
18 . The method of claim 17 , wherein the similarity function is designed to infer an importance of a particular component associated with a particular attribute of a data record.
19 . The method of claim 17 , wherein the similarity function is designed to infer a likelihood of interchanging a first component in a particular attribute of a data record with a second component.
20 . A computer program product, tangibly embodied in a non-transitory computer-readable storage medium, the computer program product including instructions operable to cause a data processing system to:
identify at least one pair of data records for which to determine a similarity value;
determine the similarity value for the at least one pair of data records based, at least in part, on a plurality of attributes associated with the at least one pair of data records; and
associate the at least one pair of data records with one or more clusters, each associated with a unique entity, based on the similarity value for the at least one pair of data records.
21 - 22 . (canceled)