Spectral neighborhood blocking for entity resolution
A processing device of an information processing system is operative to obtain a plurality of records, documents, web pages or other data objects, and to construct a binary tree using a bipartition procedure in which subsets of the data objects are associated with respective nodes of the tree. Evaluation of a designated modularity for a given one of the nodes of the tree is used as a stopping criterion to prevent further partitioning of that node and to indicate designation of that node as a leaf node of the tree. The resulting leaf nodes of the tree provide a non-overlapping partitioning of the plurality of data objects. The processing device is further operative to perform a neighborhood search on the tree to identify pairs of the plurality of data objects that match the same entity, and to store an indication of the matching pairs of data objects.
1. An apparatus comprising:
a processing device comprising a processor having an associated memory;
wherein the processing device is operative:
to obtain a plurality of data objects;
to construct a binary tree using a bipartition procedure in which subsets of the data objects are associated with respective nodes of the tree, wherein evaluation of a designated modularity for a given one of the nodes of the tree is used as a stopping criterion to prevent further partitioning of that node and to indicate designation of that node as a leaf node of the tree, and wherein resulting leaf nodes of the tree provide a non-overlapping partitioning of the plurality of data objects;
to perform a neighborhood search on the tree to identify pairs of the plurality of data objects that match a same entity based on at least one pairwise similarity metric; and
to store an indication of said identified pairs of matching data objects in the associated memory; and
wherein the neighborhood search utilizes a first similarity threshold to compare a pair of data objects that are each associated with a same leaf node of the binary tree and utilizes a second similarity threshold different than the first similarity threshold to compare another pair of data objects that are each respectively associated with a different leaf node of the binary tree.
2. The apparatus of claim 1 wherein the data objects comprise records.
3. The apparatus of claim 2 wherein the bipartition procedure comprises computing a designated singular vector of a matrix C as C=D −1/2 B, where B denotes an n×m matrix having a corresponding record-record similarity matrix given by A=BB T , where D=diag(B(B T 1)), and where diag(•) transforms a vector into a diagonal matrix, and further wherein the singular vector of C corresponds to a designated eigenvector of Laplacian matrix (A), such that the bipartition procedure is performed without requiring computation of the record-record similarity matrix A.
4. The apparatus of claim 3 wherein the singular vector comprises a second maximum singular vector of the matrix C and corresponds to a second smallest eigenvector of the Laplacian matrix (A).
5. The apparatus of claim 3 wherein the bipartition procedure assigns the records associated with the given one of the nodes to one of two subsets according to signs of corresponding entries in the singular vector.
6. The apparatus of claim 1 wherein the designated modularity for the given node comprises a Newman-Girvan modularity.
7. The apparatus of claim 1 wherein each of the data objects in a set of data objects comprises one or more substrings from a set of designated substrings and similarity between pairs of the data objects is determined based on particular numbers of each substring that appear in respective ones of the data objects.
8. The apparatus of claim 1 wherein the associated memory comprises a memory internal to the processing device.
9. The apparatus of claim 1 wherein the processing device comprises an entity resolution module having a tree generation module coupled to a neighborhood search module, the tree generation module comprising a bipartition component and a modularity computation component and being configured for constructing the binary tree.
10. An integrated circuit comprising the apparatus of claim 1 .
11. A processor-implemented method comprising:
obtaining a plurality of data objects;
constructing a binary tree using a bipartition procedure in which subsets of the data objects are associated with respective nodes of the tree, wherein evaluation of a designated modularity for a given one of the nodes of the tree is used as a stopping criterion to prevent further partitioning of that node and to indicate designation of that node as a leaf node of the tree, and wherein resulting leaf nodes of the tree provide a non-overlapping partitioning of the plurality of data objects;
performing a neighborhood search on the tree to identify pairs of the plurality of data objects that match a same entity based on at least one pairwise similarity metric; and
storing an indication of said matching pairs of data objects;
wherein the neighborhood search utilizes a first similarity threshold to compare a pair of data objects that are each associated with a same leaf node of the binary tree and utilizes a second similarity threshold different than the first similarity threshold to compare another pair of data objects that are each respectively associated with a different leaf node of the binary tree.
12. The method of claim 11 wherein the data objects comprise records.
13. The method of claim 12 wherein the bipartition procedure comprises computing a designated singular vector of a matrix C as C=D −1/2 B, where B denotes an n×m matrix having a corresponding record-record similarity matrix given by A=BB T , where D=diag(B(B T 1)), and where diag(•) transforms a vector into a diagonal matrix, and further wherein the singular vector of C corresponds to a designated eigenvector of Laplacian matrix (A), such that the bipartition procedure is performed without requiring computation of the record-record similarity matrix A.
14. The method of claim 13 wherein the singular vector comprises a second maximum singular vector of the matrix C and corresponds to a second smallest eigenvector of the Laplacian matrix (A).
15. The method of claim 13 wherein the bipartition procedure assigns the records associated with the given one of the nodes to one of two subsets according to signs of corresponding entries in the singular vector.
16. The method of claim 11 wherein the designated modularity for the given node comprises a Newman-Girvan modularity.
17. The method of claim 11 wherein each of the data objects in a set of data objects comprises one or more substrings from a set of designated substrings and similarity between pairs of the data objects is determined based on particular numbers of each substring that appear in respective ones of the data objects.
18. An article of manufacture comprising a non-transitory computer-readable storage medium having embodied therein executable program code that when executed by a processor of a processing device causes the device to perform the steps of the method of claim 11 .