Batch automated blocking and record matching
View Patent ↗Batch, or “offline”, blocking takes a set of records and generates sets (or blocks, hence the name blocking) of potentially matching records for the entire set. The blocks of potential matches are then passed to a matching process to evaluate which records match. Applications include but are not limited to individual matching such as student identification, householding, business matching, supply chain matching, financial matching, news or text matching, and other applications.
1. A non-transitory computer readable medium storing a computer program product, the computer program product comprising computer program code configured to process a bag of sets (Q) that produces a set of sets (R), the computer program code configured to perform a process, comprising:
constructing a tree data structure that has a one-to-one correspondence between nodes on every path between a root of the tree data structure and a leaf of the tree data structure and elements of a set in (R);
when two distinct sets t and u, both in (R), contain identical prefixes, mapping one path in the tree data structure, starting at the root of the tree data structure, to the prefix; and
eliminating a subset by checking whether the set u is a subset of the set t by storing each element of the set t in a hash and checking whether every element in the set u is in the set t, wherein
for every unique set (s) in (Q), when (Q) does not contain a superset of (s), then (R) contains (s), and, when (Q) does contain a superset of (s), then R does not contain (s).
2. The non-transitory computer readable medium of claim 1 , wherein the construction of (R) comprises adding sets to the tree data structure in non-increasing order of the size of the sets.
3. The non-transitory computer readable medium of claim 2 , wherein adding a set (s) to the three data structure comprises adding (s) to the three data structure when (s) is not the subset of some set in the tree data structure.
4. The non-transitory computer readable medium of claim 1 , wherein the construction of (R) comprises adding sets to the tree data structure in non-decreasing order of the size of the sets.
5. The non-transitory computer readable medium of claim 4 , wherein adding a set (s) to the tree data structure comprises:
when (s) is a superset of a set of sets (U) in the tree data structure, removing sets (U) from the tree data structure; and
adding (s) to the tree data structure.
6. The non-transitory computer readable medium of claim 1 , wherein elements of a set of sets (V) in (Q) are represented by totally ordered values, where identical elements are represented by identical values, and distinct elements are represented by distinct values.
7. The non-transitory computer readable medium of claim 2 , wherein the elements of each set (V) in (Q) are sorted in increasing order.
8. The non-transitory computer readable medium of claim 2 , wherein the elements of each set (V) in (Q) are sorted in decreasing order.