METHOD AND SYSTEM FOR HIGH PERFORMANCE DATA MATCHING AND MATCH SET GROUPING
In one aspect, a computer-implemented method for partitioning records in a data set, wherein each record has one or more data elements includes comparing the records to identify pairs of matching records in the data set, wherein records within each pair of matching records are representative of a common entity. Each matching record is assigned to a unique partition wherein said unique partition contains all matching records and does not overlap with any other partition. Each record is updated with a partition identifier representative of said unique partition. In another aspect, a computer-based information handling system for resolving and partitioning matching records in a data set is provided.
1 . A computer-implemented method for partitioning records in a data set, each record having one or more data elements, the method comprising:
comparing the records to identify pairs of matching records in the data set, wherein records within each pair of matching records are representative of a common entity;
assigning each matching record to a unique partition wherein said unique partition contains all matching records and does not overlap with any other partition; and
updating each record with a partition identifier representative of said unique partition.
2 . The method of claim 1 , further comprising:
for each of said pairs of matching records, storing a match record comprising a record identifier associated with each record in the pair of matching records.
3 . The method of claim 2 , wherein said step of assigning each matching record to a unique partition comprises, for each match record, the steps of:
(a) determining whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assigning an available partition identifier to both of the first one of the record identifiers in the match record and the second one of the record identifiers in the match record.
4 . The method of claim 3 , wherein the available partition identifier is a lowest numeric identifier and is selected from a plurality of available numeric identifiers.
5 . The method of claim 2 , wherein said step of assigning each matching record to a unique partition comprises, for each match record, the steps of:
(a) determining whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assigning a partition identifier to the second one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the first one of the record identifiers in the match record.
6 . The method of claim 2 , wherein said step of assigning each matching record to a unique partition comprises, for each match record, the steps of:
(a) determining whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, assigning a partition identifier to the first one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the second one of the record identifiers in the match record.
7 . The method of claim 2 , wherein said step of assigning each matching record to a unique partition comprises, for each match record, the steps of:
(a) determining whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(c) if the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determining whether the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to the same partition; and
(d) if the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to different partitions, merging the different partitions into a single partition, by setting a partition identifier of a first one of the different partitions to be equal to a partition identifier of a second one of the different partitions.
8 . The method of claim 7 , wherein the partition identifier of the first one of the different partitions and the partition identifier of the second one of the different partitions are numeric identifiers and wherein the step of merging the different partitions into a single partition comprises:
if the first one of the different partitions has a partition identifier that is less than a partition identifier of the second one of the different partitions, assigning the partition identifier of the first one of the different partitions to the second one of the different partitions; and
if the first one of the different partitions has a partition identifier that is greater than a partition identifier of the second one of the different partitions, assigning the partition identifier of the second one of the different partitions to the first one of the different partitions.
9 . The method of claim 1 , wherein the records in each pair of matching records have one or more matching data elements.
10 . The method of claim 1 , wherein the step of comparing the records comprises:
identifying one or more match candidates;
assigning a score to each of the one or more match candidates, the score representative of a likelihoods that the records in the match candidate resolve to a common entity; and
determining whether each score meets a preselected threshold value and, if so, generating a match record.
11 . The method of claim 1 , wherein the step of comparing the records to identify pairs of matching records in the data set is performed without writing any data back to the records.
12 . The method of claim 1 , wherein the records contain data elements representative of entities selected from the group consisting of a business entities, employers, or both.
13 . A computer-based information handling system for resolving and partitioning matching records in a data set, the system comprising:
a processor;
a memory in electronic communication with the processor; and
instructions stored in said memory, the instructions being executable to:
compare the records to identify pairs of matching records in the data set, wherein records within each pair of matching records are representative of a common entity;
assign each matching record to a unique partition wherein said unique partition contains all matching records and does not overlap with any other partition; and
update each record with a partition identifier representative of said unique partition.
14 . The computer-based information handling system of claim 13 , the instructions further executable to, for each of said pairs of matching records, store in the memory a match record comprising a record identifier associated with each record in the pair of matching records.
15 . The computer-based information handling system of claim 14 , the instructions further executable, for each match record, to:
(a) determine whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, determine whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assign an available partition identifier to both of the first one of the record identifiers in the match record and the second one of the record identifiers in the match record.
16 . The computer-based information handling system of claim 15 , wherein the partition identifier is a numeric identifier and the available partition identifier is a lowest available numeric identifier.
17 . The computer-based information handling system of claim 14 , the instructions further executable, for each match record, to:
(a) determine whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determine whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assign a partition identifier to the second one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the first one of the record identifiers in the match record.
18 . The computer-based information handling system of claim 14 , the instructions further executable, for each match record, to:
(a) determine whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, determine whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition; and
(c) if the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, assign a partition identifier to the first one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the second one of the record identifiers in the match record.
19 . The computer-based information handling system of claim 14 , the instructions further executable, for each match record, to:
(a) determine whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if the first one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determine whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(c) if the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determine whether the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to the same partition; and
(d) if the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to different partitions, merge the different partitions into a single partition by setting a partition identifier of a first one of the different partitions to be equal to a partition identifier of a second one of the different partitions.
20 . The computer-based information handling system of claim 19 , wherein the partition identifier of the first one of the different partitions and the partition identifier of the second one of the different partitions are numeric identifiers and wherein the step of merging the different partitions into a single partition comprises:
if the first one of the different partitions has a partition identifier that is less than a partition identifier of the second one of the different partitions, assigning the partition identifier of the first one of the different partitions to the second one of the different partitions; and
if the first one of the different partitions has a partition identifier that is greater than a partition identifier of the second one of the different partitions, assigning the partition identifier of the second one of the different partitions to the first one of the different partitions.
21 . The computer-based information handling system of claim 13 , wherein the records in each pair of matching records have one or more matching data elements.
22 . The computer-based information handling system of claim 13 , wherein the instructions are executable to compare the records by:
identifying one or more match candidates;
assigning a score to each of the one or more match candidates, the score representative of a likelihoods that the records in the match candidate resolve to a common entity; and
determining whether each score meets a preselected threshold value and, if so, generating a match record.
23 . The computer-based information handling system of claim 13 , wherein the instructions are executable to identify pairs of matching records in the data set is performed without modifying the records.
24 . A computer-implemented method for partitioning records in a data set, each record having one or more data elements, the method comprising:
comparing the records to identify pairs of matching records in the data set, wherein records within each pair of matching records are representative of a common entity;
for each of said pairs of matching records, storing a match record comprising a record identifier associated with each record in the pair of matching records;
storing each of said pairs of matching records as a match record comprising a record identifier associated with each record of said pair of matching records;
assigning each matching record to a unique partition wherein said unique partition contains all matching records and does not overlap with any other partition, wherein said step of assigning each matching record to a unique partition comprises, for each match record, the steps of:
(a) determining whether a first one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(b) if it is determined in step (a) that the first one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(c) if it is determined in step (b) that the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assigning an available partition identifier to both of the first one of the record identifiers in the match record and the second one of the record identifiers in the match record;
(d) if it is determined in step (a) that the first one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determining whether a second one of the record identifiers in the match record represents a record that has previously been assigned to a partition;
(e) if it is determined in step (d) that the second one of the record identifiers in the match record does not represent a record that has previously been assigned to a partition, assigning a partition identifier to the second one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the first one of the record identifiers in the match record;
(f) if it is determined in step (b) that the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, assigning a partition identifier to the first one of the record identifiers in the match record which is equal to the partition identifier previously assigned to the second one of the record identifiers in the match record;
(g) if it is determined in step (d) that the second one of the record identifiers in the match record represents a record that has previously been assigned to a partition, determining whether the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to the same partition;
(h) if it is determined in step (g) that the record represented by the first one of the record identifiers and the record represented by the second one of the record identifiers are assigned to different partitions, merging the different partitions into a single partition by setting a partition identifier of a first one of the different partitions to be equal to a partition identifier of a second one of the different partitions.