Systems and methods for entity resolution
Systems and methods for performing entity resolution. In some aspects, the system obtains a plurality of attributes represented in a plurality of records from one or more sources. The system generates a plurality of match vectors based on the plurality of attributes. Each match vector includes a set of attributes that match between a pair of records and a remaining set of attributes that are not required to match between the pair of records. The system processes the plurality of match vectors using an entity resolution model trained to output a binary indicator regarding whether one or more pairs of records for a match vector be merged. The system merges the one or more pairs of records for each match vector of the plurality of match vectors having a corresponding binary indicator output from the entity resolution model that pairs of records for the match vector be merged.
1 . A system for performing entity resolution for a plurality of records, the system comprising:
one or more processors; and
a non-transitory, computer-readable medium comprising instructions that, when executed by the one or more processors, cause operations comprising:
obtaining the plurality of records from one or more sources;
determining a plurality of attributes represented in the plurality of records;
generating a plurality of match vectors based on the plurality of attributes, wherein each match vector includes a set of attributes that match between a pair of records and a remaining set of attributes that are not required to match between the pair of records;
obtaining training data for a subset of pairs of records randomly selected from across the plurality of match vectors, wherein the training data indicates for each pair of records in the subset of pairs whether the pair of records be merged;
training an entity resolution model based on the training data and independent of a blocking key, the subset of pairs of records, and the plurality of match vectors;
determining, for each match vector in the plurality of match vectors, respective pairs of records from the plurality of records that satisfy a respective match vector, wherein the entity resolution model is trained to output a binary indicator regarding whether the respective pairs of records that satisfy the respective match vector be merged; and
merging the respective pairs of records based on the respective pairs of records satisfying the respective match vector and based on a corresponding binary indicator output from the entity resolution model indicating that the respective pairs of records for the respective match vector be merged.
2 . The system of claim 1 , the operations further comprising:
for each match vector in the plurality of match vectors:
determining a count for pairs of records from the plurality of records that satisfy the respective match vector; and
determining a probability that the pairs of records satisfying the respective match vector be merged.
3 . A method for performing entity resolution for a plurality of records, the method comprising:
obtaining a plurality of attributes represented in the plurality of records from one or more sources;
generating a plurality of match vectors based on the plurality of attributes, wherein each match vector includes a set of attributes that match between a pair of records and a remaining set of attributes that are not required to match between the pair of records;
processing, independent of a blocking key, the plurality of match vectors using an entity resolution model trained to output a binary indicator regarding whether one or more pairs of records for a match vector be merged;
determining, for each match vector in the plurality of match vectors, respective pairs of records from the plurality of records that satisfy a respective match vector; and
merging the respective pairs of records based on the respective pairs of records satisfying the respective match vector and based on a corresponding binary indicator output from the entity resolution model indicating that the respective pairs of records for the respective match vector be merged.
4 . The method of claim 3 , further comprising:
for each match vector in the plurality of match vectors:
determining a count for pairs of records from the plurality of records that satisfy the match vector; and
determining a probability that the pairs of records satisfying the match vector be merged.
5 . The method of claim 4 , wherein determining the count for the pairs of records from the plurality of records that satisfy the match vector comprises executing a series of self-join and group-by-count operations expressible in a structured query language.
6 . The method of claim 4 , wherein training the entity resolution model comprises fitting the determined counts to a binary mixture of multinomial distributions.
7 . The method of claim 3 , further comprising:
obtaining training data for a subset of pairs of records randomly selected from across the plurality of match vectors, wherein the training data indicates for each pair of records in the subset of pairs whether the pair of records be merged; and
training the entity resolution model based on the training data, the subset of pairs of records, and the plurality of match vectors, wherein the entity resolution model is trained to output the binary indicator regarding whether pairs of records for the respective match vector be merged.
8 . The method of claim 7 , wherein the entity resolution model comprises an inferred Bayesian prior computed based on the training data for the subset of pairs of records randomly selected from across the plurality of match vectors.
9 . The method of claim 3 , further comprising:
obtaining training data for pairs of records for the plurality of match vectors, wherein the training data indicates whether the pair of records be merged and is computed based on a prior target; and
training the entity resolution model based on the training data and the pairs of records for the plurality of match vectors.
10 . The method of claim 9 , wherein the entity resolution model comprises a decision tree model, an XGBoost model, or a classification model.
11 . The method of claim 3 , wherein the method is executed independent of the blocking key.
12 . A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause operations comprising:
obtaining a plurality of attributes represented in a plurality of records from one or more sources;
generating a plurality of match vectors based on the plurality of attributes, wherein each match vector includes a set of attributes that match between a pair of records and a remaining set of attributes that are not required to match between the pair of records;
obtaining training data for pairs of records selected from across the plurality of match vectors, wherein the training data indicates for each pair of records whether the pair of records be merged;
training an entity resolution model, that is independent of a blocking key, based on the training data, the pairs of records, and the plurality of match vectors; and
determining, for each match vector in the plurality of match vectors, respective pairs of records from the plurality of records that satisfy a respective match vector, wherein the entity resolution model is trained to output a binary indicator regarding the respective pairs of records that satisfy the respective match vector be merged.
13 . The non-transitory, computer-readable medium of claim 12 , further comprising:
merging the respective pairs of records for each match vector of the plurality of match vectors having a corresponding binary indicator output from the entity resolution model that the respective pairs of records that satisfy the respective match vector be merged.
14 . The non-transitory, computer-readable medium of claim 12 , further comprising:
for each match vector in the plurality of match vectors:
determining a count for pairs of records from the plurality of records that satisfy the respective match vector; and
determining a probability that the pairs of records satisfying the respective match vector be merged.
15 . The non-transitory, computer-readable medium of claim 14 , wherein determining the count for the pairs of records from the plurality of records that satisfy the respective match vector comprises executing a series of self-join and group-by-count operations expressible in a structured query language.
16 . The non-transitory, computer-readable medium of claim 14 , wherein training the entity resolution model comprises fitting the determined counts to a binary mixture of multinomial distributions.
17 . The non-transitory, computer-readable medium of claim 12 , wherein obtaining the training data for the pairs of records selected from across the plurality of match vectors comprises obtaining the training data for a subset of the pairs of records randomly selected from across the plurality of match vectors, wherein the training data indicates for each pair of records in the subset of the pairs whether the pair of records be merged.
18 . The non-transitory, computer-readable medium of claim 17 , wherein the entity resolution model comprises an inferred Bayesian prior computed based on the training data for the subset of the pairs of records randomly selected from across the plurality of match vectors.
19 . The non-transitory, computer-readable medium of claim 12 , wherein obtaining the training data for the pairs of records selected from across the plurality of match vectors comprises obtaining the training data for the pairs of records for the plurality of match vectors, wherein the training data indicates whether the pair of records be merged and is computed based on a prior target.
20 . The non-transitory, computer-readable medium of claim 19 , wherein the entity resolution model comprises a decision tree model, an XGBoost model, or a classification model.