System and method for associating related records to common entities across multiple lists
Computer implemented systems and methods are disclosed for associating records across lists, wherein the lists include a plurality of records and the plurality of records is associated with a respective entity. In accordance with some embodiments, the systems and methods further comprise grouping one or more records from a first list into a first group based on fields of the records in the first list, grouping one or more records from a second list into a second group based on fields of the records in the second list, pairing a record from the first group with a record from the second group, assessing each pair of records based on an evaluation of the respective pair according to fields of the pair, and associating records from the first group and records of the second group with an entity based on the assessment.
1. A system for associating records across a first list and a second list to a common entity, the system comprising:
one or more memory devices configured to store:
a set of instructions; and
a first list and a second list, wherein the first list and the second list both include a plurality of records associated with entities and the plurality of records includes one or more fields; and
one or more processors configured to execute the set of instructions and cause the system to:
obtain a plurality of record pairs, wherein each record pair of the plurality of record pairs includes a record from the first list and a record from the second list;
label a subset of record pairs in the plurality of record pairs;
generate a statistical model based on the labeled subset of record pairs;
evaluate the plurality of record pairs based on the statistical model, the evaluation based on field lengths or information external to at least one of the plurality of record pairs associated with the one or more fields;
output, based on the statistical model, scores representing probabilities of records in the plurality of record pairs being related to similar entities; and
associate one or more of the plurality of records of the first list and one or more of the plurality of records of the second list with the similar entities.
2. The system of claim 1 , wherein the set of instructions, when executed, further causes the system to filter record pairs of the plurality of record pairs having scores above a threshold score.
3. The system of claim 2 , wherein the threshold score is determined based on the evaluation of the record pairs of the plurality of record pairs.
4. The system of claim 1 , wherein the set of instructions, when executed, further causes the system to update the statistical model based on the output of the evaluated record pairs.
5. The system of claim 1 , wherein the information external to the plurality of record pairs includes respective geographic locations, and the evaluation is based on a calculated distance between the respective geographic locations.
6. The system of claim 1 , wherein the labeled subset of record pairs includes matching labels and non-matching labels.
7. The system of claim 1 , wherein a label for a record pair is determined based on highest combined evaluations or lowest combined evaluations of a specified field of the plurality of records.
8. A method for associating records across a first list and a second list to a common entity, the method comprising:
obtaining a first list and a second list, wherein the first list and the second list both include a plurality of records associated with entities and the plurality of records includes one or more fields;
obtaining a plurality of record pairs wherein each record pair of the plurality of record pairs includes a record from the first list and a record from the second list;
labeling a subset of record pairs in the plurality of record pairs;
generating a statistical model based on the labeled subset of record pairs;
evaluating of the plurality of record pairs based on the statistical model, the evaluating based on field lengths or information external to at least one of the plurality of record pairs associated with the one or more fields;
outputting, based on the statistical model, scores representing probabilities of records in the plurality of record pairs being related to similar entities; and
associating one or more of the records of the first list and one or more of the records of the second list with the similar entities.
9. The method of claim 8 , further comprising filtering record pairs having scores above a threshold score.
10. The method of claim 9 , wherein the threshold score is determined based on the evaluation of the record pairs of the plurality of record pairs.
11. The method of claim 8 , further comprising updating the statistical model based on the output of the evaluated record pairs.
12. The method of claim 8 , wherein the labeled subset of record pairs includes matching labels and non-matching labels.
13. The method of claim 8 , wherein a label for a record pair is determined based on highest combined evaluations or lowest combined evaluations of a specified field of the plurality of records.
14. The method of claim 8 , wherein the information external to the plurality of record pairs includes respective geographic locations, and the evaluation is based on a calculated distance between the respective geographic locations.
15. A non-transitory computer-readable medium of a computing system storing a set of instructions that, when executed by one or more processors of the computing system, cause the computing system to perform a method to associate related records across a first list and a second list to a common entity, the method comprising:
obtaining a first list and a second list, wherein the first list and the second list both include a plurality of records associated with entities and the plurality of records includes one or more fields;
obtaining a plurality of record pairs, wherein each record pair of the plurality of record pairs includes a record from the first list and a record from the second list;
labeling a subset of record pairs in the plurality of record pairs;
generating a statistical model based on the labeled subset of record pairs;
evaluating the plurality of record pairs based on the statistical model, the evaluating based on field lengths or information external to at least one of the plurality of record pairs associated with the one or more fields;
outputting, based on the statistical model, scores representing probabilities of records in the plurality of record pairs being related to similar entities; and
associating one or more of the records of the first list and one or more of the records of the second list with the similar entities.
16. The non-transitory computer readable medium of claim 15 , wherein the set of instructions, when executed, further causes the computing system to perform filtering record pairs having scores above a threshold score, wherein the threshold score is determined based on the evaluation of the record pairs of the plurality of record pairs.
17. The non-transitory computer readable medium of claim 15 , wherein the set of instructions, when executed, further causes the computing system to update the statistical model based on the output of the evaluated record pairs.
18. The non-transitory computer readable medium of claim 15 , wherein the information external to the plurality of record pairs includes respective geographic locations, and the evaluation is based on a calculated distance between the respective geographic locations.
19. The non-transitory computer readable medium of claim 15 , wherein the labeled subset of record pairs includes matching labels and non-matching labels.
20. The non-transitory computer readable medium of claim 15 , wherein a label for a record pair is determined based on highest combined evaluations or lowest combined evaluations of a specified field of the plurality of records.