Computer-based systems configured for efficient entity resolution for database merging and reconciliation
To facilitate efficient entity resolution, systems and methods include a first dataset is received from a first database associated with a first entity and a second dataset is received from a second database associated with a second entity. A geo-grid is mapped to a geographical area covering the first entity data records and the second entity data records. A grid matching area in the geo-grid is generated for each first entity data record based on latitude data and longitude data of each first entity data record. Candidate matching records are determined from the second entity data records based on respective grid matching areas. Actual matching records are determined from the candidate matching records based on a threshold for a trigram similarity between each candidate matching record and the respective first entity data record. The actual matching records are associated with the respective first entity data record in the first database.
1. A method comprising:
determining, by at least one processor, within a geo-grid of a geographical area, a first geo-grid space associated with an item location of an item;
wherein the geo-grid comprises a plurality of grid spaces representing a plurality of geographical sub-areas within the geographical area;
determining, by the at least one processor, for each data record of a plurality of data records, a second geo-grid space based on the geo-grid; and
outputting, by the at least one processor, at least one matching data record associated with the item from the plurality of data records based at least in part on a proximity between each second geo-grid space and the first geo-grid space.
2. The method of claim 1 , further comprising:
removing, by the at least one processor, non-matching records from the plurality of data records based on a matching distance threshold.
3. The method of claim 2 , wherein the matching distance threshold comprises about three miles.
4. The method of claim 1 , further comprising:
receiving, by the at least one processor, a ground-truth dataset of matching entity records;
wherein the matching entity records comprise respective distances between each respective pair of candidate matching records; and
determining, by the at least one processor, a predetermined distance threshold based on a percentile threshold of the respective distances.
5. The method of claim 4 , wherein the percentile threshold comprises approximately a ninety seventh percentile distance of the respective distances.
6. The method of claim 1 , further comprising:
determining, by the at least one processor, a longitudinal height of each grid space by converting a predetermined distance threshold into longitudinal displacement; and
determining, by the at least one processor, a latitudinal width of each grid space by converting the predetermined distance threshold into latitudinal displacement.
7. The method of claim 1 , further comprising causing to display, by the at least one processor, a match merge selection on a screen of a computing device to enable a user to select to merge the item into at least one matching data record.
8. The method of claim 1 , further comprising instructing, by the at least one processor, a computing device to display, via a graphical user interface, search results comprising the at least one matching data record.
9. The method of claim 1 , further comprising:
determining, by the at least one processor, a similarity measure quantifying a similarity between the at least one matching data record and the item based at least in part on:
at least one data record attribute of the at least one matching data record, and
at least one item attribute associated with the item.
10. The method of claim 1 , wherein the item comprises at least one entity presence.
11. A system comprising:
at least one processor in communication with a non-transitory computer readable medium having software instructions stored thereon, wherein, upon execution of the software instructions, the at least one processor is configured to:
determine, within a geo-grid of a geographical area, a first geo-grid space associated with an item location of an item;
wherein the geo-grid comprises a plurality of grid spaces representing a plurality of geographical sub-areas within the geographical area;
determine for each data record of a plurality of data records, a second geo-grid space based on the geo-grid; and
output at least one matching data record associated with the item from the plurality of data records based at least in part on a proximity between each second geo-grid space and the first geo-grid space.
12. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
remove non-matching records from the plurality of data records based on a matching distance threshold.
13. The system of claim 12 , wherein the matching distance threshold comprises about three miles.
14. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
receive a ground-truth dataset of matching entity records;
wherein the matching entity records comprise respective distances between each respective pair of candidate matching records; and
determine a predetermined distance threshold based on a percentile threshold of the respective distances.
15. The system of claim 14 , wherein the percentile threshold comprises approximately a ninety seventh percentile distance of the respective distances.
16. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
determine a longitudinal height of each grid space by converting a predetermined distance threshold into longitudinal displacement; and
determine a latitudinal width of each grid space by converting the predetermined distance threshold into latitudinal displacement.
17. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to causing to display, by the at least one processor, a match merge selection on a screen of a computing device to enable a user to select to merge the item into at least one matching data record.
18. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to instruct a computing device to display, via a graphical user interface, search results comprising the at least one matching data record.
19. The system of claim 11 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
determine a similarity measure quantifying a similarity between the at least one matching data record and the item based at least in part on:
at least one data record attribute of the at least one matching data record, and
at least one item attribute associated with the item.
20. The system of claim 11 , wherein the item comprises at least one entity presence.