Computer-based systems configured for efficient entity resolution for database merging and reconciliation
To facilitate efficient entity resolution, systems and methods include a first dataset is received from a first database associated with a first entity and a second dataset is received from a second database associated with a second entity. A geo-grid is mapped to a geographical area covering the first entity data records and the second entity data records. A grid matching area in the geo-grid is generated for each first entity data record based on latitude data and longitude data of each first entity data record. Candidate matching records are determined from the second entity data records based on respective grid matching areas. Actual matching records are determined from the candidate matching records based on a threshold for a trigram similarity between each candidate matching record and the respective first entity data record. The actual matching records are associated with the respective first entity data record in the first database.
1. A method comprising:
receiving, by at least one processor from a computing device associated with a user, at least one entity record request comprising entity search data;
wherein the entity search data comprises an entity location;
determining, by the at least one processor, a first geo-grid space in a geo-grid, the first geo-grid space being associated with the at least one entity record request based on the entity search data:
wherein the geo-grid comprises a plurality of grid spaces representing a plurality of geographical sub-areas;
wherein each grid space of the plurality of grid spaces is based on a predetermined distance threshold;
determining, by the at least one processor, for each second data record of a plurality of second data records, a second geo-grid space based on:
the geo-grid and
a corresponding location identifier of each second data record;
determining, by the at least one processor, a plurality of candidate matching second data records from the plurality of second data records, the plurality of candidate matching second data records being determined based at least in part on:
the second geo-grid space associated with each candidate matching second data record of the plurality of candidate matching second data records and
the first geo-grid space;
instructing, by the at least one processor, the computing device to display, via a graphical user interface, search results in response to the at least one entity record request;
wherein the search results comprise at least one candidate matching second data record of the plurality of candidate matching second data records.
2. The method of claim 1 , further comprising:
determining, by the at least one processor, a distance between each candidate matching second data record of the plurality of candidate matching second data records and the entity search data based at least on the entity location and the second location identifier associated with each candidate matching second data records; and
removing, by the at least one processor, non-matching records from the plurality of candidate matching second data records based on a matching distance threshold.
3. The method of claim 2 , wherein the matching distance threshold comprises about three miles.
4. The method of claim 1 , wherein the predetermined distance threshold comprises about three miles.
5. The method of claim 1 , further comprising:
receiving, by the at least one processor, a ground-truth dataset of matching entity records;
wherein the matching entity records comprise respective distances between each respective pair of candidate matching records; and
determining, by the at least one processor, the predetermined distance threshold based on a percentile threshold of the respective distances.
6. The method of claim 5 , wherein the percentile threshold comprises approximately a ninety seventh percentile distance of the respective distances.
7. The method of claim 1 , further comprising:
determining, by the at least one processor, a longitudinal height of each grid space by converting the predetermined distance threshold into longitudinal displacement; and
determining, by the at least one processor, a latitudinal width of each grid space by converting the predetermined distance threshold into latitudinal displacement.
8. The method of claim 1 , further comprising causing to display, by the at least one processor, a match merge selection on a screen of the computing device to enable the user to select to merge the entity search data into at least one candidate matching second data record of the plurality of candidate matching second data records.
9. A method comprising:
receiving, by at least one processor from a computing device associated with a user, at least one entity record request comprising entity search data;
wherein the entity search data comprises:
an entity location, and
an entity name;
determining, by the at least one processor, a first geo-grid space in a geo-grid, the first geo-grid space being associated with the at least one entity record request based on the entity search data:
wherein the geo-grid comprises a plurality of grid spaces representing a plurality of geographical sub-areas;
wherein each grid space of the plurality of grid spaces is based on a predetermined distance threshold;
determining, by the at least one processor, a plurality of candidate matching second data records from the plurality of second data records, the plurality of candidate matching second data records being determined based at least in part on:
a second geo-grid space associated with a corresponding location identifier of each candidate matching second data record of the plurality of candidate matching second data records and
the first geo-grid space;
determining, by the at least one processor, a ranked list of search results comprising the plurality of candidate matching second data records ranked based at least in part on:
a distance similarity according to a distance between corresponding location identifier of each candidate matching second data record and the entity location of the entity search data, and
a name similarity according to a similarity measure between a corresponding second entity name of each candidate matching second data record and the entity name of the entity search data; and
instructing, by the at least one processor, the computing device to display, via a graphical user interface, the ranked list of search results in response to the at least one entity record request.
10. The method of claim 9 , further comprising:
determining, by the at least one processor, a distance between each candidate matching second data record of the plurality of candidate matching second data records and the entity search data based at least on the entity location and the second location identifier associated with each candidate matching second data records; and
removing, by the at least one processor, non-matching records from the plurality of candidate matching second data records based on a matching distance threshold.
11. The method of claim 10 , wherein the matching distance threshold comprises about three miles.
12. The method of claim 9 , wherein the predetermined distance threshold comprises about three miles.
13. The method of claim 9 , further comprising:
receiving, by the at least one processor, a ground-truth dataset of matching entity records;
wherein the matching entity records comprise respective distances between each respective pair of candidate matching records; and
determining, by the at least one processor, the predetermined distance threshold based on a percentile threshold of the respective distances.
14. The method of claim 13 , wherein the percentile threshold comprises approximately a ninety seventh percentile distance of the respective distances.
15. The method of claim 9 , further comprising:
determining, by the at least one processor, a longitudinal height of each grid space by converting the predetermined distance threshold into longitudinal displacement; and
determining, by the at least one processor, a latitudinal width of each grid space by converting the predetermined distance threshold into latitudinal displacement.
16. The method of claim 9 , further comprising causing to display, by the at least one processor, a match merge selection on a screen of the computing device to enable the user to select to merge the entity search data into at least one candidate matching second data record of the plurality of candidate matching second data records.
17. A system comprising:
a second entity record database storing a plurality of second entity data records associated with a plurality of second entities;
a non-transitory computer readable medium storing:
at least one geo-grid associated with at least one geographic area, and software instructions;
at least one processor in communication with the non-transitory computer readable medium, wherein, upon execution of the software instructions, the at least one processor is configured to:
receive, from a computing device associated with a user, at least one entity record request comprising entity search data;
wherein the entity search data comprises an entity location;
determine a first geo-grid space associated with the at least one entity record request based on the entity search data:
wherein the geo-grid comprises a plurality of grid spaces representing a plurality of geographical sub-areas;
wherein each grid space of the plurality of grid spaces is based on a predetermined distance threshold;
determine a plurality of candidate matching second data records from the plurality of second data records, the plurality of candidate matching second data records being determined based at least in part on:
a second geo-grid space associated with a corresponding location identifier of each candidate matching second data record of the plurality of candidate matching second data records and
the first geo-grid space;
instruct the computing device to display, via a graphical user interface, search results in response to the at least one entity record request;
wherein the search results comprise at least one candidate matching second data record of the plurality of candidate matching second data records.
18. The system of claim 17 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
determine a distance between each candidate matching second data record of the plurality of candidate matching second data records and the entity search data based at least on the entity location and the second location identifier associated with each candidate matching second data records; and
remove non-matching records from the plurality of candidate matching second data records based on a matching distance threshold.
19. The system of claim 17 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
receive a ground-truth dataset of matching entity records;
wherein the matching entity records comprise respective distances between each respective pair of candidate matching records; and
determine the predetermined distance threshold based on a percentile threshold of the respective distances.
20. The system of claim 17 , wherein, upon execution of the software instructions, the at least one processor is further configured to:
cause to display a match merge selection on a screen of the computing device to enable the user to select to merge the entity search data into at least one candidate matching second data record of the plurality of candidate matching second data records.