Automatic entity resolution with rules detection and generation system
Entity resolution (i.e., record linkage) involves the analysis/discovering of datasets that refer to the same real world entity. Analysis typically involves transformation and comparison of different fields of the dataset followed by the application of often domain/data specific logic for determining datasets that refer to the same real world entity (e.g., person). Consider, a bulk mailing of product catalogs to potential customers. Some individuals may have numerous public records that identify the individual differently. Illustratively, several records associated with Jane Doe at her current home address may exist: one record with her name listed as J. Doe, a second record as Jane H. Doe, a third record as Doe, Jane, and a fourth record as Jan Doe (a misspelling). Conceivably, the bulk mailing could unwittingly send multiple catalogs to Jane Doe at her current address, one for each name variation. The entity resolution process described herein can overcome such problems.
1. A method for a system to detect whether records belong to a same entity or a different entity, the method comprising:
in a training stage:
receiving, as input, a plurality of records known to be associated with a same entity, each record including a plurality of fields containing data about the same entity;
defining multiple link features by, for each link feature of the multiple link features, pairing a field in one of the records with a field in another of the records to obtain paired fields and associating with the paired fields a similarity metric used to determine an amount of similarity between the data in those paired field;
applying the similarity metric of the at least one link feature to the data in the paired fields to produce a link feature value for each link feature;
generating a linkage data instance comprised of the multiple defined link features that will be used, in a subsequent usage stage, to determine whether records of unknown association are associated with the same entity; and
in a usage stage:
applying the link features to two records of unknown association to produce a linkage data instance;
using the linkage data instance to determine whether the two records are associated with a same individual; and
outputting an indication as to whether the two records of unknown association are associated with the same individual.
2. The method of claim 1 , further comprising reducing a number of link features in a link feature set.
3. The method of claim 1 , further comprising saving the linkage data instance as a template for determining whether records of unknown association belong to a new same entity.
4. The method of claim 1 , further comprising receiving, as input, a plurality of records known to be associated with different entities, each record including a plurality of fields containing data, wherein the linkage data instance includes one or more defined link features that are used to determine whether records of unknown association belong to different entities.
5. The method of claim 4 , further comprising determining whether the two records of unknown association are associated with the same entity based on whether the application of the linkage data instance produced by applying the link features to the two records of unknown association produces one or more feature link values which can be classified by a linkage model.
6. The method of claim 1 , wherein the similarity metric of a given link feature evaluates phonetic similarity between the data in the fields of that given link feature.
7. The method of claim 1 , wherein the similarity metric of a given link feature evaluates one or more of a syntactic, numerical and color similarity between the data in the fields of that given link feature.
8. The method of claim 1 , further comprising:
providing a description of certain types of errors and variations that may appear in records that are associated with the same individual; and
applying the description to determine the similarity metric used to generate one or more of the at least one link feature.
9. A non-transitory computer readable information storage media having stored thereon instructions that when executed by one or more computers cause the one or more computers to perform a method comprising:
in a training stage:
receiving, as input, a plurality of records known to be associated with a same entity, each record including a plurality of fields containing data about the same entity;
defining multiple link features by, for each link feature of the multiple link features, pairing a field in one of the records with a field in another of the records to obtain paired fields and associating with the paired fields a similarity metric used to determine an amount of similarity between the data in those paired fields;
applying the similarity metric of the at least one link feature to the data in the paired fields to produce a link feature value for each link feature; and
generating a linkage data instance comprised of the multiple defined link features that will be used, in a subsequent usage stage, to determine whether records of unknown association are associated with the same entity;
in a usage stage:
applying the link features to two records of unknown association to produce a linkage data instance;
using the linkage data instance to determine whether the two records are associated with a same individual; and
outputting an indication as to whether the two records of unknown association are associated with the same individual.
10. The non-transitory computer readable information storage media of claim 9 , further comprising reducing a number of link features in a link feature set.
11. The non-transitory computer readable information storage media of claim 9 , further comprising saving a link feature set as a template used to determine whether records of unknown association belong to a new same individual.
12. The non-transitory computer readable information storage media of claim 9 , further comprising receiving, as input, a plurality of records known to be associated with different entities, each record including a plurality of fields containing data, wherein a link feature set includes one or more defined link features that are used to determine whether records of unknown association belong to different entities.
13. The non-transitory computer readable information storage media of claim 9 , wherein the similarity metric of a given link feature evaluates phonetic similarity between the data in the fields of that given link feature.
14. The non-transitory computer readable information storage media of claim 9 , wherein the similarity metric of a given link feature evaluates one or more of a syntactic, numerical and color similarity between the data in the fields of that given link feature.
15. The non-transitory computer readable information storage media of claim 9 , further comprising computer readable program code that, if executed, determines whether the two records of unknown association are associated with the same entity based on whether the application of a link feature set to the two records of unknown association produces one or more feature link values that exceed a predefined threshold or satisfy a statistical evaluation.
16. The non-transitory computer readable information storage media of claim 9 , further comprising:
accessing a description of certain types of errors and variations that may appear in records that are associated with the same individual; and
applying the description to determine the similarity metric used to generate one or more of the at least one link feature.
17. An entity resolution system comprising:
in I/O interface coupled to memory or storage that receives, as input, a plurality of records known to be associated with a same individual, each record including a plurality of fields containing data about the individual;
a linking and pairing subsystem that defines multiple link features by, for each link feature of the multiple link features, pairing a field in one of the records with a field in another of the records to obtain paired fields and associating with the paired fields a similarity metric used to determine an amount of similarity between the data in those paired fields;
a link feature value assignor that applies the similarity metric of the at least one link feature to the data in the paired fields to produce a link feature value for each link feature;
a similarity determiner that generates a linkage data instance comprised of the multiple defined link features that will be used, in a subsequent usage stage, to determine whether records of unknown association are associated with the same individual wherein in the usage stage the link features are applied to two records of unknown association to produce a linkage data instance and the linkage data instance is used to determine whether the two records are associated with a same individual; and
an output device that outputs an indication as to whether the two records of unknown association are associated with the same individual.
18. The entity resolution system of claim 17 , further comprising a dimension reducer that reduces a number of link features in the link feature set.
19. The entity resolution system of claim 17 , wherein the link feature set is saved as a template for determining whether records of unknown association belong to a new same individual.
20. The entity resolution system of claim 17 , wherein the I/O interface further receives, as input, a plurality of records known to be associated with different individuals, each record including a plurality of fields containing data, wherein the link feature set includes one or more defined link features that are used to determine whether records of unknown association belong to different individuals.
21. The entity resolution system of claim 20 , wherein the similarity determiner further determines whether the two records of unknown association are associated with the same individual based on whether the application of the link feature set to the two records of unknown association produces one or more feature link values which can be classified by a linkage model.
22. The entity resolution system of claim 17 , wherein the similarity metric of a given link feature evaluates phonetic similarity between the data in the fields of that given link feature.
23. The entity resolution system of claim 17 , wherein the similarity metric of a given link feature evaluates one or more of a syntactic, numerical and color similarity between the data in the fields of that given link feature.
24. The entity resolution system of claim 17 , further comprising an output device that provides a description of certain types of errors and variations that may appear in records that are associated with the same individual and applies the description to determine the similarity metric used to generate one or more of the at least one link feature.