Automatic entity resolution with rules detection and generation system
Entity resolution (i.e., record linkage) involves the analysis/discovering of datasets that refer to the same real world entity. Analysis typically involves transformation and comparison of different fields of the dataset followed by the application of often domain/data specific logic for determining datasets that refer to the same real world entity (e.g., person). Consider, a bulk mailing of product catalogs to potential customers. Some individuals may have numerous public records that identify the individual differently. Illustratively, several records associated with Jane Doe at her current home address may exist: one record with her name listed as J. Doe, a second record as Jane H. Doe, a third record as Doe, Jane, and a fourth record as Jan Doe (a misspelling). Conceivably, the bulk mailing could unwittingly send multiple catalogs to Jane Doe at her current address, one for each name variation. The entity resolution process described herein can overcome such problems.
1. A challenging training sample selection method comprising:
consideration of record fields as independent;
selection of challenging training samples based on one or more of: a number of multiple labels/classes that record field values, and link feature value ambiguity;
proportional selection of challenging training samples based on one or more of; a level of the labels/classes that record field values, and link feature values ambiguity;
determining a contribution of each linkage feature to the training sample by its relative level of ambiguity;
determining a contribution of a linkage feature value within a linkage feature by its relative level of ambiguity;
determining a contribution of value pairs associated with the same linkage feature by its relative level of ambiguity; and
use of the labels/classes that record field values from positive labels to optimize the determination of one or more of: record field values, and link feature value ambiguity.
2. The method of claim 1 , wherein one or more of: the level of ambiguity of the linkage feature, the linkage feature value, and value pair are used to determine the contribution of samples to the training set.
3. The method of claim 1 , where a process is used to determine the level of ambiguity of one or more of: the linkage feature, the linkage feature value, and value pair.
4. A non-transitory computer readable information storage media having stored thereon instructions, that when executed by a processor perform a challenging training sample selection method comprising:
consideration of record fields as independent;
selection of challenging training samples based on one or more of: a number of multiple labels/classes that record field values, and link feature value ambiguity;
proportional selection of challenging training samples based on one or more of: a level of the labels/classes that record field values, and link feature values ambiguity;
contribution of each linkage feature to the training sample is determined by its relative level of ambiguity;
contribution of a linkage feature value within a linkage feature is determined by its relative level of ambiguity;
contribution of value pairs associated with the same linkage feature is determined by its relative level of ambiguity; and
use of the labels/classes that record field values from positive labels to optimize one or more of: the determination of record field values, and link feature values ambiguity.
5. The media of claim 4 , wherein one or more of: the level of ambiguity of the linkage feature, the linkage feature value, and a value pair are used to determine the contribution of samples to the training set.
6. The media of claim 4 , where any technique is used to determine one or more of: the level of ambiguity of the linkage feature, the linkage feature value and, a value pair.
7. A challenging training sample selection system comprising:
a processor;
an I/O interface; and
storage, the storage having stored thereon instructions, that when executed by the processor, cause to be performed:
consideration of record fields as independent;
selection of challenging training samples based on one or more of: a number of multiple labels/classes that record field values, and link feature value ambiguity;
proportional selection of challenging training samples based on one or more of: a level of labels/classes that record field values, and link feature values ambiguity;
determining a contribution of each linkage feature to the training sample by its relative level of ambiguity;
determining a contribution of a linkage feature value within a linkage feature by its relative level of ambiguity;
determining a contribution of value pairs associated with the same linkage feature by its relative level of ambiguity; and
use of the labels/classes that record field values from positive labels to optimize one or more of: the determination of record field values, and link feature values ambiguity.
8. The system of claim 7 , wherein the level of ambiguity of the linkage feature, the linkage feature value, and a value pair are used to determine the contribution of samples to the training set.
9. The system of claim 7 , where in any technique is used to determine the level of ambiguity of the linkage feature, the linkage feature value, and a value pair in any combination.