Statistical measure and calibration of internally inconsistent search criteria where one or both of the search criteria and database is incomplete
Disclosed is a system for, and method of, searching for and identifying an entity representation. Some embodiments permit search criteria that are internally inconsistent. Such internally inconsistent criteria may include, for example, a maiden last name and a married last name. Certain embodiments account for such criteria in an intelligent manner and identify matching entity representations with a known confidence level of accuracy.
1. A method of identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations each corresponding to a definitive identifier, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight indicating the likelihood that a record or entity representation chosen at random contains the associated field value, the method comprising:
receiving a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;
receiving at least one match template specifying an ordered plurality of fields;
forming and electronically storing, for each match template, a table comprising field value weights for a plurality of records having matches between a search criteria field value and a field value appearing in a record in the universal database, wherein the at least two search criteria field values match field values in two records corresponding to a same entity representation, and wherein at least one table comprises an inclusion field comprising a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records corresponding to the same entity representation;
merging the tables according to entity representation, resulting in a merged table, wherein the at least two search criteria field values that are not identical are grouped in the merged table and associated with the same definitive identifier corresponding to the entity representation;
summing field value weights according to entity representation in the merged table, resulting in a plurality of summed weights, one summed weight for each entity representation;
ranking entity representations in the merged table according to the plurality of summed weights; determining a highest ranked entity representation;
calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and
outputting, when the confidence level exceeds a predetermined threshold, an identifier sufficient to identify the entity representation for the highest ranked entity representation.
2. The method of claim 1 , wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.
3. The method of claim 1 , wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a nonidentical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.
4. The method of claim 1 , wherein at least one match template comprises a fixed field portion and an optional field portion.
5. The method of claim 4 , wherein an arbitrary record matches an arbitrary search criteria when all fixed fields of the arbitrary record are populated with field values that match corresponding fixed field values of the arbitrary search criteria and when all optional fields of the arbitrary record are each populated with one of a null field value and a field value that matches a corresponding optional field value of the arbitrary search criteria.
6. The method of claim 1 , wherein the identifier comprises a social security number.
7. A method of identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations each corresponding to a definitive identifier, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight indicating the likelihood that a record or entity representation chosen at random contains the associated field value, the method comprising:
receiving a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;
forming and electronically storing a table comprising field value weights for a plurality of records having matches between a search criteria field value and a field value appearing in a record in the universal database, wherein the at least two search criteria field values match field values in two records corresponding to a same entity representation, and wherein at least one table comprises an inclusion field comprising a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records corresponding to the same entity representation;
merging the tables according to entity representation, resulting in a merged table, wherein the at least two search criteria field values that are not identical are grouped in the merged table and associated with the same definitive identifier corresponding to the entity representation;
summing field value weights according to entity representation in the merged table, resulting in a plurality of summed weights, one summed weight for each entity representation;
ranking entity representations in the merged table according to the plurality of summed weights;
determining a highest ranked entity representation;
calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and
outputting, when the confidence level exceeds a predetermined threshold, an identifier sufficient to identify the entity representation for the highest ranked entity representation.
8. The method of claim 7 , wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.
9. The method of claim 7 , wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.
10. The method of claim 7 , wherein the identifier comprises a social security number.
11. A system for identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations each corresponding to a definitive identifier, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight indicating the likelihood that a record or entity representation chosen at random contains the associated field value, the system comprising:
an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight; an electronic memory storing a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;
an electronic memory storing at least one match template specifying an ordered plurality of fields;
a processor programmed to form and electronically store, for each match template, a table comprising field value weights for a plurality of records having matches between a search criteria field value and a field value appearing in a record in the universal database, wherein the at least two search criteria field values match field values in two records corresponding to a same entity representation, and wherein at least one table comprises an inclusion field comprising a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records corresponding to the same entity representation;
a processor programmed to merge and electronically store the tables according to entity representation, resulting in a stored merged table, wherein the at least two search criteria field values that are not identical are grouped in the merged table and associated with the same definitive identifier corresponding to the entity representation;
a processor programmed to sum and electronically store field value weights according to entity representation in the merged table, resulting in a plurality of stored summed weights, one summed weight for each entity representation;
a processor programmed to rank entity representations in the merged table according to the plurality of summed weights; a processor programmed to determine a highest ranked entity representation;
a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and
an output configured to output, when the confidence level exceeds a predetermined threshold, an identifier sufficient to identify the entity representation for the highest ranked entity representation.
12. The system of claim 11 , wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.
13. The system of claim 11 , wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.
14. The system of claim 11 , wherein at least one match template comprises a fixed field portion and an optional field portion.
15. The system of claim 14 , wherein an arbitrary record matches an arbitrary search criteria when all fixed fields of the arbitrary record are populated with field values that match corresponding fixed field values of the arbitrary search criteria and when all optional fields of the arbitrary record are each populated with one of a null field value and a field value that matches a corresponding optional field value of the arbitrary search criteria.
16. The system of claim 11 , wherein the identifier comprises a social security number.
17. A system for identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations each corresponding to a definitive identifier, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight indicating the likelihood that a record or entity representation chosen at random contains the associated field value, the system comprising:
an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;
an electronic memory storing a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;
a processor programmed to form and electronically store a table comprising field value weights for a plurality of records having matches between a search criteria field value and a field value appearing in a record in the universal database, wherein the at least two search criteria field values match field values in two records corresponding to a same entity representation, and wherein at least one table comprises an inclusion field comprising a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records corresponding to the same entity representation;
a processor programmed to merge and electronically store the tables according to entity representation, resulting in a stored merged table, wherein the at least two search criteria field values that are not identical are grouped in the merged table and associated with the same definitive identifier corresponding to the entity representation;
a processor programmed to sum and electronically store field value weights according to entity representation in the merged table, resulting in a plurality of stored summed weights, one summed weight for each entity representation; a processor programmed to rank entity representations in the merged table according to the plurality of summed weights;
a processor programmed to determine a highest ranked entity representation;
a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and
an output configured to output, when the confidence level exceeds a predetermined threshold, an identifier sufficient to identify the entity representation for the highest ranked entity representation.
18. The system of claim 17 , wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.
19. The system of claim 17 , wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.
20. The system of claim 17 , wherein the identifier comprises a social security number.