Record management for database systems using fuzzy field matching
View Patent ↗Devices, systems and methods are provided for managing records. One method involves constructing a fuzzy match graph data structure with edges between different field values for a fuzzy matched field of a plurality of records in accordance with a matching rule, assigning unique identifiers to different subgraphs of field values within the fuzzy match graph data structure, determining a match key value associated with a respective record based on a respective unique identifier assigned to a respective subgraph of field values including a respective field value for the fuzzy matched field of the respective record, constructing a heterogenous graph data structure having edges between the respective match key values associated with the respective records of the plurality of records, identifying a subset of related records using the heterogenous graph data structure, and performing one or more actions on the subset of related records at a database system.
1 . A method of managing a plurality of existing records at a database system, the method comprising:
obtaining, at the database system, a data stream for ingestion by the database system from a first data source different from a second data source, the second data source being associated with the plurality of existing records at the database system, the data stream comprising one or more new or updated records;
clustering respective records of the plurality of existing records and the one or more new or updated records into clusters of related records based at least in part on an actual field value for an exact matched field of the respective record comprising a common value for the exact matched field of a respective cluster of related records and one or more respective values for one or more fuzzy matched fields of the respective record;
identifying a distinct subset of related records to be assigned to an entity from among the respective cluster of related records using one or more fuzzy matching criteria for the one or more fuzzy matched fields of the respective cluster of related records, the distinct subset of related records including at least one new or updated record from the first data source and at least one existing record associated with the second data source, wherein query operations to identify the distinct subset are t×log(k), where t is a number of fuzzy match identifiers that is less than a number of fields per record and k is a number of distinct field values for the one or more fuzzy matched fields that is less than a total number of records;
selecting one or more values for the one or more fuzzy matched fields from a respective record of the distinct subset of related records, resulting in one or more selected values for the one or more fuzzy matched fields; and
creating, in a data table maintaining unified profiles at the database system, a unified record for the entity corresponding to the distinct subset of related records using the distinct subset of related records at the database system, the unified record comprising the common value for the exact matched field and the one or more selected values for the one or more fuzzy matched fields.
2 . The method of claim 1 , wherein creating the unified record comprises creating a top level entity for the distinct subset of related records at the database system.
3 . The method of claim 1 , wherein the one or more fuzzy matching criteria include at least one of semantic similarity, typographical similarity and phonetic similarity.
4 . The method of claim 1 , wherein:
the exact matched field comprises an email field; and
clustering respective records of the plurality of existing records into the clusters of related records comprises identifying a group of records of the plurality of existing records and the one or more new or updated records having the common value for the email field of the respective records of the group of records.
5 . The method of claim 4 , wherein the one or more fuzzy matched fields comprises a name field and identifying the distinct subset of related records comprises identifying a distinct subset of the group of records having respective values for the name field that fuzzy match with a desired similarity.
6 . The method of claim 5 , further comprising determining the name field values fuzzy match based on a pairwise distance between the respective values for the name field.
7 . The method of claim 1 , wherein identifying the distinct subset of related records comprises identifying the distinct subset of related records based on a pairwise distance between a fuzzy matched field of respective pairs of related records of the respective cluster of related records.
8 . The method of claim 1 , further comprising assigning a common match key value to the distinct subset of related records that fuzzy match a fuzzy matched field.
9 . The method of claim 8 , further comprising calculating the common match key value associated with a respective record of the distinct subset of related records as a function of a respective field value for the fuzzy matched field of the respective record.
10 . The method of claim 1 , wherein creating the unified record comprises selecting the one or more values for the one or more fuzzy matched fields of the unified record from a respective record of the distinct subset of related records having a most recent timestamp.
11 . The method of claim 1 , further comprising deduplicating a database at the database system by deleting the distinct subset of related records.
12 . At least one non-transitory machine-readable storage medium that provides instructions that, when executed by at least one processor, are configurable to cause the at least one processor to perform operations comprising:
obtaining a data stream for ingestion by a database system from a first data source different from a second data source, the second data source being associated with a plurality of existing records at the database system, the data stream comprising one or more new or updated records;
clustering respective records of the plurality of existing records at the database system and the one or more new or updated records into clusters of related records based at least in part on an actual field value for an exact matched field of the respective record comprising a common value for the exact matched field of a respective cluster of related records and one or more respective values for one or more fuzzy matched fields of the respective record;
identifying a distinct subset of related records to be assigned to an entity from among the respective cluster of related records using one or more fuzzy matching criteria for the one or more fuzzy matched fields of the respective cluster of related records, the distinct subset of related records including at least one new or updated record from the first data source and at least one existing record associated with the second data source, wherein query operations to identify the distinct subset are t×log(k), where t is a number of fuzzy match identifiers that is less than a number of fields per record and k is a number of distinct field values for the one or more fuzzy matched fields that is less than a total number of records;
selecting one or more values for the one or more fuzzy matched fields from a respective record of the distinct subset of related records, resulting in one or more selected values for the one or more fuzzy matched fields; and
creating a unified record for the entity corresponding to the distinct subset of related records in a data table maintaining unified profiles at the database system using the distinct subset of related records at the database system, the unified record comprising the common value for the exact matched field and the one or more selected values for the one or more fuzzy matched fields.
13 . The at least one non-transitory machine-readable storage medium of claim 12 , wherein the unified record comprises a top level entity for the distinct subset of related records at the database system.
14 . The at least one non-transitory machine-readable storage medium of claim 12 , wherein the exact matched field comprises an email field.
15 . The at least one non-transitory machine-readable storage medium of claim 14 , wherein the one or more fuzzy matched fields comprises a name field.
16 . The at least one non-transitory machine-readable storage medium of claim 12 , wherein a first cluster of the clusters of related records comprises a group of records of the plurality of existing records and the one or more new or updated records having the common value for an email field of the respective records of the group of records.
17 . The at least one non-transitory machine-readable storage medium of claim 16 , wherein the distinct subset of related records comprises a distinct subset of the group of records having respective values for a name field of the respective records of the distinct subset of the group of records that fuzzy match with a desired similarity.
18 . The method of claim 1 , wherein selecting one or more values for the one or more fuzzy matched fields comprises selecting the one or more values for the one or more fuzzy matched fields from the respective record of the distinct subset of related records that is associated with one of the first data source and the second data source based on a quality metric.
19 . The method of claim 1 , further comprising referencing, by an instance of a virtual application dynamically created by an application platform of the database system at run time, the unified record rather than another record associated with the first data source or the second data source.