User identifier match and merge process
A method that includes receiving a first configuration and a second configuration that define a set of rules for matching and merging a set of source data objects that are associated with a tenant and that are received from a plurality of data sources. The method may further include generating a set of merged data objects from the set of source data objects based on an identification of matching values from fields of the set of source data objects and selecting a value for each field of each merged data object having multiple values. The method may further include generating a mapping between primary keys associated with each merged data object and corresponding primary keys of the source data objects. The method may further include storing the merged data objects and the mappings in a first datastore and a second datastore that is different from the first datastore.
1 . A method, comprising:
configuring, for a data processing platform, a data processing job based at least in part on a selection of a scheduled trigger condition for the data processing job, a set of duplicate detection rules to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiating the data processing job for the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of duplicate detection rules configured for the data processing platform, the duplicate data records comprising matching data and conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
2 . The method of claim 1 , wherein the scheduled trigger condition comprises an enablement condition, a repeated failure threshold condition, a daily quota condition, a mapping condition, a new data condition, a mapping update condition, a start time for the data processing job, a scheduled time for a subsequent data processing job, or any combination thereof.
3 . The method of claim 1 , further comprising:
receiving an indication of one or more parameters associated with the set of duplicate detection rules, wherein identifying the duplicate data records is based at least in part on the one or more parameters.
4 . The method of claim 1 , further comprising:
receiving a selection of a primary record to use for merging two or more of the duplicate data records, wherein data from the two or more duplicate data records is merged into the primary record in accordance with the selection.
5 . The method of claim 1 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
6 . The method of claim 1 , wherein the set of duplicate detection rules comprise fuzzy matching rules that indicate duplicate detection based at least in part on matching at least a first portion of a first field with at least a second portion of a second field.
7 . The method of claim 1 , wherein initiating the data processing job comprises:
checking a status of one or more conditions, including the scheduled trigger condition, that are configured for the data processing platform.
8 . An apparatus, comprising:
one or more memories storing processor-executable code; and
one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
configure, for a data processing platform, a data processing job based at least in part on a selection of a scheduled trigger condition for the data processing job, a set of duplicate detection rules to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiate the data processing job for the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of duplicate detection rules configured for the data processing platform, the duplicate data records comprising matching data and conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
9 . The apparatus of claim 8 , wherein the scheduled trigger condition comprises an enablement condition, a repeated failure threshold condition, a daily quota condition, a mapping condition, a new data condition, a mapping update condition, a start time for the data processing job, a scheduled time for a subsequent data processing job, or any combination thereof.
10 . The apparatus of claim 8 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
receive an indication of one or more parameters associated with the set of duplicate detection rules, wherein identify the duplicate data records is based at least in part on the one or more parameters.
11 . The apparatus of claim 8 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
receive a selection of a primary record to use for merging two or more of the duplicate data records, wherein data from the two or more duplicate data records is merged into the primary record in accordance with the selection.
12 . The apparatus of claim 8 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
13 . The apparatus of claim 8 , wherein the set of duplicate detection rules comprise fuzzy matching rules that indicate duplicate detection based at least in part on matching at least a first portion of a first field with at least a second portion of a second field.
14 . The apparatus of claim 8 , wherein, to initiate the data processing job, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
check a status of one or more conditions, including the scheduled trigger condition, that are configured for the data processing platform.
15 . A non-transitory computer-readable medium storing code, the code comprising instructions executable by one or more processors to:
configure, for a data processing platform, a data processing job based at least in part on a selection of a scheduled trigger condition for the data processing job, a set of duplicate detection rules to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiate the data processing job for the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of duplicate detection rules configured for the data processing platform, the duplicate data records comprising matching data and conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
16 . The non-transitory computer-readable medium of claim 15 , wherein the scheduled trigger condition comprises an enablement condition, a repeated failure threshold condition, a daily quota condition, a mapping condition, a new data condition, a mapping update condition, a start time for the data processing job, a scheduled time for a subsequent data processing job, or any combination thereof.
17 . The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the one or more processors to:
receive an indication of one or more parameters associated with the set of duplicate detection rules;
wherein identify the duplicate data records is based at least in part on the one or more parameters.
18 . The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the one or more processors to:
receive a selection of a primary record to use for merging two or more of the duplicate data records, wherein data from the two or more duplicate data records is merged into the primary record in accordance with the selection.
19 . The non-transitory computer-readable medium of claim 15 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
20 . The non-transitory computer-readable medium of claim 15 , wherein the set of duplicate detection rules comprise fuzzy matching rules that indicate duplicate detection based at least in part on matching at least a first portion of a first field with at least a second portion of a second field.