User identifier match and merge process
A method that includes receiving a first configuration and a second configuration that define a set of rules for matching and merging a set of source data objects that are associated with a tenant and that are received from a plurality of data sources. The method may further include generating a set of merged data objects from the set of source data objects based on an identification of matching values from fields of the set of source data objects and selecting a value for each field of each merged data object having multiple values. The method may further include generating a mapping between primary keys associated with each merged data object and corresponding primary keys of the source data objects. The method may further include storing the merged data objects and the mappings in a first datastore and a second datastore that is different from the first datastore.
1. A method, comprising:
configuring a data processing job for a tenant of a data processing platform, wherein configuring the data processing job comprises receiving a selection of a scheduled trigger condition for the data processing job, a set of matching criteria to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiating the data processing job for the tenant of the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of matching criteria configured for the tenant of the data processing platform, the duplicate data records comprising a first set of fields with matching data and a second set of fields with conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the tenant of the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
2. The method of claim 1 , wherein identifying the duplicate data records comprises:
identifying one or more data records that have matching values for the first set of fields and mismatching values for the second set of fields; and
determining that the one or more data records satisfy the set of matching criteria based at least in part on the matching values for the first set of fields.
3. The method of claim 1 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
4. The method of claim 1 , wherein initiating the data processing job comprises:
checking a status of one or more conditions, including the scheduled trigger condition, that are configured for the tenant of the data processing platform.
5. The method of claim 1 , further comprising:
assigning unique identifiers to at least one merged data record of the set of merged data records; and
updating one or more fields of a first merged data record in association with a subsequent data processing job.
6. The method of claim 1 , wherein the scheduled trigger condition comprises a start time for the data processing job and a scheduled time for a subsequent data processing job.
7. The method of claim 1 , further comprising:
receiving a selection of a primary record to use for merging two or more of the duplicate data records, wherein data from the two or more duplicate data records is merged into the primary record in accordance with the selection.
8. An apparatus, comprising:
one or more memories configured to store code; and
one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
configure a data processing job for a tenant of a data processing platform, wherein configuring the data processing job comprises receiving a selection of a scheduled trigger condition for the data processing job, a set of matching criteria to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiate the data processing job for the tenant of the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of matching criteria configured for the tenant of the data processing platform, the duplicate data records comprising a first set of fields with matching data and a second set of fields with conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the tenant of the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
9. The apparatus of claim 8 , wherein to identify the duplicate data records, the one or more processors are operable to execute the code to cause the apparatus to:
identify one or more data records that have matching values for the first set of fields and mismatching values for the second set of fields; and
determine that the one or more data records satisfy the set of matching criteria based at least in part on the matching values for the first set of fields.
10. The apparatus of claim 8 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
11. The apparatus of claim 8 , wherein to initiate the data processing job, the one or more processors are operable to execute the code to cause the apparatus to:
check a status of one or more conditions, including the scheduled trigger condition, that are configured for the tenant of the data processing platform.
12. The apparatus of claim 8 , wherein the one or more processors are further operable to execute the code to cause the apparatus to:
assign unique identifiers to at least one merged data record of the set of merged data records; and
update one or more fields of a first merged data record in association with a subsequent data processing job.
13. The apparatus of claim 8 , wherein the scheduled trigger condition comprises a start time for the data processing job and a scheduled time for a subsequent data processing job.
14. The apparatus of claim 8 , wherein the one or more processors are further operable to execute the code to cause the apparatus to:
receive a selection of a primary record to use for merging two or more of the duplicate data records, wherein data from the two or more duplicate data records is merged into the primary record in accordance with the selection.
15. A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by one or more processors to:
configure a data processing job for a tenant of a data processing platform, wherein configuring the data processing job comprises receiving a selection of a scheduled trigger condition for the data processing job, a set of matching criteria to use for identifying duplicate data records, and a plurality of data records to check for the duplicate data records; and
initiate the data processing job for the tenant of the data processing platform based at least in part on the scheduled trigger condition being satisfied, wherein the data processing job comprises:
identifying the duplicate data records in the plurality of data records based at least in part on the set of matching criteria configured for the tenant of the data processing platform, the duplicate data records comprising a first set of fields with matching data and a second set of fields with conflicting data;
receiving a selection of a set of merging criteria to use for combining the duplicate data records into a set of merged data records; and
merging the duplicate data records into the set of merged data records based at least in part on the set of merging criteria configured for the tenant of the data processing platform, wherein merging the duplicate data records comprises selecting, from the conflicting data of the duplicate data records, data to include in one or more fields of the set of merged data records.
16. The non-transitory computer-readable medium of claim 15 , wherein to identify the duplicate data records, the one or more processors are operable to execute the code to:
identify one or more data records that have matching values for the first set of fields and mismatching values for the second set of fields; and
determine that the one or more data records satisfy the set of matching criteria based at least in part on the matching values for the first set of fields.
17. The non-transitory computer-readable medium of claim 15 , wherein the set of merging criteria comprises one or more rules associated with merging the duplicate data records based at least in part on a data source type associated with the duplicate data records, a recency of the duplicate data records, a data input frequency associated with the duplicate data records, or a combination thereof.
18. The non-transitory computer-readable medium of claim 15 , wherein to initiate the data processing job, the one or more processors are operable to execute the code to:
check a status of one or more conditions, including the scheduled trigger condition, that are configured for the tenant of the data processing platform.
19. The non-transitory computer-readable medium of claim 15 , wherein the one or more processors are further operable to execute the code to:
assign unique identifiers to at least one merged data record of the set of merged data records; and
update one or more fields of a first merged data record in association with a subsequent data processing job.
20. The non-transitory computer-readable medium of claim 15 , wherein the scheduled trigger condition comprises a start time for the data processing job and a scheduled time for a subsequent data processing job.