Systems and methods for synchronization of data
Computer-implemented systems and methods for synchronizing data for dataset execution. The system includes a source database that stores a canonical dataset, a secondary database that stores a processed dataset, and a synchronization server that comprises a processor and a memory. The processor is configured to monitor for a publication of one or more source tables and when the publication is detected, identify the processed tables, corresponding to the source tables, to be updated in the processed dataset. The processor determines a tolerance level corresponding to each processed table and updates the processed tables in the processed dataset. The processor determines whether the processed tables in the processed dataset were successfully updated within the tolerance levels and transmits a notification based on determining whether the processed tables in the processed dataset were successfully updated within the tolerance levels.
1. A system for synchronizing data for dataset execution, the system comprising:
a source database storing a canonical dataset;
a secondary database storing a processed dataset;
a synchronization server comprising a processor and a memory, the processor configured to:
monitor for a publication of one or more source tables;
when the publication is detected, identify one or more processed tables to be updated in the processed dataset, the one or more processed tables corresponding to the one or more source tables;
determine one or more tolerance level corresponding to each of the one or more processed tables, the one or more tolerance level based on an execution requirement of a downstream application, wherein the execution requirement includes an expected start time for execution of the downstream application, and wherein the one or more tolerance level includes an allowable time buffer before the expected start time;
update the one or more processed tables in the processed dataset;
determine whether the one or more processed tables in the processed dataset were successfully updated within the allowable time buffer before the expected start time; and
transmit a notification based on determining whether the one or more processed tables in the processed dataset were successfully updated within the allowable time buffer before the expected start time.
2. The system of claim 1 , wherein the processor is further configured to create a checkpoint prior to updating the one or more processed tables in the processed dataset.
3. The system of claim 1 , wherein a selected table of the one or more processed tables has a selected tolerance level of the one or more tolerance level associated therewith, wherein the selected tolerance level is a time buffer, and wherein the execution requirement for the selected table is determined by:
analyzing an update frequency of the selected table;
analyzing the expected start time of the downstream application;
determining whether the selected table can be successfully updated prior to the expected start time based on the update frequency; and
setting the selected tolerance level based on the determining whether the selected table can be successfully updated prior to the expected start time.
4. The system of claim 1 , wherein the downstream application is a machine learning model.
5. The system of claim 4 , wherein the one or more processed tables are transformed into a format compatible with the machine learning model.
6. The system of claim 1 , further comprising a publisher server comprising a first processor and a first memory, the first processor configured to publish the one or more source tables.
7. The system of claim 1 , wherein the one or more source tables are generated by processing the canonical dataset to remove sensitive information.
8. A method for synchronizing data for dataset execution, the method comprising:
detecting, by a synchronization server, publication of one or more source tables forming a canonical dataset;
in response to detecting the publication of the one or more source tables, identifying, by the synchronization server, one or more processed tables to be updated in a processed dataset, the one or more processed tables corresponding to the one or more source tables;
determining one or more tolerance level corresponding to each of the one or more processed tables, the one or more tolerance level based on an execution requirement of a downstream application, wherein the execution requirement includes an expected start time for execution of the downstream application, and wherein the one or more tolerance level includes an allowable time buffer before the expected start time;
updating, by the synchronization server, the one or more processed tables in the processed dataset;
determining, by the synchronization server, whether the one or more processed tables in the processed dataset were successfully updated within the allowable time buffer before the expected start time; and
transmitting, by the synchronization server, a notification based on determining whether the one or more processed tables in the processed dataset were successfully updates within the allowable time buffer before the expected start time.
9. The method of claim 8 , further comprising:
monitoring, by a synchronization server, publication of one or more processed tables.
10. The method of claim 9 , wherein a selected table of the one or more processed tables has a selected tolerance level of the one or more tolerance level associated therewith, wherein the selected tolerance level is a time buffer, and wherein the execution requirement for the selected table is determined by:
analyzing an update frequency of the selected table;
analyzing the expected start time of the downstream application;
determining whether the selected table can be successfully updated prior to the expected start time based on the update frequency; and
setting the selected tolerance level based on the determining whether the selected table can be successfully updated prior to the expected start time.
11. The method of claim 9 , wherein the downstream application is a machine learning model.
12. The method of claim 11 , further comprising:
transforming the one or more processed tables into a format compatible with the machine learning model.
13. The method of claim 8 , further comprising:
creating a checkpoint prior to the updating the one or more processed tables in the processed dataset.
14. The method of claim 8 , further comprising:
publishing, by a publisher server, the one or more processed tables.
15. The method of claim 8 , further comprising:
processing the canonical dataset to remove sensitive information.
16. A non-transitory computer readable medium storing computer executable instructions which, when executed by at least one computer processor, cause the at least one computer processor to carry out a method for synchronizing data for dataset execution, the method comprising:
detecting, by a synchronization server, publication of one or more source tables forming a canonical dataset;
in response to detecting the publication of the one or more source tables, identifying, by the synchronization server, one or more processed tables to be updated in a processed dataset, the one or more processed tables corresponding to the one or more source tables;
determining one or more tolerance level corresponding to each of the one or more processed tables, the one or more tolerance level based on an execution requirement of a downstream application, wherein the execution requirement includes an expected start time for execution of the downstream application, and wherein the one or more tolerance level includes an allowable time buffer before the expected start time;
updating, by the synchronization server, the one or more processed tables in the processed dataset;
determining, by the synchronization server, whether the one or more processed tables in the processed dataset were successfully updated within the allowable time buffer before the expected start time; and
transmitting, by the synchronization server, a notification based on determining whether the one or more processed tables in the processed dataset were successfully updates within the allowable time buffer before the expected start time.