Self-healing data synchronization
A self-healing data synchronization process includes an initial stage in which a collection of data change events is received, a set of data record(s) corresponding to the data change event(s) is identified, and a syncing of the set of data record(s) is initiated. Data that indicates which data record(s) successfully synced and which failed is stored. During a subsequent stage of the self-healing process, data change events that occurred during a preceding time horizon are identified, a corresponding first set of data record(s) are identified, a difference between the first set and a second set of data record(s) that successfully synced during the time horizon is determined as a third set of data record(s), and any data record that was attempted to be synced during the time horizon but failed is excluded from the third set. A sync of any data record remaining in the third set is then initiated.
1. A computer-implemented method for syncing data from a first database accessible by a first computing system to a second database accessible by a second computing system, the method comprising:
receiving a first set of data change events occurring in the first database over a period of time, the first set of data change events possibly failing to include particular data change events occurring in the first database during the period of time;
storing the first set of data change events in a synchronization store, the first set of data change events corresponding to a first set of data records;
synchronizing the first data of data records from the first database to the second database;
reviewing a bin log of the first database to identify a second set of data change events occurring in the first database over the period of time, the second set of data change events corresponding to a second set of data records;
accessing the synchronization store to retrieve the first set of data change events;
comparing the second set of data change events against the first set of data change events to determine the particular data change events, the particular data change events corresponding to a third set of one or more data records;
determining a fourth set of one or more data records that failed to successfully sync to the second database;
writing sync failure information to a failure store;
removing the fourth set of one or more data records from the third set of one or more data records; and
synchronizing the third set of one or more data records from the first database to the second database.
2. The computer-implemented method of claim 1 , wherein the receiving the first set of data change events occurs in real time and comes from an upstream data service.
3. The computer-implemented method of claim 2 , wherein the upstream data service is configured to convert tabular data to object data, and wherein the particular data change events are due, at least in part, to a data conversion error by the upstream data service.
4. The computer-implemented method of claim 2 , further comprising:
ordering bin log files from the bin log based at least in part on respective timestamps associated with the second set of data change events; and
identifying each data change event of the second set of data change events having a respective timestamp that falls within the period of time.
5. The computer-implemented method of claim 4 , further comprising:
monitoring, by the upstream data service, the bin log over the period of time to identify the first set of data change events.
6. The computer-implemented method of claim 5 , wherein the particular data change events are due, at least in part, to downtime of the upstream data service during at least a portion of the period of time.
7. The computer-implemented method of claim 1 , further comprising prior to retrieving the third set of one or more data records:
accessing the second store to identify the fourth set of one or more data records.
8. The computer-implemented method of claim 1 , wherein each data record in the first set of data records are a new data record created during the period of time or a modification to existing data record during the period of time.
9. A system configured to sync data from a first database accessible by a first computing system to a second database accessible by a second computing system, the system comprising:
at least one processor; and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
receive a first set of data change events occurring in the first database over a period of time, the first set of data change events possibly failing to include particular data change events occurring in the first database during the period of time;
store the first set of data change events in a synchronization store, the first set of data change events corresponding to a first set of data records;
synchronize the first data of data records from the first database to the second database;
review a bin log of the first database to identify a second set of data change events occurring in the first database over the period of time, the second set of data change events corresponding to a second set of data records;
access the synchronization store to retrieve the first set of data change events;
compare the second set of data change events against the first set of data change events to determine the particular data change events, the particular data change events corresponding to a third set of one or more data records;
determine a fourth set of one or more data records that failed to successfully sync to the second database;
write sync failure information to a failure store;
remove the fourth set of one or more data records from the third set of one or more data records; and
synchronize the third set of one or more data records from the first database to the second database.
10. The system of claim 9 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
receive the first set of data change events occurs in real time and comes from an upstream data service.
11. The system of claim 10 , wherein the upstream data service is configured to convert tabular data to object data, and wherein the particular data change events are due, at least in part, to a data conversion error by the upstream data service.
12. The system of claim 10 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
order bin log files from the bin log based at least in part on respective timestamps associated with the second set of data change events; and
identify each data change event of the second set of data change events having a respective timestamp that falls within the period of time.
13. The system of claim 12 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
monitor, by the upstream data service, the bin log over the period of time to identify the first set of data change events.
14. The system of claim 13 , wherein the particular data change events are due, at least in part, to downtime of the upstream data service during at least a portion of the period of time.
15. The system of claim 9 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
access the second store to identify the fourth set of one or more data records.
16. The system of claim 9 , wherein each data record in the first set of data records are a new data record created during the period of time or a modification to existing data record during the period of time.
17. A computer program product for syncing data from a first database accessible by a first computing system to a second database accessible by a second computing system, the computer program product comprising a non-transitory computer-readable medium readable by a processing circuit, the non-transitory computer-readable medium storing instructions executable by the processing circuit to cause a method to be performed, the method comprising:
receiving a first set of data change events occurring in the first database over a period of time, the first set of data change events possibly failing to include particular data change events occurring in the first database during the period of time;
storing the first set of data change events in a synchronization store, the first set of data change events corresponding to a first set of data records;
synchronizing the first data of data records from the first database to the second database;
reviewing a bin log of the first database to identify a second set of data change events occurring in the first database over the period of time, the second set of data change events corresponding to a second set of data records;
accessing the synchronization store to retrieve the first set of data change events;
comparing the second set of data change events against the first set of data change events to determine the particular data change events, the particular data change events corresponding to a third set of one or more data records;
determining a fourth set of one or more data records that failed to successfully sync to the second database;
writing sync failure information to a failure store;
removing the fourth set of one or more data records from the third set of one or more data records; and
synchronizing the third set of one or more data records from the first database to the second database.