System and method for reconciling transactions between a replication system and a recovered database
View Patent ↗The invention is a reconcile system for providing data replication between a source system and a target system. The reconcile system includes a reconcile process that determines a recovery position for a failed or otherwise desynchronized target system, and purges queues of the reconcile system of stale transactions occurring before the recovery position, thereby ensuring synchronization between the source and target systems.
1. A device for performing replication between a source system and a target system, the device comprising:
a source system having data files, and log files storing replication transactions corresponding to changes made to the data files;
a recovered target system wherein the recovered target system comprises a rolled back copy of the data files in the source system and wherein the rolled back copy of the data files comprises data that is associated with at least one replication transaction stored in the log files; and
a replication system performing replication of at least portions of the data files of the source system to the recovered target system by reading the log files and posting the changes from the log files to the recovered target system, the replication system comprising:
transaction-level poster queues, each poster queue storing statements corresponding to a particular replication transaction from the source system, and
a reconcile process which purges replication transactions from the poster queues when the replication transactions have already been applied to the recovered target system; and
wherein the replication system performs the replication transactions by rolling forward at least some of the information rolled back during the recovery of the recovered target system such that the purged replication transactions in the poster queues are not applied while rolling forward such that the source database remains available during recovery of the recovered target database.
2. A replication system for replicating at least portions of the data contained in a source database to a target database, the replication system comprising:
poster queues which store information corresponding to changes made to at least portions of a source system;
at least one poster process which reads the information stored in the poster queues and generates commands interpretable by a target system and designed to change the target system to reflect the changes made to the at least portions of the source system;
instantiation or recovery process of the target system that rolls back information previously applied to the target system such that the target system comprises at least a copy of some of the information stored in the poster queues; and
a reconcile process which purges stale information stored in the poster queues, the stale information corresponding to changes made to the target system during the instantiation or recovery thereof; and wherein the poster queues roll forward at least some of the information rolled back during the recovery of the recovered target system such that the purged stale information is not applied while rolling forward and wherein the source system remains available during instantiation or recovery of the target database.
3. The replication system of claim 2 , wherein the information comprises transactions.
4. The replication system of claim 2 , wherein the at least one poster process reads a completion indicator from the poster queues, wherein the completion indicator corresponds to one or more finalized changes made to the source system.
5. The replication system of claim 4 , wherein the completion indicator corresponds to a COMMIT statement.
6. The replication system of claim 2 , wherein the reconcile process employs placement indicators to determine which information stored in the poster queues is stale.
7. The replication system of claim 6 , wherein one of the placement indicators corresponds to a recovery marker placed by the target system, wherein the recovery marker identifies how much of the information the target system already applied during recovery thereof.
8. The replication system of claim 6 , wherein one of the placement indicators corresponds to a particular portion of the information.
9. The replication system of claim 6 , wherein each placement indicator comprises a sequence number identifying a log file where a particular portion of the information originated.
10. The replication system of claim 6 , wherein each placement indicator comprises a displacement number identifying the displacement within a log file where a particular portion of the information originated.
11. The replication system of claim 2 , further comprising a reader process which reads the information from the source system.
12. The replication system of claim 2 , further comprising a reader queue which stores information read from the source system.
13. The replication system of claim 2 , wherein the replication includes mirroring at least portions of the source system on at least one target system.
14. The replication system of claim 2 , wherein the replication includes load balancing functions based on one of software and hardware configurations of the source and target systems.
15. The replication system of claim 2 , wherein the replication provides broadcast functions.
16. The replication system of claim 2 , wherein the replication provides consolidation functions.
17. A method of recovering or instantiating a target database during replication from a source database to the target database, the method comprising:
creating a target database that has a copy of data from a source database;
logging replication transactions in a replication system wherein the replication transactions are associated with changes made to the source database;
applying the replication transactions to the target database to maintain in the target database a copy of the source database;
identifying an unsynchronizing event associated with the target database;
recovering the target database by rolling back information previously applied to the target database such that the recovered target database contains a copy of at least some of the changes represented by the replication transactions contained in the replication system;
reconciling the recovered target database with the replication transactions contained in the replication system, thereby purging stale replication transactions from the replication system; and
restarting replication by rolling forward at least some of the information rolled back during the recovery of the target database such that the purged stale replication transactions are not applied during replication and wherein the source database remains available during recovering, reconciling and restarting.
18. The method of claim 17 , further comprising restarting replication.
19. The method of claim 18 , wherein the restarting of the replication includes restarting at least one poster process.
20. The method of claim 17 , wherein the creation of the copy includes employing a hot backup mode of a database management system of the source database.
21. The method of claim 17 , wherein the recovery of the copy includes employing a database management system associated with the copy.
22. The method of claim 17 , wherein the recovery of the copy includes placing a recovery marker in the recovered copy, thereby identifying a recovery position therein.
23. The method of claim 22 , wherein the placement of the recovery marker occurs substantially near the end of recovering the copy.
24. The method of claim 22 , wherein the reconciling finds the recovery marker and the stale transactions of the information correspond to those transactions that were completed on the source system before the placement of the recovery marker.
25. A method of reconciling transactional information stored in a replication system with a recovered database, the method comprising:
parsing a log file of a recovered database to determine a placement indicator of a recovery flag;
rolling back information previously applied to a recovered database, wherein the recovered database comprises at least some data that is also represented by transaction data that exists in the log file;
reading the transaction data corresponding to changes made to a source database to determine placement indicators of completed transactions;
purging the transactional data when the placement indicator corresponding to the completed transaction occurred before the placement indicator of the recovery flag; and
replicating the transactional data to the recovered database by rolling forward at least some of the information rolled back during the recovery of the recovered database such that the purged transaction data are not applied during replication such that the source database remains available while replicating the transactional data to the recovered database.
26. The method of claim 25 , wherein the transaction data is read from the source database.
27. The method of claim 25 , wherein the placement indicator of the completed transaction comprises a sequence number of a log file of the source database.
28. The method of claim 27 , wherein the sequence number uniquely identifies the log file.
29. The method of claim 25 , wherein the placement indicator of the recovery flag comprises a sequence number of a log file of the recovered database.
30. The method of claim 29 , wherein the sequence number uniquely identifies the log file.
31. The method of claim 25 , wherein the placement indicator comprises a displacement within a log file.
32. The method of claim 25 , wherein the recovery flag is placed by a database management system of the recovered database.
33. A device comprising:
a source system having a source database management system (SDBMS) which governs the storage of data within the source system and creates a log file tracking changes made to the source system;
a target system having a target database management system (TDBMS) which governs the storage of data within the target system and creates a log file tracking the changes made to the target system wherein the target database management system is configured to recover the target system by rolling back portions of previously applied information; and
a replication system having queues and communicating with the log file of the TDBMS and the log file of the SDBMS, thereby purging from the queues the replication transactions applied after the beginning, but before the completion, of the recovery of the target system, wherein the replication transactions correspond to the changes made to the source system wherein the replication system is configured to roll forward at least some of the information rolled back during the recovery of the target system such that the purged replication transactions are not applied during replication such that the source system remains available during recovery of the target database.