Apparatus and method for log based replication of distributed transactions using globally acknowledged commits
A computer readable storage medium includes executable instructions to read source node transaction logs to capture transaction data, including local transaction data, global transaction identifiers and participating node data. The global transaction identifiers and participating node data are stored in target node queues. The target node queues are accessed to form global transaction data. Target tables are constructed based upon the local transaction data and the global transaction data.
1. A method of analyzing distributed transactions performed on a primary database system and replicating the distributed transactions on a secondary database system, the method comprising:
at the primary database system:
performing a global transaction on the primary database system, wherein:
the primary database system comprises a first plurality of nodes;
each of the first plurality of nodes comprises a transaction log; and
the global transaction comprises a plurality of local transactions performed on a subset of the first plurality of nodes;
storing records of the local transactions in the transaction logs, wherein the records comprise:
a global transaction identifier; and
a list of nodes involved in the global transaction; and
sending the records of the local transactions from the transaction logs over a network to the secondary database system; and
at the secondary database system:
storing the records of the local transactions, wherein:
the secondary database system comprises a second plurality of nodes;
each of the second plurality of nodes comprises a commit-ordered transaction queue; and
the records of the local transactions are stored in the commit-ordered transaction queues of nodes in the secondary database system that correspond to nodes in the primary database system;
waiting until the global transaction is completely represented in the commit-ordered transaction queues by monitoring the global transaction identifier and the list of nodes involved in the global transaction; and
performing, after the global transaction is completely represented in the commit-ordered transaction queues, the plurality of local transactions on corresponding nodes in the second plurality of nodes in the secondary database system, such that data affected by the global transaction in the secondary database system matches corresponding data affected by the global transaction in the primary database system.
2. The method claim 1 , wherein a process coordinator that is shared by the second plurality of nodes in the secondary database system determines when the global transaction is completely represented in the commit-ordered transaction queues.
3. The method of claim 2 , wherein the process coordinator utilizes a two-phase commit protocol to determine when the global transaction is completely represented in the commit-ordered transaction queues.
4. The of claim 1 , wherein each of the first plurality of nodes comprises a resource manager and a database table.
5. The method of claim 1 , wherein a one-to-one relationship exists between the first plurality of nodes and the second plurality of nodes.
6. The method of claim 1 , wherein the commit-ordered transaction queues additionally store non-global transactions performed on individual nodes in the primary database system.
7. The method of claim 1 , wherein the commit-ordered transaction queues are scanned by the secondary database system, and a starting position of each of the records of the local transactions is recorded.
8. A non-transitory computer-readable memory comprising a sequence of instructions which, when executed by one or more processors, causes the one or more processors to analyze distributed transactions performed on a primary database system and replicate the distributed transactions on a secondary database system by:
at the primary database system:
performing a global transaction on the primary database system, wherein:
the primary database system comprises a first plurality of nodes;
each of the first plurality of nodes comprises a transaction log; and
the global transaction comprises a plurality of local transactions performed on a subset of the first plurality of nodes;
storing records of the local transactions in the transaction logs, wherein the records comprise:
a global transaction identifier; and
a list of nodes involved in the global transaction; and
sending the records of the local transactions from the transaction logs over a network to the secondary database system; and
at the secondary database system:
storing the records of the local transactions, wherein:
the secondary database system comprises a second plurality of nodes;
each of the second plurality of nodes comprises a commit-ordered transaction queue; and
the records of the local transactions are stored in the commit-ordered transaction queues of nodes in the secondary database system that correspond to nodes in the primary database system;
waiting until the global transaction is completely represented in the commit-ordered transaction queues by monitoring the global transaction identifier and the list of nodes involved in the global transaction; and
performing, after the global transaction is completely represented in the commit-ordered transaction queues, the plurality of local transactions on corresponding nodes in the second plurality of nodes in the secondary database system, such that data affected by the global transaction in the secondary database system matches corresponding data affected by the global transaction in the primary database system.
9. The non-transitory computer-readable memory according to claim 8 wherein a process coordinator that is shared by the second plurality of nodes in the secondary database system determines when the global transaction is completely represented in the commit-ordered transaction queues.
10. The non-transitory computer-readable memory according to claim 9 wherein the process coordinator utilizes a two-phase commit protocol to determine when the global transaction is completely represented in the commit-ordered transaction queues.
11. The non-transitory computer-readable memory according to claim 8 wherein each of the first plurality of nodes comprises a resource manager and a database table.
12. The non-transitory computer-readable memory according to claim 8 wherein a one-to-one relationship exists between the first plurality of nodes and the second plurality of nodes.
13. The non-transitory computer-readable memory according to claim 8 wherein the commit-ordered transaction queues additionally store non-global transactions performed on individual nodes in the primary database system.
14. The non-transitory computer-readable memory according to claim 8 wherein the commit-ordered transaction queues are scanned by the secondary database system, and a starting position of each of the records of the local transactions is recorded.
15. A system comprising:
one or more processors; and
a memory communicatively coupled with and readable by the one or more processors and comprising a sequence of instructions which, when executed by the one or more processors, cause the one or more processors to analyze distributed transactions performed on a primary database system and replicate the distributed transactions on a secondary database system by:
at the primary database system:
performing a global transaction on the primary database system, wherein:
the primary database system comprises a first plurality of nodes;
each of the first plurality of nodes comprises a transaction log; and
the global transaction comprises a plurality of local transactions performed on a subset of the first plurality of nodes;
storing records of the local transactions in the transaction logs, wherein the records comprise:
a global transaction identifier; and
a list of nodes involved in the global transaction; and
sending the records of the local transactions from the transaction logs over a network to the secondary database system; and
at the secondary database system:
storing the records of the local transactions, wherein:
the secondary database system comprises a second plurality of nodes;
each of the second plurality of nodes comprises a commit-ordered transaction queue; and
the records of the local transactions are stored in the commit-ordered transaction queues of nodes in the secondary database system that correspond to nodes in the primary database system;
waiting until the global transaction is completely represented in the commit-ordered transaction queues by monitoring the global transaction identifier and the list of nodes involved in the global transaction; and
performing, after the global transaction is completely represented in the commit-ordered transaction queues, the plurality of local transactions on corresponding nodes in the second plurality of nodes in the secondary database system, such that data affected by the global transaction in the secondary database system matches corresponding data affected by the global transaction in the primary database system.
16. The system of claim 15 wherein a process coordinator that is shared by the second plurality of nodes in the secondary database system determines when the global transaction is completely represented in the commit-ordered transaction queues.
17. The system of claim 16 wherein the process coordinator utilizes a two-phase commit protocol to determine when the global transaction is completely represented in the commit-ordered transaction queues.
18. The system of claim 15 wherein each of the first plurality of nodes comprises a resource manager and a database table.
19. The system of claim 15 wherein a one-to-one relationship exists between the first plurality of nodes and the second plurality of nodes.
20. The system of claim 15 wherein the commit-ordered transaction queues additionally store non-global transactions performed on individual nodes in the primary database system.