Real-time cross-system database replication for hybrid-cloud elastic scaling and high-performance data virtualization
RTR of a database transaction to a replica table may include receiving replication and transaction commit log entries (representing a database transaction). The replication log entry has a row-ID value, and the row at the replica table has a row-ID value. The replication log entry may be dispatched to a parallel log replayer and the associated transaction commit log entry to a transaction commit log replayer. The row-ID values may be compared, and the replication log entry is replayed at the parallel log replayer based on the comparison. The database transaction may then be committed to the replica table by replaying the associated transaction commit log entry at the transaction log replayer, wherein the database transaction is associated with row-level parallel replay having transactional consistency and DDL replication and reconstruction of a DDL statement at the replica system is associated with one or multiple metadata update log entries.
1. A system for Real-time Table Replication (“RTR”) of a database transaction to a replica table, comprising:
a computer memory; and
at least one computer processor coupled to the memory and configured to:
receive a replication log entry and an associated transaction commit log entry, the replication log entry and the associated transaction commit log entry together representing a database transaction to be replayed to a row at a replica table, the replication log entry having a row-ID value and the row at the replica table having a row-ID value,
dispatch the replication log entry to a parallel log replayer and the associated transaction commit log entry to a transaction commit log replayer,
compare the row-ID value of the replication log entry to the row-ID value of the row at the replica table,
replay the replication log entry at the parallel log replayer based on the comparison, and
commit the database transaction to the replica table by replaying the associated transaction commit log entry at the transaction log replayer, wherein the database transaction is associated with row-level parallel replay having transactional consistency and Data Dictionary Language (“DDL”) replication and reconstruction of a DDL statement at the replica system is associated with one or multiple metadata update log entries, and further wherein push-based and early log shipping reduce propagation delay between a source system and a replica system.
2. The system of claim 1 , wherein multiple replication object granularities are supported, including at least one of: (i) a set of tables, (ii) a table, (iii) a sub-table, (iv) one or more columns, and (v) one or more partitions.
3. The system of claim 1 , wherein replication having a topology from multiple distinct remote source systems is supported as N-to-1 replication.
4. The system of claim 1 , wherein replication having a topology from to multiple distinct remote replica systems is supported as 1-to-N replication.
5. The system of claim 1 , wherein replication having a topology with a replica table being a source of another replica table is supported as chain replication.
6. The system of claim 1 , wherein in-memory log replication does not rely on a store-and-forward mechanism.
7. The system of claim 1 , wherein there is a separate transaction domain between the source system and the replica system.
8. The system of claim 1 , wherein there is a separate metadata domain between the source system and the replica system.
9. The system of claim 1 , wherein there are different software binary versions between the source system and the replica system.
10. A computer-implemented method for real-time table replication of a database transaction to a replica table, comprising:
receiving, by at least one processor, a replication log entry and an associated transaction commit log entry, the replication log entry and the associated transaction commit log entry together representing a database transaction to be replayed to a row at a replica table, the replication log entry having a row-ID value and the row at the replica table having a row-ID value;
dispatching, by the at least one processor, the replication log entry to a parallel log replayer and the associated transaction commit log entry to a transaction commit log replayer;
comparing, by the at least one processor, the row-ID value of the replication log entry to the row-ID value of the row at the replica table;
replaying, by the at least one processor, the replication log entry at the parallel log replayer based on the comparison; and
committing, by the at least one processor, the database transaction to the replica table by replaying the associated transaction commit log entry at the transaction log replayer, wherein the database transaction is associated with row-level parallel replay having transactional consistency and Data Dictionary Language (“DDL”) replication and reconstruction of a DDL statement at the replica system is associated with one or multiple metadata update log entries, and further wherein push-based and early log shipping reduce propagation delay between a source system and a replica system.
11. The method of claim 10 , wherein multiple replication object granularities are supported, including at least one of: (i) a set of tables, (ii) a table, (iii) a sub-table, (iv) one or more columns, and (v) one or more partitions.
12. The method of claim 10 , wherein replication having a topology from multiple distinct remote source systems is supported as N-to-1 replication.
13. The method of claim 10 , wherein replication having a topology from to multiple distinct remote replica systems is supported as 1-to-N replication.
14. The method of claim 10 , wherein replication having a topology with a replica table being a source of another replica table is supported as chain replication.
15. The method of claim 10 , wherein in-memory log replication does not rely on a store-and-forward mechanism.
16. The method of claim 10 , wherein there is a separate transaction domain between the source system and the replica system.
17. The method of claim 10 , wherein there is a separate metadata domain between the source system and the replica system.
18. The method of claim 10 , wherein there are different software binary versions between the source system and the replica system.