Parallel replication across formats
Disclosed herein are system, method, and computer readable device embodiments for parallel replication of databases across row-store and column-store table formats. An embodiment operates by maintaining a replication log and a storage-level recovery log formatted according to separate respective log formats, logging a record-level SQL execution result, and replicating at least one database table from a primary server to at least one replica server. The primary server and the at least one replica server may each be configured to store data according to one table format of a row-store table format and a column-store table format, such that the primary server's table format is different from the at least one replica server's table format.
1. A computer-implemented method for parallel cross-format replication of databases, comprising:
maintaining, by at least one processor, a replication log and a storage-level recovery log, wherein the replication log and the storage-level recovery log are each formatted according to separate respective log formats;
logging, by the at least one processor, a record-level SQL execution result in at least one of the replication log and the storage-level recovery log; and
replicating, by the at least one processor, at least one database table from a primary server to at least one replica server based at least in part on the replication log,
wherein the primary server is configured to use a row-store table format, and the at least one replica server is configured to use a column-store table format, and
wherein the replicating is performed asynchronously, according to an early log-shipping protocol, with respect to a specific entry of the replication log before completion of a transaction corresponding to the specific entry.
2. The computer-implemented method of claim 1 , further comprising:
replaying, by the at least one processor, at least one entry from at least one of the replication log and the storage-level recovery log, wherein the at least one entry comprises the record-level SQL execution result; and
bypassing, by the at least one processor, an SQL execution engine upon the replaying of the at least one entry comprising the record-level SQL execution result.
3. The computer-implemented method of claim 1 , wherein:
at least one of the primary server and the at least one replica server is configured to store data according to at least one partitioning scheme.
4. The computer-implemented method of claim 3 , wherein:
the primary server's partitioning scheme is different from the at least one replica server's partitioning scheme.
5. The computer-implemented method of claim 1 , wherein:
the logging further comprises storing a snapshot timestamp associated with at least one entry committed to at least one of the replication log and the storage-level recovery log.
6. A system, comprising:
a memory; and
at least one processor coupled to the memory and configured to perform operations comprising:
maintaining a replication log and a storage-level recovery log, wherein the replication log and the storage-level recovery log are each formatted according to separate respective log formats;
logging a record-level SQL execution result in at least one of the replication log and the storage-level recovery log; and
replicating at least one database table from a primary server to at least one replica server based at least in part on the replication log,
wherein the primary server and the at least one replica server are each configured to store data according to one table format of a row-store table format and a column-store table format, and
wherein the replicating is performed asynchronously, according to an early log-shipping protocol, with respect to a specific entry of the replication log before completion of a transaction corresponding to the specific entry.
7. The system of claim 6 , wherein the processor is further configured to:
replay at least one entry from at least one of the replication log and the storage-level recovery log, wherein the at least one entry comprises the record-level SQL execution result; and
bypass an SQL execution engine upon the replaying of the at least one entry comprising the record-level SQL execution result.
8. The system of claim 6 , wherein:
at least one of the primary server and the at least one replica server is configured to store data according to at least one partitioning scheme.
9. The system of claim 8 , wherein:
the primary server's partitioning scheme is different from the at least one replica server's partitioning scheme.
10. The system of claim 6 , wherein:
the logging further comprises storing a snapshot timestamp associated with at least one entry committed to at least one of the replication log and the storage-level recovery log.
11. A computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
maintaining a replication log and a storage-level recovery log, wherein the replication log and the storage-level recovery log are each formatted according to separate respective log formats;
logging a record-level SQL execution result in at least one of the replication log and the storage-level recovery log; and
replicating at least one database table from a primary server to at least one replica server based at least in part on the replication log,
wherein the primary server is configured to use a column-store table format, and the at least one replica server is configured to use row-store table format, and
wherein the replicating is performed asynchronously, according to an early log-shipping protocol, with respect to a specific entry of the replication log before completion of a transaction corresponding to the specific entry.
12. The computer-readable device of claim 11 , the operations further comprising:
replaying at least one entry from at least one of the replication log and the storage-level recovery log, wherein the at least one entry comprises the record-level SQL execution result; and
bypassing an SQL execution engine upon the replaying of the at least one entry comprising the record-level SQL execution result.
13. The computer-readable device of claim 11 , wherein:
at least one of the primary server and the at least one replica server is configured to store data according to at least one partitioning scheme.
14. The computer-readable device of claim 13 , wherein:
the primary server's partitioning scheme is different from the at least one replica server's partitioning scheme.
15. The computer-readable device of claim 11 , wherein:
the logging further comprises storing a snapshot timestamp associated with at least one entry committed to at least one of the replication log and the storage-level recovery log.
16. The system of claim 6 , wherein:
at least one of the replication log or the storage-level recovery log comprises a data manipulation language (DML) log entry for replay with the transaction corresponding to the specific entry.
17. The system of claim 16 , the operations further comprising:
replaying the DML log entry in parallel with another transaction corresponding to another specific entry, following a multi-version concurrency control (MVCC) protocol.
18. The system of claim 6 , wherein:
the replicating is performed synchronously with respect to another replica in the system.
19. The system of claim 6 , wherein:
the replicating further comprises adaptive query routing, in accordance with an adaptive query-routing scheme comprising at least one rule.
20. The system of claim 19 , wherein:
the at least one rule comprises exclusion of a given replica in response to logged metadata indicating that the given replica is unavailable.