Maintaining a current representation of a transactional database at an analytical database with partial re-synchronization
Systems and methods for managing representations of transactional tables of a transactional database in an analytical database are disclosed. In response to an event causing change replication for a given one of the table representations to be paused or re-started, a separate replication process is initialized for the given table representation and used to catch the given table representation up to a same point-in-time with regard to replicated changes as other ones of the table replications maintained at the analytical database. Once caught-up, the replication processes are combined and a unified replication process proceeds to apply other changes to the table representations maintained at the analytical database. In this way an individual table representation or set of table representations can gracefully fail and recover without interrupting change replication for other table representations for other tables that are replicated from the transactional database to the analytical database.
1 . A system, comprising:
one or more computing devices configured to implement a transactional database, wherein the one or more computing devices are configured to:
maintain a plurality of tables; and
write transactional changes made to the plurality of tables to a change-data-capture log; and
one or more compute nodes organized into a node cluster and configured to implement an analytical database, wherein the one or more compute nodes of the node cluster are configured to:
initiate a first replication process to apply transactional changes of the change-data-capture log to respective ones of a plurality of representations of the plurality of tables of the transactional database that are maintained at the analytical database;
in response to an instance of re-starting or pausing of replication for a given one of the plurality of table representations, initiate a second replication process for the given table representation, wherein the second replication process applies transactional changes from the change-data-capture log from a different point in time than the first replication process is currently applying transactional changes from; and
in response to determining the first replication process and the second replication process have converged to a same point in time with regard to transactional changes being applied from the change-data-capture log, merge the first and second replication processes into a combined replication process for applying the transactional changes to the respective ones of the plurality of table representations.
2 . The system of claim 1 , wherein to initiate the second replication process, the one or more compute nodes of the node cluster are configured to:
access a snapshot of a given one of the tables of the transactional database that corresponds to the given table representation being maintained at the analytical database;
re-initialize, at the analytical database, the given table representation from the snapshot; and
apply transactional changes from the change-data-capture log, for transactional changes made to the given table at the transactional database subsequent to a point-in-time of the snapshot, to the given table representation maintained at the analytical database.
3 . The system of claim 2 , wherein the change-data-capture log is transported from the transactional database to the analytical database via a plurality of checkpoints stored to a data storage service accessible by the analytical database.
4 . The system of claim 2 , wherein the snapshot is transported from the transactional database to the analytical database via storage of the data storage service.
5 . The system of claim 1 , wherein the one or more compute nodes of the node cluster are configured to:
detect a failure associated with the given table representation; and
initiate the instance of re-starting of replication for the given table representation in response to detecting the failure.
6 . The system of claim 1 , wherein the one or more compute nodes of the node cluster are configured to:
identify an un-supported domain definition language (DDL) command included in the change-data-capture log; and
initiate the instance of re-starting of replication for the given table representation, wherein a snapshot that is used in re-starting the replication of the given table representation is a snapshot of a corresponding table of the transactional database for which the DDL command has been applied.
7 . The system of claim 1 , wherein the one or more compute nodes of the node cluster are further configured to:
for a particular one of the plurality of table representations,
pause replication for a first portion of the particular table representation while continuing to perform replication for a second portion of the particular table representation; and
in response to determining replication for the first portion of the particular table representation and replication for the second portion of the particular table representation have converged to a same point in time with regard to transactional changes being applied from the change-data-capture log, merge replication for the first portion of the particular table representation and replication for the second portion of the particular table representation into a combined replication process for the particular table representation.
8 . A method, comprising:
performing a first replication process to apply transactional changes of a change-data-capture log to respective ones of a plurality of representations of a plurality of tables of a transactional database that are maintained at an analytical database;
in response to an instance of re-starting or pausing of replication for a given one of the plurality of table representations, initiating a second replication process for the given table representation; and
in response to determining the first replication process and the second replication process have converged to a same point in time with regard to transactional changes being applied from the change-data-capture log, merging the first and second replication processes into a combined replication process for applying the transactional changes to the respective ones of the plurality of tables.
9 . The method of claim 8 , wherein said initiating the second replication process comprises:
re-initializing, at the analytical database, the given table representation from a snapshot; and
applying transactional changes from the change-data-capture log for transactional changes made subsequent to a point-in-time of the snapshot.
10 . The method of claim 8 , further comprising:
transporting portions of the change-data-capture log from the transactional database to the analytical database via a plurality of checkpoint files.
11 . The method of claim 10 , wherein the checkpoint files are transported via storage of the checkpoint files at a data storage service accessible by the analytical database.
12 . The method of claim 8 , further comprising:
transporting the transactional changes of the change-data-capture log from the transactional database to the analytical database via a streaming service.
13 . The method of claim 8 , further comprising:
detecting a failure associated with the given table representation; and
initiating the instance of re-starting of replication for the given table representation in response to detecting the failure.
14 . The method of claim 8 , further comprising:
identifying an un-supported domain definition language (DDL) command included in the change-data-capture log; and
initiating the instance of re-starting of replication for the given table representation, wherein a snapshot that is used in re-starting the replication of the given table representation is a snapshot of a corresponding table of the transactional database for which the DDL command has been applied.
15 . The method of claim 8 , further comprising:
modifying, performed by a control plane of the analytical database, one or more resource allocations between resources allocated to perform the first replication process and resource allocated to the second replication process, wherein the one or more resource allocations are selected to promote convergence of the first replication process and the second replication process to the same point in time with regard to transactional changes being applied from the change-data-capture log.
16 . The method of claim 8 , further comprising:
pausing performance of queries on the given table representation until the second replication process converges to the same point in time as the first replication process with regard to transactional changes being applied from the change-data-capture log; and
continuing to perform queries for other ones of the plurality of table representations while query performance against the given table is paused.
17 . One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, cause the one or more processors to:
perform a first replication process to apply transactional changes of a change-data-capture log to respective ones of a plurality of representations of a plurality of tables of a transactional database that are maintained at an analytical database;
in response to an instance of re-starting or pausing of replication for a given one of the plurality of table representations, initiate a second replication process for the given table representation; and
in response to determining the first replication process and the second replication process have converged to a same point in time with regard to transactional changes being applied from the change-data-capture log, merge the first and second replication processes into a combined replication process for applying the transactional changes to the respective ones of the plurality of tables.
18 . The one or more non-transitory computer-readable storage media of claim 17 , wherein the first replication process applies transactional changes with later logical sequence numbers than the second replication process prior to the convergence to applying transactional changes corresponding to the same point in time.
19 . The one or more non-transitory computer-readable storage media of claim 17 , wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to:
detect a failure associated with the given table representation; and
initiate re-starting of replication for the given table representation in response to detecting the failure.
20 . The one or more non-transitory computer-readable storage media of claim 17 , wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to:
detect a domain definition language (DDL) command associated with the given table representation; and
initiate re-starting of replication for the given table representation from a snapshot in response to detecting the DDL command.