Distributed journal for high performance continuous data protection
A set of data changes to a storage associated with a source system is received. One or more data logs among a plurality of data logs stored in different nodes of a storage system is dynamically selected based at least in part on a dynamic analysis of metrics of the different nodes of the storage system. The at least one data change of the set of data changes are logged in the one or more selected data logs.
1 . A method, comprising:
receiving a set of data changes to a storage associated with a source system;
dynamically selecting one or more data logs among a plurality of data logs stored in respective nodes of a storage system based at least in part on a dynamic analysis of metrics of the respective nodes of the storage system; and
logging at least one data change of the set of data changes in the one or more selected data logs.
2 . The method of claim 1 , wherein the metrics of the respective nodes of the storage system include one or more of an ingest queue length or a central processing unit (CPU) utilization.
3 . The method of claim 1 , wherein dynamically selecting the one or more data logs among the plurality of data logs further includes analyzing the metrics of the respective nodes of the storage system.
4 . The method of claim 1 , wherein dynamically selecting the one or more data logs among the plurality of data logs further comprises determining whether a node that has received the set of data changes is capable of ingesting the set of data changes at a particular rate and/or logging the set of data changes in a manner that satisfies one or more of a specified recovery point objective (RPO) or a specified recovery time objective (RTO).
5 . The method of claim 4 , wherein logging the data changes in the one or more selected data logs comprises logging the set of data changes in a data log associated with the node that has received the set of data changes based on a determination that the node that has received the set of data changes is capable of one or more of ingesting the set of data changes at the particular rate and/or logging the set of data changes in the manner that satisfies the one or more of the specified RPO or the specified RTO.
6 . The method of claim 4 , wherein dynamically selecting the one or more data logs among the plurality of data logs comprises selecting the one or more data logs among the plurality of data logs stored in the respective nodes of the storage system based on a determination that at least one node of the nodes of the storage system is not capable of one or more of ingesting the set of data changes at a particular rate or logging the set of data changes in the manner that satisfies the one or more of the specified RPO or the specified RTO.
7 . The method of claim 1 , wherein the one or more data logs includes the plurality of data logs stored in respective nodes of the storage system.
8 . The method of claim 1 , wherein the one or more data logs includes a data log associated with the node that has received the set of data changes.
9 . The method of claim 1 , wherein the one or more data logs excludes a data log associated with the node that has received the set of data changes.
10 . The method of claim 1 , wherein dynamically selecting the one or more data logs among the plurality of data logs comprises dividing the set of data changes into a plurality of subsets and assigning the subsets to the one or more selected data logs, respectively.
11 . The method of claim 10 , wherein a number of subsets in the plurality of subsets is selected to satisfy the one or more of a specified recovery point objective (RPO) or a specified recovery time objective (RTO).
12 . The method of claim 1 , wherein logging the at least one data change of the set of data changes in the one or more selected data logs includes logging the set of data changes.
13 . The method of claim 1 , wherein logging the at least one data change of the set of data changes in the one or more selected data logs includes logging a subset of the set of data changes.
14 . The method of claim 1 , further comprising:
receiving an indication to update a locator register log; and
recording, based on the indication, in the locator register log a reference to a portion of the one or more selected data logs associated with storing the data changes.
15 . The method of claim 14 , further comprising:
recording, based on determining an update to a locator register log is a consistent update, in the locator register log a reference to a portion of the one or more selected data logs associated with storing the data changes.
16 . The method of claim 14 , further comprising:
storing, based on determining an update to a locator register log is an inconsistent update, the update in a buffer.
17 . Non-transitory computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to:
receive a set of data changes to a storage associated with a source system;
dynamically select one or more data logs among a plurality of data logs stored in respective nodes of a storage system based at least in part on a dynamic analysis of metrics of the respective nodes of the storage system; and
log at least one data change of the set of data changes in the one or more selected data logs.
18 . The media of claim 17 , wherein the metrics of the respective nodes of the storage system include one or more of an ingest queue length or a central processing unit (CPU) utilization.
19 . The media of claim 17 , wherein to dynamically select the one or more data logs among the plurality of data logs, the instructions cause the one or more processors to determine whether a node that has received the set of data changes is capable of ingesting the set of data changes at a particular rate and/or logging the set of data changes in a manner that satisfies one or more of a specified recovery point objective (RPO) or a specified recovery time objective (RTO).
20 . A system, comprising:
memory comprising instructions;
one or more processors having access to the memory, the one or more processors configured to execute the instructions to cause the one or more processors to:
receive a set of data changes to a storage associated with a source system;
dynamically select one or more data logs among a plurality of data logs stored in respective nodes of a storage system based at least in part on a dynamic analysis of metrics of the respective nodes of the storage system; and
log at least one data change of the set of data changes in the one or more selected data logs.