Snapshot replication operations based on incremental block change tracking
A system according to certain aspects improves the process of performing snapshot replication operations (e.g., maintaining a mirror copy of primary data at a secondary location by generating snapshots of the primary data). The system can collect and maintain cumulative block-level changes to the primary data after each sub-interval of a plurality of sub-intervals between the snapshots. When a snapshot is generated, any changes to the primary data not reflected in the cumulative block-level changes are identified based on the snapshot and transmitted to the secondary location along with the cumulative block-level changes. By the time the snapshot is generated, some or all of the changes to the primary data associated with the given snapshot have already been included in the cumulative block-level changes, thereby reducing the time and computing resources spent to identify and collect the changes for transmission to the secondary location.
1. A method for performing a snapshot replication operation in a data storage system, the method comprising:
with a data agent executing on a client computing device (i) comprising one or more hardware processors and (ii) in networked communication with a destination computing device,
performing a first snapshot replication at a first time period for primary data associated with the client computing device, the first snapshot replication comprising transmitting first change data to the destination computing device, the first change data indicating one or more changes to the primary data that have occurred since a most recent snapshot replication associated with the client computing device that precedes the first time period;
for each sub-interval of a plurality of sub-intervals subsequent to the first time period,
querying a filter driver to receive a change bitmap, the change bitmap indicating one or more block-level changes to the primary data that have occurred since a most recent query to the filter driver, and
based on the received change bitmap, updating cumulative block-level changes stored in an intermediate staging area, the cumulative block-level changes indicating one or more changes to the primary data that occurred between the first time period and the most recent query to the filter driver; and
subsequent to the plurality of sub-intervals, performing a second snapshot replication for the primary data at a second time period, the second snapshot replication comprising transmitting second change data to the destination computing device, the second change data including (i) the cumulative block-level changes stored in the intermediate staging area and (ii) one or more changes to the primary data since the most recent query to the filter driver that precedes the second time period.
2. The method of claim 1 , further comprising:
determining that the intermediate staging area does not include all changes to the primary data that have occurred since the first snapshot replication;
querying the filter driver to receive another change bitmap indicating one or more additional changes to the primary data that are not stored in the intermediate staging area; and
transmitting the one or more additional changes to the destination computing device along with the cumulative block-level changes stored in the intermediate staging area.
3. The method of claim 1 , wherein the change bitmap indicates one or more blocks in the primary data that have changed since the most recent query to the filter driver, the method further comprising storing copies of the one or more blocks indicated by the change bitmap in the intermediate staging area.
4. The method of claim 1 , further comprising:
determining that the client computing device is using an amount of computing resources that is less than a threshold amount; and
based on determining that the client computing device is using an amount of computing resources that is less than the threshold amount, querying the filter driver to receive the change bitmap.
5. The method of claim 1 , further comprising:
accessing historical metric data indicative of usage of computing resources by the client computing device;
identifying, based on the historical metric data, a subsequent time period during which the client computing device is expected to use an amount of computing resources that is less than a threshold amount; and
querying the filter driver to receive the change bitmap at the subsequent time period.
6. The method of claim 1 , further comprising:
determining that an amount of computing resources used by the client computing device has exceeded a threshold amount; and
increasing a length of each sub-interval such that the filter driver is queried less frequently.
7. The method of claim 1 , wherein the intermediate staging area is remotely located from the client computing device such that copying data from the intermediate staging area to the destination computing device does not affect performance of the client computing device.
8. The method of claim 1 , wherein a frequency at which the filter driver is queried is substantially greater than a frequency at which snapshot replications are performed for the primary data associated with the client computing device.
9. The method of claim 1 , further comprising:
determining that a first block indicated by the change bitmap to have changed since the most recent query to the filter driver is stored in the intermediate staging area; and
updating the first block stored in the intermediate staging area to reflect the change since the most recent query to the filter driver.
10. The method of claim 1 , further comprising:
determining that a first block indicated by the change bitmap to have changed since the most recent query to the filter driver is not stored in the intermediate staging area;
accessing the first block in the primary data associated with the client computing device; and
storing the first block in the intermediate staging area.
11. A system for performing a data protection operation in a data storage system, the system comprising:
a first computing device comprising computer hardware; and
a second computing device comprising computer hardware and in networked communication with the first computing device, wherein the second computing device is configured to:
perform a first data protection operation at a first time period for primary data associated with the first computing device, the first data protection operation comprising transmitting first change data to the first computing device, the first change data indicating one or more changes to the primary data that have occurred since a most recent data protection operation associated with the first computing device that precedes the first time period;
for each sub-interval of a plurality of sub-intervals subsequent to the first time period,
query a filter driver to receive a change bitmap, the change bitmap indicating one or more block-level changes to the primary data that have occurred since a most recent query to the filter driver, and
based on the received change bitmap, update cumulative block-level changes stored in an intermediate staging area, the cumulative block-level changes indicating one or more changes to the primary data that occurred between the first time period and the most recent query to the filter driver; and
subsequent to the plurality of sub-intervals, perform a second data protection operation for the primary data at a second time period, the second data protection operation comprising transmitting second change data to the first computing device, the second change data including (i) the cumulative block-level changes stored in the intermediate staging area and (ii) one or more changes to the primary data since the most recent query to the filter driver that precedes the second time period.
12. The system of claim 11 , wherein the second computing device is further configured to:
determine that the intermediate staging area does not include all changes to the primary data that have occurred since the first data protection operation;
query the filter driver to receive another change bitmap indicating one or more additional changes to the primary data that are not stored in the intermediate staging area; and
transmit the one or more additional changes to the first computing device along with the cumulative block-level changes stored in the intermediate staging area.
13. The system of claim 11 , wherein the change bitmap indicates one or more blocks in the primary data that have changed since the most recent query to the filter driver, the second computing device is further configured to store copies of the one or more blocks indicated by the change bitmap in the intermediate staging area.
14. The system of claim 11 , wherein the second computing device is further configured to:
determine that the second computing device is using an amount of computing resources that is less than a threshold amount; and
based on the determination that the second computing device is using an amount of computing resources that is less than the threshold amount, query the filter driver to receive the change bitmap.
15. The system of claim 11 , wherein the second computing device is further configured to:
access historical metric data indicative of usage of computing resources by the second computing device;
identify, based on the historical metric data, a subsequent time period during which the second computing device is expected to use an amount of computing resources that is less than a threshold amount; and
query the filter driver to receive the change bitmap at the subsequent time period.
16. The system of claim 11 , further comprising:
determine that an amount of computing resources used by the client computing device has exceeded a threshold amount; and
increase a length of each sub-interval such that the filter driver is queried less frequently.
17. The system of claim 11 , wherein the intermediate staging area is remotely located from the second computing device such that copying data from the intermediate staging area to the first computing device does not affect performance of the second computing device.
18. The system of claim 11 , wherein a frequency at which the filter driver is queried is substantially greater than a frequency at which data protection operations are performed for the primary data associated with the second computing device.
19. The system of claim 11 , wherein the second computing device is further configured to:
determine that a first block indicated by the change bitmap to have changed since the most recent query to the filter driver is stored in the intermediate staging area; and
update the first block stored in the intermediate staging area to reflect the change since the most recent query to the filter driver.
20. The system of claim 11 , wherein the second computing device is further configured to:
determine that a first block indicated by the change bitmap to have changed since the most recent query to the filter driver is not stored in the intermediate staging area;
access the first block in the primary data associated with the second computing device; and
store the first block in the intermediate staging area.