Change block tracking for transfer of data for backups
In one approach, a set of data blocks or files is tracked for changes between snapshots. This may be done by a file system filter running in kernel mode. The data blocks or files that are tagged as unchanged are not transferred to backup because there is no need to update since the last backup. Other data blocks and files may be first tested for change, for example by comparing digital fingerprints of the current data versus the previously backed up data, before transferring to backup.
1. A method for data management, comprising:
transmitting, from a data management system (DMS) to a compute infrastructure, signaling that instructs the compute infrastructure to take a snapshot of a fileset of the compute infrastructure;
receiving, at the DMS from the compute infrastructure, tracking data generated during a tracking session to track which data blocks of the fileset are write accessed during the tracking session, wherein the tracking data indicates which data blocks of the fileset were write accessed during the tracking session, and wherein the tracking session begins before the compute infrastructure took a snapshot of the fileset previous to the snapshot and ends in response to the compute infrastructure taking the snapshot; and
determining, based at least in part on the generated tracking data whether to transfer the indicated data blocks from the compute infrastructure to the DMS for backup of the snapshot.
2. The method of claim 1 , wherein determining whether to transfer the indicated data blocks from the compute infrastructure to the DMS comprises, for data blocks in the fileset:
determining from the tracking data whether a data block was write accessed during the tracking session; and
if the data block was not write accessed according to the tracking data, then not transferring the data block from the compute infrastructure to the DMS.
3. The method of claim 1 , wherein determining whether to transfer the indicated data blocks from the compute infrastructure to the DMS comprises, for data blocks in the fileset:
determining from the tracking data whether a data block was write accessed during the tracking session; and
if the data block was write accessed according to the tracking data, then:
transferring a digital fingerprint of the previous snapshot of the data block to the compute infrastructure;
causing the compute infrastructure to calculate a digital fingerprint of the data block and to determine whether the digital fingerprints of the data block and of the previous snapshot of the data block are the same; and
if the digital fingerprints are not the same, then transferring the data block from the compute infrastructure to the DMS but, if the digital fingerprints are the same, then not transferring the data block.
4. The method of claim 1 , wherein determining whether to transfer the indicated data blocks from the compute infrastructure to the DMS comprises, for data blocks in the fileset:
determining from the tracking data whether a data block was write accessed during the tracking session; and
if the data block was write accessed according to the tracking data, then transferring the data block from the compute infrastructure to the DMS.
5. The method of claim 1 , wherein the tracking data comprises a bitmap of bits, each bit representing one of the data blocks and indicating whether that data block was write accessed during the tracking session.
6. The method of claim 5 , wherein a size of the data block represented by each bit is configurable.
7. The method of claim 5 , wherein, during the tracking session, the bitmap is stored in kernel space memory.
8. The method of claim 1 , wherein the tracking data comprises a linked list.
9. The method of claim 1 , wherein the tracking data comprises a listing of files in the fileset, and a bitmap of bits for each of said files, each bit representing one of the data blocks in said file and indicating whether that data block was write accessed during the tracking session.
10. The method of claim 1 , wherein a file system filter on the compute infrastructure maintains a list of tracking sessions comprising the tracking session.
11. The method of claim 1 , wherein a file system filter on the compute infrastructure is automatically called by a file system when the file system makes a write access during the tracking session.
12. The method of claim 1 , wherein:
the DMS comprises a DMS cluster of peer DMS nodes, a distributed data store implemented across the peer DMS nodes, and a DMS agent installed on the compute infrastructure;
the previous snapshot is stored in the distributed data store;
the DMS agent determines whether to transfer data blocks from the compute infrastructure to the DMS for backup of the snapshot;
jobs to transfer data blocks from the compute infrastructure to the distributed data store are posted to a job queue accessible by the peer DMS nodes; and
the peer DMS nodes autonomously fetch and execute jobs from the job queue.
13. The method of claim 12 , wherein the DMS agent starts the tracking session and/or stops the tracking session.
14. The method of claim 12 , wherein the DMS agent instructs the compute infrastructure to take the snapshot of the fileset and then ends the tracking session.
15. The method of claim 12 , wherein the DMS agent starts the tracking session and then instructs the compute infrastructure to take the previous snapshot of the fileset.
16. The method of claim 12 , wherein, after the end of the tracking session, the DMS agent receives the tracking data from a file system filer on the compute infrastructure.
17. The method of claim 12 , wherein the DMS agent runs in user mode on the compute infrastructure.
18. A method for data management, comprising:
transmitting, from a data management system (DMS) to a compute infrastructure, signaling that instructs the compute infrastructure to take a snapshot of a fileset of the compute infrastructure;
receiving, at the DMS from the compute infrastructure, tracking data generated during a tracking session to track which data blocks of the fileset are write accessed during the tracking session, wherein the tracking data indicates which data blocks of the fileset were write accessed during the tracking session, and wherein the tracking session begins before the compute infrastructure took a snapshot of the fileset previous to the snapshot and ends in response to the compute infrastructure taking the snapshot;
determining whether a data block in the fileset is currently tagged by the DMS as unchanged based at least in part on whether the data block comprises an incremental difference indicated by the tracking data; and
if the data block is currently tagged as unchanged, then not transferring the data block from the compute infrastructure to the DMS.
19. The method of claim 18 , further comprising:
if the data block is not currently tagged as unchanged, then:
transferring a digital fingerprint of a previous snapshot of the data block to the compute infrastructure;
causing the compute infrastructure to calculate a digital fingerprint of the data block and to determine whether the digital fingerprints of the data block and of the previous snapshot of the data block are the same; and
if the digital fingerprints are not the same, then transferring the data block from the compute infrastructure to the DMS but, if the digital fingerprints are the same, then not transferring the data block.
20. The method of claim 18 , wherein:
the DMS comprises a DMS cluster of peer DMS nodes, a distributed data store implemented across the peer DMS nodes, and a DMS agent installed on the compute infrastructure;
the previous snapshot is stored in the distributed data store;
the DMS agent determines whether to transfer data blocks from the compute infrastructure to the DMS for backup of the snapshot;
jobs to transfer data blocks from the compute infrastructure to the distributed data store are posted to a job queue accessible by the peer DMS nodes; and
the peer DMS nodes autonomously fetch and execute jobs from the job queue.