Efficient file system parsing using snap based replication
View Patent ↗Systems and methods are disclosed for efficiently performing backups using snapshots. A splitter receives write operations and replicates the write operations to a data protection appliance and to a primary storage. The data embodied in the replicated writes is stored in a high-speed storage in a snapshot over a time increment. The snapshot represents data blocks that have changed since a previous time increment. A list of changed blocks is kept in high-speed storage that corresponds to the snapshot blocks. The snapshot blocks are associated with files and file system structure by (a) traversing a file system structure (FSS) for a file system or (b) by traversing the list of changed blocks and looking up changed blocks that are in inode format in the FSS, an inode tree structure, or directly. The snapshot, list of changed blocks, and list of associated changed files are transferred from high-speed storage to backup storage.
1. A computer-implemented method comprising:
generating, in a high-speed storage, a second disk snapshot of blocks of a storage that have changed since a first, previous, snapshot of blocks of the storage;
generating, in the high-speed storage, a first list of the changed blocks in the second disk snapshot;
generating a second list of changed files corresponding to the first list of changed blocks by traversing only those blocks of a file system that are identified in the first list of changed blocks contained in high-speed storage;
transferring the changed blocks from high-speed storage to a backup storage using a differential virtual machine disk;
transferring metadata identifying each of the one or more files corresponding to the changed blocks from high-speed storage to the backup storage using a delta marker stream; and
transferring the second list of changed files from high-speed storage to a backup storage.
2. The method of claim 1 , wherein the first and second snapshots are generated from write operations that are replicated asynchronously to both a production virtual machine disk and to a storage appliance having a virtual machine for data protection.
3. The method of claim 1 , wherein the high-speed storage comprises at least one of:
random access memory;
flash memory; or
a bit storage based on a change of bulk resistance, in conjunction with a stackable cross-gridded data access array.
4. The method of claim 1 wherein transferring the changed blocks comprises:
reading the changed blocks from the high-speed storage;
performing one of:
(a) arranging the changed blocks in an order determined at least in part by a file system organization; and
transferring the changed blocks to storage in the determined order;
or
(b) transferring the changed blocks to storage; and
transferring metadata to storage describing an order of the changed blocks based at least in part on the file system organization.
5. The method of claim 1 , wherein generating the second list of changed files comprises:
traversing a file system structure (FSS) of a file system, the FSS having inode entries corresponding to one or more files and a directory structure of the file system;
in response to determining that an inode of the FSS corresponds to a block in the first list of changed blocks:
traversing one or more inodes from the inode that corresponds to the changed block, to identify a file name of each of one or more files having changed blocks;
adding each of the one or more identified files to the second list of changed files corresponding to one or more blocks in the first list of changed blocks.
6. The method of claim 1 , wherein generating the second list of changed files comprises:
traversing the blocks in the first list of changed blocks;
in response to determining that a traversed block comprises an inode of a file or a directory:
traversing one or more inodes of the file or directory from the inode that corresponds to the changed block, to identify a file having the changed block;
adding the identified file to the second list of changed files corresponding to the first list of changed blocks.
7. A non-transitory computer readable medium, programmed with executable instructions that, when executed by a processing system, perform operations comprising:
generating, in a high-speed storage, a second disk snapshot of blocks of a storage that have changed since a first, previous, snapshot of blocks of the storage;
generating, in the high-speed storage, a first list of the changed blocks in the second disk snapshot;
generating a second list of changed files corresponding to the first list of changed blocks by traversing only those blocks of a file system that are identified in the first list of changed blocks contained in high-speed storage;
transferring the changed blocks from high-speed storage to the backup storage using a differential virtual machine disk;
transferring metadata identifying each of the one or more files corresponding to the changed blocks from high-speed storage to the backup storage using a delta marker stream; and
transferring the second list of changed files from high-speed storage to a backup storage.
8. The medium of claim 7 , wherein the first and second snapshots are generated from write operations that are replicated asynchronously to both a production virtual machine disk and to a storage appliance having a virtual machine for data protection.
9. The medium of claim 7 , wherein the high-speed storage comprises at least one of:
random access memory;
flash memory; or
a bit storage based on a change of bulk resistance, in conjunction with a stackable cross-gridded data access array.
10. The medium of claim 7 wherein transferring the changed blocks comprises:
reading the changed blocks from the high-speed storage;
performing one of:
(a) arranging the changed blocks in an order determined at least in part by a file system organization; and
transferring the changed blocks to storage in the determined order;
or
(b) transferring the changed blocks to storage; and
transferring metadata to storage describing an order of the changed blocks based at least in part on the file system organization.
11. The medium of claim 7 , wherein generating the second list of changed files comprises:
traversing a file system structure (FSS) of a file system, the FSS having inode entries corresponding to one or more files and a directory structure of the file system;
in response to determining that an inode of the FSS corresponds to a block in the first list of changed blocks:
traversing one or more inodes of the FSS from the inode that corresponds to the changed block, to identify a file name of each of one or more files having changed blocks;
adding each of the one or more identified files to the second list of changed files corresponding to one or more blocks in the first list of changed blocks.
12. The medium of claim 7 , wherein generating the second list of changed files comprises:
traversing the blocks in the first list of changed blocks;
in response to determining that a traversed block comprises an inode of a file or a directory:
traversing one or more inodes of the file or directory from the inode that corresponds to the changed block, to identify a file having the changed block;
adding the identified file to the second list of changed files corresponding to the first list of changed blocks.
13. A processing system, comprising a hardware processor coupled to a memory programmed with executable instructions, that when executed by the processing system, perform operations comprising:
generating, in a high-speed storage, a second disk snapshot of blocks of a storage that have changed since a first, previous, snapshot of blocks of the storage;
generating, in the high-speed storage, a first list of the changed blocks in the second disk snapshot;
generating a second list of changed files corresponding to the first list of changed blocks by traversing only those blocks of a file system that are identified in the first list of changed blocks contained in high-speed storage;
transferring the changed blocks from high-speed storage to the backup storage using a differential virtual machine disk;
transferring metadata identifying each of the one or more files corresponding to the changed blocks from high-speed storage to the backup storage using a delta marker stream; and
transferring the second list of changed files, from high-speed storage to a backup storage.
14. The system of claim 13 , wherein the first and second snapshots are generated from write operations that are replicated asynchronously to both a production virtual machine disk and to a storage appliance having a virtual machine for data protection.
15. The system of claim 13 , wherein the high-speed storage comprises at least one of:
random access memory;
flash memory; or
a bit storage based on a change of bulk resistance, in conjunction with a stackable cross-gridded data access array.
16. The system of claim 13 wherein transferring the changed blocks comprises:
reading the changed blocks from the high-speed storage;
performing one of:
(a) arranging the changed blocks in an order determined at least in part by a file system organization; and
transferring the changed blocks to storage in the determined order;
or
(b) transferring the changed blocks to storage; and
transferring metadata to storage describing an order of the changed blocks based at least in part on the file system organization.
17. The system of claim 13 , wherein generating the second list of changed files comprises:
traversing a file system structure (FSS) of a file system, the FSS having inode entries corresponding to one or more files and a directory structure of the file system;
in response to determining that an inode of the FSS corresponds to a block in the first list of changed blocks:
traversing one or more inodes from the inode that corresponds to the changed block, to identify a file name of each of one or more files having changed blocks;
adding each of for the one or more identified files names to the second list of changed files corresponding to one or more blocks in the first list of changed blocks.
18. The system of claim 13 , wherein generating the second list of changed files comprises:
traversing the blocks in the first list of changed blocks;
in response to determining that a traversed block comprises an inode of a file or a directory:
traversing one or more inodes of the file or directory from the inode that corresponds to the changed block, to a file having the changed block;
adding the identified file to the second list changed files corresponding to the first list of changed blocks.