ASYNCHRONOUS REPLICATION VIA FILE-ORIENTED SNAPSHOTS
A replication policy management and scheduling technique of a Files service is configured for deployment in multi-site data replication environments. The technique involves policy management for data distribution and/or data consolidation (concentration) where multiple sources, e.g., file system (FS) clusters or sites, replicate the data to a central target FS site, e.g., in a spoke and hub arrangement typical of remote office/branch office environments. The technique also involves creation and configuration of a main replication policy by a customer at a central resource manager configured to interact and manage the FS sites, each of which includes one or more file server virtual machines.
1 . A method comprising:
generating a base snapshot of a distributed share used to replicate data from a source file system (FS) site over a network to a target FS site, wherein the base snapshot is a file system-level snapshot employing metadata of a virtualized file system to preserve a file system hierarchy during snapshot generation;
generating one or more subsequent file-system level snapshots of the distributed share at the source FS site;
performing a hierarchical construct level compare between the base and subsequent snapshots to identify altered inodes resulting from metadata corresponding to one or more changed files;
computing deltas of the files wherein the deltas are associated with blocks of the changed files identified via traversal of the blocks using the altered inodes; and
replicating the data corresponding to the deltas of the changed files to the target FS site.
2 . The method of claim 1 , wherein the metadata relates to one of modification timestamps of the files or file length.
3 . (canceled)
4 . The method of claim 1 , wherein replicating further comprises spanning the data across parallel synchronization threads of the source FS site by partitioning the data into predetermined chunks.
5 . The method of claim 1 , further comprising reusing the computed deltas to replicate the data across parallel synchronization threads from a portion of the source FS over the network to another target FS site.
6 . The method of claim 1 , further comprising maintaining a replay list for files failing replication, wherein the failed files are replicated during a next replication cycle.
7 . The method of claim 1 , further comprising scanning a source directory from the base snapshot at the source FS site to generate a file list of the data based on mapping locations of the distributed share at the target FS site.
8 . The method of claim 1 , wherein replicating the data corresponding to the deltas of the changed files is screened by a file-oriented pathname filter.
9 . The method of claim 1 , wherein computing the deltas of the changed files comprises computing a modified directories/files list employed during incremental replication to replicate the data corresponding to the deltas of the changed files.
10 . The method of claim 9 , further comprising applying customer-provided filters to the modified directories/files list to (i) mark directories to which the changed files belong and (ii) eliminate types of files excluded from replication.
11 . A non-transitory computer readable medium including program instructions for execution on a processor, the program instructions configured to:
generate a base snapshot of a distributed share used to replicate data from a source file system (FS) site over a network to a target FS site, wherein the base snapshot is a file system-level snapshot employing metadata of a virtualized file system to preserve a file system hierarchy during snapshot generation;
generate one or more subsequent file-system level snapshots of the distributed share at the source FS site;
perform a hierarchical construct level compare between the base and subsequent snapshots to identify altered inodes resulting from metadata corresponding to one or more changed files;
compute deltas of the changed files wherein the deltas are associated with blocks of the changed files identified via traversal of the blocks using the altered inodes; and
replicate the data corresponding to the deltas of the changed files to the target FS site.
12 . The non-transitory computer readable medium of claim 11 , wherein the metadata relates to one of modification timestamps of the files or file length.
13 . (canceled)
14 . The non-transitory computer readable medium of claim 11 , wherein the program instructions configured to replicate are further configured to span the data across parallel synchronization threads of the source FS site by partitioning the data into predetermined chunks.
15 . The non-transitory computer readable medium of claim 11 , wherein the program instructions are further configured to reuse the computed deltas to replicate the data across parallel synchronization threads from a portion of the source FS over the network to the remote target FS site.
16 . The non-transitory computer readable medium of claim 11 , wherein the program instructions are further configured to maintain a replay list for files failing replication, wherein the failed files are replicated during a next replication cycle.
17 . The non-transitory computer readable medium of claim 11 , wherein the program instructions are further configured to scan a source directory from the base snapshot at the source FS site to generate a file list of the data based on mapping locations of the distributed share at the target FS site.
18 . The non-transitory computer readable medium of claim 11 , wherein the program instructions configured to replicate the data corresponding to the deltas of the changed files are further configured to screen the replicated data by a file-oriented pathname filter.
19 . The non-transitory computer readable medium of claim 11 , wherein the program instructions configured to compute the deltas of the files are further configured to compute a modified directories/files list employed during incremental replication to replicate the data corresponding to the deltas of the changed files.
20 . The non-transitory computer readable medium of claim 19 , wherein the program instructions are further configured to apply customer-provided filters to the modified directories/files list to (i) mark directories to which the changed files belong and (ii) eliminate types of files excluded from replication.
21 . An apparatus comprising:
a network connecting one or more nodes of source file system (FS) sites to one or more target FS sites, each node having a processor configured to execute program instructions to:
generate a base snapshot of a distributed share used to replicate data from a source file system (FS) site over a network to a target FS site, wherein the base snapshot is a file system-level snapshot employing metadata of a virtualized file system to preserve a file system hierarchy during snapshot generation;
generate one or more subsequent file-system level snapshots of the distributed share at the source FS site;
perform a hierarchical construct level compare between the base and subsequent snapshots to identify altered inodes resulting from metadata corresponding to one or more changed files;
compute deltas of the changed files wherein the deltas are associated with blocks of the changed files identified via traversal of the blocks using the altered inodes; and
replicate the data corresponding to the deltas of the changed files to the target FS site.
22 . The apparatus of claim 21 , wherein the metadata relates to one of modification timestamps of the files or file length.
23 . (canceled)
24 . The apparatus of claim 21 , wherein the processor configured to execute program instructions to replicate is further configured to execute program instructions to span the data across parallel synchronization threads of the source FS site by partitioning the data into predetermined chunks.
25 . The apparatus of claim 21 , wherein the processor is further configured to execute program instructions to reuse the computed deltas to replicate the data across parallel synchronization threads from a portion of the source FS over the remote target FS site.
26 . The apparatus of claim 21 , wherein the processor is further configured to execute program instructions to maintain a replay list for files failing replication, wherein the failed files are replicated during a next replication cycle.
27 . The apparatus of claim 21 , wherein the processor is further configured to execute program instructions to scan a source directory from the base snapshot at the source FS site to generate a file list of the data based on mapping locations of the distributed share at the target FS site.
28 . The apparatus of claim 21 , wherein the processor configured to execute program instructions to replicate the data corresponding to the deltas of the changed files is further configured to execute program instructions to screen the replicated data by a file-oriented pathname filter.
29 . The apparatus of claim 21 , wherein the processor configured to execute program instructions to compute the deltas of the changed files is further configured to execute program instructions to compute a modified directories/files list employed during incremental replication to replicate the data corresponding to the deltas of the changed files.
30 . The apparatus of claim 29 , wherein the processor is further configured to execute program instructions to apply customer-provided filters to the modified directories/files list to (i) mark directories to which the changed files belong and (ii) eliminate types of files excluded from replication.