Replication policy management and scheduling technique of a files service
A replication policy management and scheduling technique of a Files service is configured for deployment in multi-site data replication environments. The technique involves policy management for data distribution and/or data consolidation (concentration) where multiple sources, e.g., file system (FS) clusters or sites, replicate the data to a central target FS site, e.g., in a spoke and hub arrangement typical of remote office/branch office environments. The technique also involves creation and configuration of a main replication policy by a customer at a central resource manager configured to interact and manage the FS sites, each of which includes one or more file server virtual machines.
1. A method comprising:
translating a main replication policy for replicating data from source file system (FS) sites to one or more target FS sites into a plurality of sub-policies for execution at corresponding source FS sites;
creating, for each sub-policy, a storage location mapping of top-level directories (TLDs) for the source FS sites to corresponding TLDs of the one or more target FS sites;
determining files of the source FS sites that have changed blocks based on block-level snapshot deltas at the source FS sites;
filtering, at each source FS site, the changed files according to a customer-provided filter, wherein the filter is file-oriented and operates on a pathname according to a syntax and grammar; and
executing the sub-policies at the source FS sites to transmit the changed blocks of the filtered changed files to the one or more target FS sites.
2. The method of claim 1 , wherein determining the files of the source FS sites that have changed blocks further comprises traversing blocks of files having changed directory entry metadata.
3. The method of claim 1 , wherein the sub-policies are executed concurrently on two or more of the source FS sites.
4. The method of claim 1 , wherein determining the files of the source FS sites that have changed blocks further comprises:
creating a first list of the determined files for filtering according to the customer-provided filter;
creating a second list from the filtered files of the first list that fail replication to the one or more target FS sites; and
appending the second list to the filtered changed files for replication during a subsequent execution of the sub-policy at the corresponding source FS site.
5. The method of claim 1 , wherein the transmission of the changed blocks of the filtered changed files to the one or more target FS sites occurs via a file-level remote synchronization protocol.
6. The method of claim 1 , wherein execution of the sub-policies concentrate data from the source FS sites to the one or more target FS sites.
7. The method of claim 6 , wherein the data concentration is in a form of a hub and spoke arrangement from the source FS sites acting as spokes to the target FS site acting as a hub.
8. The method of claim 1 , wherein execution of the sub-policies form a peer-to-peer environment for 2-way synchronization between two FS sites.
9. The method of claim 1 , wherein the main policy is created at a central resource manager configured to interact and manage the source and target FS sites.
10. The method of claim 1 , wherein the customer-provided filter is denoted by pathname according to a regular expression syntax.
11. A non-transitory computer readable medium including program instructions for execution on a processor, the program instructions configured to:
translate a main replication policy for replicating data from source file system (FS) sites to one or more target FS sites into a plurality of sub-policies for execution at corresponding source FS sites;
create, for each sub-policy, a storage location mapping of top-level directories (TLDs) for the source FS sites to corresponding TLDs of the one or more target FS sites;
determine files of the source FS sites that have changed blocks based on block-level snapshot deltas at the source FS sites;
filter, at each source FS site, the changed files according to a customer-provided filter, wherein the filter is file-oriented and operates on a pathname according to a syntax and grammar; and
execute the sub-policies at the source FS sites to transmit the changed blocks of the filtered changed files to the one or more target FS sites.
12. The non-transitory computer readable medium of claim 11 , wherein the program instructions configured to determine the files of the source FS sites that have changed blocks are further configured to traverse blocks of files having changed directory entry metadata.
13. The non-transitory computer readable medium of claim 11 , wherein the sub-policies are executed concurrently on two or more of the source FS sites.
14. The non-transitory computer readable medium of claim 11 , wherein the program instructions configured to determine the files of the source FS sites that have changed blocks are further configured to:
create a first list of the determined files for filtering according to the customer-provided filter;
create a second list from the filtered files of the first list that fail replication to the one or more target FS sites; and
append the second list to the filtered changed files for replication during a subsequent execution of the sub-policy at the corresponding source FS site.
15. The non-transitory computer readable medium of claim 11 , wherein the transmission of the changed blocks of the filtered changed files to the one or more target FS sites occurs via a file-level remote synchronization protocol.
16. The non-transitory computer readable medium of claim 11 , wherein execution of the sub-policies concentrate data from the source FS sites to the one or more target FS sites.
17. The non-transitory computer readable medium of claim 16 , wherein the data concentration is in a form of a hub and spoke arrangement from the source FS sites acting as spokes to the target FS site acting as a hub.
18. The non-transitory computer readable medium of claim 11 , wherein execution of the sub-policies form a peer-to-peer environment for 2-way synchronization between two FS sites.
19. The non-transitory computer readable medium of claim 11 , wherein the main policy is created at a central resource manager configured to interact and manage the source and target FS sites.
20. The non-transitory computer readable medium of claim 11 , wherein the customer-provided filter is denoted by pathname according to a regular expression syntax.
21. An apparatus comprising:
a network connecting nodes of source file system (FS) sites to one or more target FS sites, each node having a processor configured to execute program instructions to:
translate a main replication policy for replicating data from the source FS sites to the one or more target FS sites into a plurality of sub-policies for execution at corresponding source FS sites;
create, for each sub-policy, a storage location mapping of top-level directories (TLDs) for the source FS sites to corresponding TLDs of the one or more target FS sites;
determine files of the source FS sites that have changed blocks based on block-level snapshot deltas at the source FS sites;
filter, at each source FS site, the changed files according to a customer-provided filter, wherein the filter is file-oriented and operates on a pathname according to a syntax and grammar; and
execute the sub-policies at the source FS sites to transmit the changed blocks of the filtered changed files to the one or more target FS sites.
22. The apparatus of claim 21 , wherein the processor configured to execute program instructions to determine the files of the source FS sites that have changed blocks is further configured to execute program instructions to traverse blocks of files having changed directory entry metadata.
23. The apparatus of claim 21 , wherein the sub-policies are executed concurrently on two or more of the source FS sites.
24. The apparatus of claim 21 , wherein the processor configured to execute program instructions to determine the files of the source FS sites that have changed blocks is further configured to execute program instructions to:
create a first list of the determined files for filtering according to the customer-provided filter;
create a second list from the filtered files of the first list that fail replication to the one or more target FS sites; and
append the second list to the filtered changed files for replication during a subsequent execution of the sub-policy at the corresponding source FS site.
25. The apparatus of claim 21 , wherein the transmission of the changed blocks of the filtered changed files to the one or more target FS sites occurs via a file-level remote synchronization protocol.
26. The apparatus of claim 21 , wherein execution of the sub-policies concentrate data from the source FS sites to the one or more target FS sites.
27. The apparatus of claim 26 , wherein the data concentration is in a form of a hub and spoke arrangement from the source FS sites acting as spokes to the target FS site acting as a hub.
28. The apparatus of claim 21 , wherein execution of the sub-policies form a peer-to-peer environment for 2-way synchronization between two FS sites.
29. The apparatus of claim 21 , wherein the main policy is created at a central resource manager configured to interact and manage the source and target FS sites.
30. The apparatus of claim 21 , wherein the customer-provided filter is denoted by pathname according to a regular expression syntax.