Data migration and namespace management across domains in multi-domain clustered file systems

1. A method of facilitating data migration in a multi-domain network, comprising:

partitioning a clustered network into a number of separate domains, wherein each domain contains resources including processors, memory, storage devices, as well as data deduplication and compression services, wherein the clustered network comprises a Santorini filesystem network that stores file system metadata in a distributed key-value (KV) store and file data in an object store;

presenting a single global namespace for the network is presented to a user or backup process by combining all of the namespaces in the domains;

providing a multi-domain manager component having an analytics engine to facilitate analytics-driven migration of data among nodes in different domain;

providing a plurality of storage devices in a multi-tier arrangement, wherein tiers of the multi-tier arrangement provide storage with different cost and performance characteristics to satisfy different service level agreement (SLA) requirements;

creating files comprising migrating data with a backup SLA attribute to map each file to a corresponding storage tier;

mapping the files to similarity groups through a hashing function based on a hash of some data spanned by the data of each of the files, each similarity group assigned a similarity group ID;

assigning an integer value tier ID to each tier, each having its own KV store; and

performing a deduplicated migration operation within a single similarity group ID and tier ID KV pair so that deduplication does not span tiers for a similarity group.

2. The method of claim 1 wherein each domain contains data and metadata represented by a namespace in the form of a managed directory tree (Mtree).

3. The method of claim 1 further comprising: distributing nodes of the clustered network into different domains with each domain containing unique Mtree data and metadata.

4. The method of claim 3 further comprising migrating data among the nodes by re-distributing data files through one of similarity indexes, greedy iterative processes, integer linear programming (ILP) algorithms, hierarchical clustering processes.

5. The method of claim 1 wherein a similarity index is used, the method further comprising:

generating a plurality of file cluster candidates for the files, each of the plurality of file cluster candidates comprising some of the files;

determining a similarity index is determined for each of the plurality of file cluster candidates based on similarity of the files comprised in the file cluster candidate; and

generating a ranked recipe list comprising a plurality of recipes, wherein each recipe is associated with one of the plurality of file cluster candidates, comprises a destination data node for the associated file cluster candidate, and is associated with a deduplication space savings determined based on a total file size and the similarity index of the associated file cluster candidate.

6. The method of claim 5 further comprising:

sorting the plurality of recipes in the ranked recipe list based on data movement cost-adjusted deduplication space savings; and

moving at least some of the plurality of files between the plurality of data nodes based on the recipes in the ranked recipe list to improve deduplication space savings in the network, such that file movements based on highest ranked recipes in the ranked recipe list are performed first.

7. The method of claim 6 wherein the single global namespace is utilized by a deduplication backup system to backup and restore data as part of the migration from one or nodes in the clustered network.

8. A system for migrating data in a multi-domain network, comprising:

a clustered network partitioned into a number of separate domains, wherein each domain contains resources including processors, memory, storage devices, as well as data deduplication and compression services, wherein the clustered network comprises a Santorini filesystem network that stores file system metadata in a distributed key-value (KV) store and file data in an object store;

a single global namespace for the network presented to a user or backup process by combining all of the namespaces in the domains;

a multi-domain manager component having an analytics engine to facilitate analytics-driven migration of data among nodes in different domains;

a plurality of storage devices in a multi-tier arrangement, wherein tiers of the multi-tier arrangement provide storage with different cost and performance characteristics to satisfy different service level agreement (SLA) requirements, wherein files comprising migrating data are created with a backup SLA attribute to map each file to a corresponding storage tier;

a component mapping the files to similarity groups through a hashing function based on a hash of some data spanned by the data of each of the files, each similarity group assigned a similarity group ID, and assigning an integer value tier ID to each tier, each having its own KV store; and

a deduplicated backup server performing a deduplicated migration operation within a single similarity group ID and tier ID KV pair so that deduplication does not span tiers for a similarity group.

9. The system of claim 8 wherein each domain contains data and metadata represented by a namespace in the form of a managed directory tree (Mtree), the system further distributing nodes of the clustered network into different domains with each domain containing unique Mtree data and metadata.

10. The system of claim 9 further comprising an analytics engine facilitating migration of data among the nodes by re-distributing data files through one of similarity indexes, greedy iterative processes, integer linear programming (ILP) algorithms, hierarchical clustering processes.

11. The system of claim 10 wherein the analytics engine utilizes similarity index-based migration that:

generates a plurality of file cluster candidates for the files, each of the plurality of file cluster candidates comprising some of the files;

determines a similarity index is determined for each of the plurality of file cluster candidates based on similarity of the files comprised in the file cluster candidate; and

generates a ranked recipe list comprising a plurality of recipes, wherein each recipe is associated with one of the plurality of file cluster candidates, comprises a destination data node for the associated file cluster candidate, and is associated with a deduplication space savings determined based on a total file size and the similarity index of the associated file cluster candidate;

sorts the plurality of recipes in the ranked recipe list based on data movement cost-adjusted deduplication space savings; and

moves at least some of the plurality of files between the plurality of data nodes based on the recipes in the ranked recipe list to improve deduplication space savings in the network, such that file movements based on highest ranked recipes in the ranked recipe list are performed first.

12. The system of claim 11 wherein the clustered network comprises part of a deduplication backup system performing backup and restore operations as the data migration for the nodes.