Cloud-native global file system with direct-to-cloud migration
A cloud-native global file system used to provide primary file storage for enterprise data is augmented to provide “direct-to-cloud” (D 2 C) migration of an initial data set. In addition to providing a filer, the service provider configures a direct-to-cloud migration tool, typically as a downloadable virtual machine (VM), that is executed in association with the hardware of the enterprise filesystem whose data set is being migrated. The migration tool reads the existing filesystem, transforms the data into a format consistent with that done by a filer, and pushes the result (a single version) directly to an object store in the cloud. In a first pass (run), typically a large bulk of the data is moved and, depending on how the migration tool is configured, one or more additional passes may then be carried out. When migration is complete (or substantially complete), the filer then is activated to complete the migration (if need be) and take over the go-forward operations.
1. A method to migrate a local file system to a cloud-based object store associated with a cloud-based storage service provider, comprising:
for each of two or more portions of the local file system, migrating a subtree of the portion by:
retrieving directories and files of the subtree and packing information therein according to a data format associated with the cloud-based object store to create packed data;
uploading the packed data to the cloud-based object store, the packed data having an associated handle, the handle representing a directory manifest for a root of migrated data of the subtree, wherein the root is associated with a versioned filesystem;
receiving data identifying a volume and a location therein at which the root is to be stitched into the volume; and
linking the subtree at the location to complete a direct-to-cloud migration of the subtree.
2. The method as described in claim 1 wherein the subtrees for the two or more portions are migrated concurrently.
3. The method as described in claim 2 wherein the direct-to-cloud migration for the subtrees is providing according to a quality-of-service.
4. The method as described in claim 3 wherein the quality-of-service utilizes bandwidth shaping.
5. The method as described in claim 4 wherein the bandwidth shaping is configurable.
6. The method as described in claim 1 further including applying a transformation rule to remap the portion during the direct-to-cloud migration of the subtree.
7. A method to migrate a local file system to a cloud-based object store associated with a cloud-based storage service provider, comprising:
associating portions of the local file system to a cluster of instances of a migration tool; and
executing the instances of the migration tool to migrate the portions of the local file system to the cloud-based object store;
wherein executing an instance of the migration tool migrates a subtree of the portion by:
retrieving directories and files of the subtree and packing information therein according to a data format associated with the cloud-based object store to create packed data;
uploading the packed data to the cloud-based object store, the packed data having an associated handle, the handle representing a directory manifest for a root of migrated data of the subtree, wherein the root is associated with a versioned filesystem;
receiving data identifying a volume and a location therein at which the root is to be stitched into the volume; and
linking the subtree at the location to complete a direct-to-cloud migration of the subtree.
8. The method as described in claim 7 further including applying a quality-of-service to the cluster of instances.
9. The method as described in claim 7 wherein the local file system is located on physical hardware on-premises in an enterprise computing environment.
10. The method as described in claim 7 wherein the instance of the migration tool also implements deduplication of the packed data.