Data migration in a distributive file system
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans a plurality of the storage devices. The plurality of computing devices maintains metadata that maps each failure resilient address space to one of the plurality of computing devices. The metadata is grouped into buckets. Each bucket is stored in the backend of a computing device. Data may be migrated from an external file system to the plurality of storage devices using inode stubs to represent directories and files of the external file system. As the contents of the external file system are copied, the inode stubs are replaced with real inodes.
1. A system comprising:
a plurality of storage devices; and
a computing device comprising a backend, wherein:
the backend comprises a bucket that is operable to generate a first data structure of a first data structure type and a second data structure of a second data structure type,
the first data structure is operable to direct a data access to an external storage that is outside of the plurality of storage devices, and
the second data structure is operable to direct the data access to a failure-protected stripe of a plurality of a failure-protected stripes, wherein each failure-protected stripe of the plurality of failure-protected stripes comprises a plurality of storage blocks distributed across a unique group of storage devices of the plurality of storage devices, and wherein the plurality of storage blocks in the plurality of storage devices are determined before data is migrated from the external storage to the plurality of storage devices.
2. The system of claim 1 , wherein the storage blocks of the failure-protected stripe are distributed across at least two storage devices of the plurality of storage devices.
3. The system of claim 1 , wherein at least one storage block of the plurality of storage blocks of the failure-protected stripe is reserved for error correction data that is associated with the other storage blocks of the plurality of storage blocks of the failure-protected stripe.
4. The system of claim 1 , wherein the second data structure comprises an inode and one or more extents that point to the plurality of storage blocks of the failure-protected stripe.
5. The system of claim 1 , wherein the first data structure type is used when data is copied from the external storage to the plurality of storage devices.
6. The system of claim 5 , wherein the second data structure is used after data has been copied from the external storage to the plurality of storage devices.
7. The system of claim 5 , wherein a storage block of the plurality of storage blocks is designated as a sparse file if the storage block is not used for migrated data or associated error correction data.
8. The system of claim 1 , wherein the first data structure comprises a stub inode.
9. The system of claim 8 , wherein the stub inode corresponds to a file in the external storage.
10. The system of claim 8 , wherein the stub inode corresponds to a directory in the external storage.
11. The system of claim 1 , wherein the system comprises a network interface card (NIC).
12. A method comprising:
generating, by a bucket in a computing device backend, a first data structure comprises a stub inode;
designating, by the bucket in the computing device backend, a stripe, wherein the stripe is one of a plurality of stripes, and wherein each stripe of the plurality of stripes comprises a plurality of storage blocks distributed across a unique group of storage devices of a plurality of storage devices;
migrating data from an external storage to one or more storage blocks of the plurality of storage blocks in the plurality of storage devices, wherein data migration comprises generating and storing error correction data associated with the data from the external storage, and wherein the plurality of storage blocks in the plurality of storage devices are determined before data is migrated from the external storage to the plurality of storage devices;
accessing data from the external storage during data migration, via the stub inode in the first data structure; and
accessing data from the stripe after data migration, via a second data structure.
13. The method of claim 12 , wherein the plurality of storage blocks of the stripe are distributed across at least two storage devices of the plurality of storage devices.
14. The method of claim 12 , wherein at least one storage block of the plurality of storage blocks of the stripe is reserved for a block of error correction data that is generated according to the other storage blocks of the plurality of storage blocks of the stripe.
15. The method of claim 12 , wherein the second data structure comprises an inode and one or more extents that point to the plurality of storage blocks of the stripe.
16. The method of claim 12 , wherein a storage block of the plurality of storage blocks is designated as a sparse file if the storage block is not used for migrated data or associated error correction data.
17. The method of claim 12 , wherein the stub inode corresponds to a file in the external storage.
18. The method of claim 12 , wherein the stub inode corresponds to a directory in the external storage.
19. The method of claim 12 , wherein the computing device backend is on a network interface card (NIC).
20. The method of claim 12 , wherein the stripe is a failure-protected stripe.