Scalable cloud backup
Implementations are provided for scalable cloud backup. A coordinator process can manage worker processes on nodes to package file system data that is targeted for cloud backup into node local upload objects. File data can be arranged into distinct block offsets of the node local upload object. A set of metadata tables can be generated that characterize each file that is backed up as well as file block location information for each data block. The node local upload objects can be uploaded to a cloud service provider. The set of metadata tables generated by the worker process can be coalesced into a global set of metadata tables that describe the data that has been backed up. In one implementation, after an initial cloud backup has occurred, a snapshot service of the file system can be used to incrementally backup blocks of the file that have been changed.
1. A method comprising:
maintaining a distributed file system operable within a cluster of nodes wherein files and directories of the distributed file system are associated with a unique logical inode (“LIN”) and a set of data blocks;
establishing a cloud backup policy wherein the cloud backup policy is associated with a backup set of files and directories of the distributed file system;
initiating a cloud backup coordinator process on a coordinator node among the cluster of nodes;
initiating a set of worker processes by the cloud backup coordinator process wherein a worker process among the set of worker processes is assigned to each node in the cluster of nodes, wherein the set of worker processes are in communication with the cloud backup coordinator process;
assigning at least one of the backup set of files and directories to each worker processes in the set of worker processes;
for each worker process in the set of worker processes:
packaging at least one assigned backup set of files and directories into a node local upload object wherein the packaging includes arranging the set of data blocks associated with the at least one backup set of files and directories into the node local upload object and generating a set of metadata tables associated with the node local upload object;
in response to at least one of the node local upload object reaching an object capacity or the worker process finishing packing the at least one backup set of files and directories into the node local upload object, uploading the node local object and the set of metadata tables associated with the node local object to a cloud storage provider; and
coalescing the set of metadata tables associated with each uploaded node local object into a cloud hosted metadata table and a cloud hosted file block location table.
2. The method of claim 1 , wherein the cloud hosted metadata table includes an entry for each backed up file and wherein each file entry includes at least a cloud backup policy name, a LIN, a file path, a snapshot identifier, a timestamp, a set of file permissions, a file type, a group identifier, a user identifier, a last modification time, a parent LIN, and a file size.
3. The method of claim 1 , wherein the cloud hosted file block location table maps sets of a fixed number of blocks from each backed up file to a set of cloud object locations.
4. The method of claim 1 , further comprising:
In response to detecting at least one changed file and directory in the backup set of files and directories, assigning by the coordinator process the at least one changed file and directory to worker processes in the set of worker processes;
Packaging the at least one changed file and directory into a changed node local update object wherein the packaging includes arranging a set of changed data blocks associated with the at least one changed file and directory and generating a set of changed metadata tables associated with the changed node local upload object; and
coalescing the set of changed metadata tables with the cloud hosted metadata table and the cloud hosted file block location table.
5. The method of claim 4 , further comprising:
determining an uploaded node local upload object contains no active reference within the cloud hosted metadata table and the cloud hosted file block location table;
in response to the determining, delating the uploaded node local upload object from the cloud storage provider.
6. The method of claim 1 , further comprising:
determining an uploaded node local upload object has been moved to cold storage;
requesting the cloud storage provider restore the uploaded node local upload object from cold storage;
in response to the uploaded node local upload object being restored from cold storage, restoring a set of files associated with the uploaded node local upload object, wherein the set of files associated with the uploaded node local object are determined based on the cloud hosted metadata table and the cloud hosted file block location table.
7. A system comprising at least one storage device and at least one hardware processor configured to:
maintaining a distributed file system operable within a cluster of nodes wherein files and directories of the distributed file system are associated with a unique logical inode (“LIN”) and a set of data blocks;
establishing a cloud backup policy wherein the cloud backup policy is associated with a backup set of files and directories of the distributed file system;
initiating a cloud backup coordinator process on a coordinator node among the cluster of nodes;
initiating a set of worker processes by the cloud backup coordinator process wherein a worker process among the set of worker processes is assigned to each node in the cluster of nodes, wherein the set of worker processes are in communication with the cloud backup coordinator process;
assigning at least one of the backup set of files and directories to each worker processes in the set of worker processes;
for each worker process in the set of worker processes:
packaging at least one assigned backup set of files and directories into a node local upload object wherein the packaging includes arranging the set of data blocks associated with the at least one backup set of files and directories into the node local upload object and generating a set of metadata tables associated with the node local upload object;
in response to at least one of the node local upload object reaching an object capacity or the worker process finishing packing the at least one backup set of files and directories into the node local upload object, uploading the node local object and the set of metadata tables associated with the node local object to a cloud storage provider; and
coalescing the set of metadata tables associated with each uploaded node local object into a cloud hosted metadata table and a cloud hosted file block location table.
8. The system of claim 7 , wherein the cloud hosted metadata table includes an entry for each backed up file and wherein each file entry includes at least a cloud backup policy name, a LIN, a file path, a snapshot identifier, a timestamp, a set of file permissions, a file type, a group identifier, a user identifier, a last modification time, a parent LIN, and a file size.
9. The system of claim 7 , wherein the cloud hosted file block location table maps sets of a fixed number of blocks from each backed up file to a set of cloud object locations.
10. The system of claim 7 , further configured to:
In response to detecting at least one changed file and directory in the backup set of files and directories, assigning by the coordinator process the at least one changed file and directory to worker processes in the set of worker processes;
Packaging the at least one changed file and directory into a changed node local update object wherein the packaging includes arranging a set of changed data blocks associated with the at least one changed file and directory and generating a set of changed metadata tables associated with the changed node local upload object; and
coalescing the set of changed metadata tables with the cloud hosted metadata table and the cloud hosted file block location table.
11. The system of claim 10 , further configured to:
determining an uploaded node local upload object contains no active reference within the cloud hosted metadata table and the cloud hosted file block location table;
in response to the determining, delating the uploaded node local upload object from the cloud storage provider.
12. The system of claim 7 , further configured to:
determining an uploaded node local upload object has been moved to cold storage;
requesting the cloud storage provider restore the uploaded node local upload object from cold storage;
in response to the uploaded node local upload object being restored from cold storage, restoring a set of files associated with the uploaded node local upload object, wherein the set of files associated with the uploaded node local object are determined based on the cloud hosted metadata table and the cloud hosted file block location table.
13. A non-transitory computer readable medium with program instructions stored thereon to cause a computer to perform the following acts:
maintaining a distributed file system operable within a cluster of nodes wherein files and directories of the distributed file system are associated with a unique logical inode (“LIN”) and a set of data blocks;
establishing a cloud backup policy wherein the cloud backup policy is associated with a backup set of files and directories of the distributed file system;
initiating a cloud backup coordinator process on a coordinator node among the cluster of nodes;
initiating a set of worker processes by the cloud backup coordinator process wherein a worker process among the set of worker processes is assigned to each node in the cluster of nodes, wherein the set of worker processes are in communication with the cloud backup coordinator process;
assigning at least one of the backup set of files and directories to each worker processes in the set of worker processes;
for each worker process in the set of worker processes:
packaging at least one assigned backup set of files and directories into a node local upload object wherein the packaging includes arranging the set of data blocks associated with the at least one backup set of files and directories into the node local upload object and generating a set of metadata tables associated with the node local upload object;
in response to at least one of the node local upload object reaching an object capacity or the worker process finishing packing the at least one backup set of files and directories into the node local upload object, uploading the node local object and the set of metadata tables associated with the node local object to a cloud storage provider; and
coalescing the set of metadata tables associated with each uploaded node local object into a cloud hosted metadata table and a cloud hosted file block location table.
14. The non-transitory computer readable medium of claim 13 , wherein the cloud hosted metadata table includes an entry for each backed up file and wherein each file entry includes at least a cloud backup policy name, a LIN, a file path, a snapshot identifier, a timestamp, a set of file permissions, a file type, a group identifier, a user identifier, a last modification time, a parent LIN, and a file size.
15. The non-transitory computer readable medium of claim 13 , wherein the cloud hosted file block location table maps sets of a fixed number of blocks from each backed up file to a set of cloud object locations.
16. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further cause a computer to perform the following acts:
In response to detecting at least one changed file and directory in the backup set of files and directories, assigning by the coordinator process the at least one changed file and directory to worker processes in the set of worker processes;
Packaging the at least one changed file and directory into a changed node local update object wherein the packaging includes arranging a set of changed data blocks associated with the at least one changed file and directory and generating a set of changed metadata tables associated with the changed node local upload object; and
coalescing the set of changed metadata tables with the cloud hosted metadata table and the cloud hosted file block location table.
17. The non-transitory computer readable medium of claim 16 , with program instructions stored thereon to further cause a computer to perform the following acts:
determining an uploaded node local upload object contains no active reference within the cloud hosted metadata table and the cloud hosted file block location table;
in response to the determining, delating the uploaded node local upload object from the cloud storage provider.
18. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further cause a computer to perform the following acts:
determining an uploaded node local upload object has been moved to cold storage;
requesting the cloud storage provider restore the uploaded node local upload object from cold storage;
in response to the uploaded node local upload object being restored from cold storage, restoring a set of files associated with the uploaded node local upload object, wherein the set of files associated with the uploaded node local object are determined based on the cloud hosted metadata table and the cloud hosted file block location table.