Fileset partitioning for data storage and management
In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.
1 . A method, comprising:
receiving, by a data management and storage (DMS) cluster that stores data from a compute infrastructure comprising a plurality of machines, a request to take a snapshot of a fileset from a first machine of the plurality of machines, wherein the DMS cluster has N available DMS nodes, N being a variable representing an integer quantity of available DMS nodes, and wherein partitions for the fileset are undefined prior to the request to take the snapshot being received;
obtaining, by the DMS cluster, fileset metadata associated with the fileset;
determining, by the DMS cluster, whether to divide the fileset into a plurality of partitions based on the fileset metadata prior to taking the snapshot of the fileset and in response to the request to take the snapshot;
defining, by the DMS cluster, based on determining to divide the fileset in response to the request to take the snapshot, and prior to taking the snapshot of the fileset, the plurality of partitions for the fileset based on the fileset metadata;
creating, by the DMS cluster, to take the snapshot of the fileset, a plurality of jobs for taking snapshots associated with the plurality of partitions, wherein a job of the plurality of jobs corresponds to a single partition of the plurality of partitions;
executing in parallel, by the N available DMS nodes of the DMS cluster, the plurality of jobs to obtain a plurality of snapshots associated with the plurality of partitions, wherein a respective job obtains a respective snapshot of the plurality of snapshots for a respective partition of the plurality of partitions;
storing respective snapshots of the plurality of snapshots associated with the plurality of partitions in respective DMS nodes of a distributed data store implemented across the N available DMS nodes of the DMS cluster, wherein the N available DMS nodes are physically separated from one another; and
restoring in parallel, two or more partitions of the plurality of partitions of the fileset using the respective snapshots of the two or more partitions stored in the respective DMS nodes.
2 . The method of claim 1 , wherein determining whether to divide the fileset into the plurality of partitions is based on a size of the fileset.
3 . The method of claim 1 , wherein determining whether to divide the fileset into the plurality of partitions is based on a predetermined partition size.
4 . The method of claim 1 , wherein the first machine is a virtual machine, and the fileset comprises a virtual disk file.
5 . The method of claim 1 , wherein the DMS cluster maintains information identifying a correspondence between the plurality of partitions and the fileset.
6 . The method of claim 5 , wherein the information associates a first partition of the plurality of partitions to a first DMS node of the N available DMS nodes, and the first DMS node stores data associated with the first partition.
7 . The method of claim 1 , wherein the fileset includes multiple files.
8 . A data management and storage (DMS) system, wherein the DMS system comprises:
one or more processors; and
one or more non-transitory computer-readable storage media, operatively coupled with at least one of the one or more processors, comprising instructions that, when executed by the one or more processors, cause the DMS system to:
receive a request to take a snapshot of a fileset from a first machine of a plurality of machines, wherein the DMS system comprises a DMS cluster that stores data from a compute infrastructure, the compute infrastructure comprises the plurality of machines, and the DMS cluster has N available DMS nodes, N being a variable representing an integer quantity of available DMS nodes, and wherein partitions for the fileset are undefined prior to the request to take the snapshot being received;
obtain fileset metadata associated with the fileset;
determine whether to divide the fileset into a plurality of partitions based on the fileset metadata prior to taking the snapshot of the fileset and in response to the request to take the snapshot;
define, based on determining to divide the fileset in response to the request to take the snapshot and prior to taking the snapshot of the fileset, the plurality of partitions for the fileset based on the fileset metadata;
create, to take the snapshot of the fileset, a plurality of jobs for taking snapshots associated with the plurality of partitions, wherein a job of the plurality of jobs corresponds to a single partition of the plurality of partitions;
execute in parallel, by the N available DMS nodes of the DMS cluster, the plurality of jobs to obtain a plurality of snapshots associated with the plurality of partitions, wherein a respective job obtains a respective snapshot of the plurality of snapshots for a respective partition of the plurality of partitions;
store respective snapshots of the plurality of snapshots associated with the plurality of partitions in respective DMS nodes of a distributed data store implemented across the N available DMS nodes of the DMS cluster, wherein the N available DMS nodes are physically separated from one another; and
restore in parallel, two or more partitions of the plurality of partitions of the fileset using the respective snapshots of the two or more partitions stored in the respective DMS nodes.
9 . The DMS system of claim 8 , wherein determining whether to divide the fileset into the plurality of partitions is based on a size of a file of the fileset indicated by the fileset metadata.
10 . The DMS system of claim 8 , wherein determining whether to divide the fileset into the plurality of partitions is based on a predetermined partition size.
11 . The DMS system of claim 8 , wherein the first machine is a virtual machine, and the fileset comprises a virtual disk file.
12 . The DMS system of claim 8 , wherein the DMS cluster maintains information identifying a correspondence between the plurality of partitions and the fileset.
13 . The DMS system of claim 12 , wherein the information associates a first partition of the plurality of partitions to a first DMS node of the N available DMS nodes, and the first DMS node stores data associated with the first partition.
14 . The DMS system of claim 8 , wherein the fileset includes multiple files.
15 . One or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors, cause a data management and storage (DMS) system to:
receive a request to take a snapshot of a fileset from a first machine of a plurality of machines, wherein the DMS system comprises a DMS cluster that stores data from a compute infrastructure, the compute infrastructure comprises the plurality of machines, and the DMS cluster has N available DMS nodes, N being a variable representing an integer quantity of available DMS nodes, and wherein partitions for the fileset are undefined prior to the request to take the snapshot being received;
obtain fileset metadata associated with the fileset;
determine whether to divide the fileset into a plurality of partitions based on the fileset metadata prior to taking the snapshot of the fileset and in response to the request to take the snapshot;
define, based on determining to divide the fileset in response to the request to take the snapshot and prior to taking the snapshot of the fileset, the plurality of partitions for the fileset based on the fileset metadata;
create, to take the snapshot of the fileset, a plurality of jobs for taking snapshots associated with the plurality of partitions, wherein a job of the plurality of jobs corresponds to a single partition of the plurality of partitions;
execute in parallel, by the N available DMS nodes of the DMS cluster, the plurality of jobs to obtain a plurality of snapshots associated with the plurality of partitions, wherein a respective job obtains a respective snapshot of the plurality of snapshots for a respective partition of the plurality of partitions;
store respective snapshots of the plurality of snapshots associated with the plurality of partitions in respective DMS nodes of a distributed data store implemented across the N available DMS nodes of the DMS cluster, wherein the N available DMS nodes are physically separated from one another; and
restore in parallel, two or more partitions of the plurality of partitions of the fileset using the respective snapshots of the two or more partitions stored in the respective DMS nodes.
16 . The one or more non-transitory computer-readable storage media of claim 15 , wherein determining whether to divide the fileset into the plurality of partitions is based on a size of the fileset.
17 . The one or more non-transitory computer-readable storage media of claim 15 , wherein determining whether to divide the fileset into the plurality of partitions is based on a predetermined partition size.
18 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the first machine is a virtual machine, and the fileset comprises a virtual disk file.
19 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the DMS cluster maintains information identifying a correspondence between the plurality of partitions and the fileset.
20 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the fileset includes multiple files.