Scaling single file snapshot performance across clustered system
In some embodiments, a process for restoring a version of a virtual machine using a data storage system comprises identifying a particular version of a virtual machine to be restored, determining a base image from which the particular version may be derived, determining a set of incremental files for generating the particular version, generating a file associated with the particular version using the base image and the set of incremental files, and outputting at least a portion of the file.
1. A method comprising:
sharding a file in a fileset into a plurality of shard files, wherein the plurality of shard files correspond to a plurality of partitions of the fileset, and wherein sharding the file is based at least in part on metadata associated with the file;
distributing the plurality of shard files to a plurality of nodes, wherein at least a first shard file of the plurality of shard files is distributed to a first node of the plurality of nodes and at least a second shard file of the plurality of shard files is distributed to a second node of the plurality of nodes; and
using the plurality of nodes to generate a plurality of snapshots corresponding to the file, wherein at least a first snapshot of the plurality of snapshots is generated at the first node based at least in part on the first shard file and at least a second snapshot of the plurality of snapshots is generated at the second node based at least in part on the second shard file.
2. The method of claim 1 , wherein the plurality of partitions are based at least in part on a logical size of the fileset.
3. The method of claim 1 , wherein sharding the file into the plurality of shard files is based at least in part on a size of the file satisfying a threshold.
4. The method of claim 3 , further comprising:
refraining from sharding a second file in the fileset based at least in part on a size of the second file being below the threshold.
5. The method of claim 1 , wherein the first shard file corresponds to a first partition of the plurality of partitions, wherein the second shard file corresponds to a second partition of the plurality of partitions.
6. The method of claim 1 , further comprising:
scanning the fileset to generate the metadata associated with the file.
7. The method of claim 1 , wherein a hash is included in or appended to a name of the first shard file.
8. An apparatus, comprising:
at least one processor;
memory coupled with the at least one processor; and
instructions stored in the memory and executable by the at least one processor to cause the apparatus to:
shard a file in a fileset into a plurality of shard files, wherein the plurality of shard files correspond to a plurality of partitions of the fileset, and wherein sharding the file is based at least in part on metadata associated with the file;
distribute the plurality of shard files to a plurality of nodes, wherein at least a first shard file of the plurality of shard files is distributed to a first node of the plurality of nodes and at least a second shard file of the plurality of shard files is distributed to a second node of the plurality of nodes; and
use the plurality of nodes to generate a plurality of snapshots corresponding to the file, wherein at least a first snapshot of the plurality of snapshots is generated at the first node based at least in part on the first shard file and at least a second snapshot of the plurality of snapshots is generated at the second node based at least in part on the second shard file.
9. The apparatus of claim 8 , wherein the plurality of partitions are based at least in part on a logical size of the fileset.
10. The apparatus of claim 8 , wherein the instructions are executable by the at least one processor to cause the apparatus to shard the file into the plurality of shard files based at least in part on a size of the file satisfying a threshold.
11. The apparatus of claim 10 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
refrain from sharding a second file in the fileset based at least in part on a size of the second file being below the threshold.
12. The apparatus of claim 8 , wherein the first shard file corresponds to a first partition of the plurality of partitions, wherein the second shard file corresponds to a second partition of the plurality of partitions.
13. The apparatus of claim 8 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
scan the fileset to generate the metadata associated with the file.
14. The apparatus of claim 8 , wherein the instructions are further executable by the at least one processor to cause the apparatus to:
generate a name of the first shard file, the name based at least in part on a hash.
15. A non-transitory computer-readable medium storing code, the code comprising instructions executable by at least one processor to:
shard a file in a fileset into a plurality of shard files, wherein the plurality of shard files correspond to a plurality of partitions of the fileset, and wherein sharding the file is based at least in part on metadata associated with the file;
distribute the plurality of shard files to a plurality of nodes, wherein at least a first shard file of the plurality of shard files is distributed to a first node of the plurality of nodes and at least a second shard file of the plurality of shard files is distributed to a second node of the plurality of nodes; and
use the plurality of nodes to generate a plurality of snapshots corresponding to the file, wherein at least a first snapshot of the plurality of snapshots is generated at the first node based at least in part on the first shard file and at least a second snapshot of the plurality of snapshots is generated at the second node based at least in part on the second shard file.
16. The computer-readable medium of claim 15 , wherein the plurality of partitions are based at least in part on a logical size of the fileset.
17. The computer-readable medium of claim 15 , wherein the instructions are executable by the at least one processor to shard the file into the plurality of shard files based at least in part on a size of the file satisfying a threshold.
18. The computer-readable medium of claim 17 , wherein the instructions are further executable by the at least one processor to:
refrain from sharding a second file in the fileset based at least in part on a size of the second file being below the threshold.
19. The computer-readable medium of claim 15 , wherein the first shard file corresponds to a first partition of the plurality of partitions, wherein the second shard file corresponds to a second partition of the plurality of partitions.
20. The computer-readable medium of claim 15 , wherein the instructions are further executable by the at least one processor to:
scan the fileset to generate the metadata associated with the file.