Scaling single file snapshot performance across clustered system
In some embodiments, a process for restoring a version of a virtual machine using a data storage system comprises identifying a particular version of a virtual machine to be restored, determining a base image from which the particular version may be derived, determining a set of incremental files for generating the particular version, generating a file associated with the particular version using the base image and the set of incremental files, and outputting at least a portion of the file.
1. A method comprising:
sharding a file in a filesystem into a plurality of shard files based on filesystem metadata;
partitioning the filesystem into a plurality of partitions based on the plurality of shard files, each shard file, of the plurality of shard files, corresponding to a partition, of the plurality of partitions; and
distributing each shard file, in the plurality of shard files, to a respective node, the distributing of the plurality of shard files being related to the plurality of partitions.
2. The method of claim 1 , wherein the plurality of partitions includes a first partition and a second partition, wherein the plurality of shard files includes a first shard file and a second shard file, wherein the distributing includes distributing the first shard file to the first partition of the filesystem and distributing the second shard file to the second partition of the filesystem.
3. The method of claim 1 , wherein sharding the file into the plurality of shard files includes creating the plurality of shard files on a fourth extended (EXT4) filesystem under a Linux operating system.
4. The method of claim 1 , wherein each of the partitions, in the plurality of partitions, respectively corresponds to a shard file in the plurality of shard files.
5. The method of claim 1 , further comprising scanning at least one file in the filesystem to generate the filesystem metadata, wherein the scanning is performed by partitioning logic.
6. The method of claim 1 , wherein the plurality of shard files includes a first shard file and wherein the first shard file is appended with a random hash.
7. A system comprising:
a memory storing instructions;
one or more processors configured by the instructions to perform operations in a method, the operations comprising at least:
sharding a file in a filesystem into a plurality of shard files based on filesystem metadata;
partitioning the filesystem into a plurality of partitions based on the plurality of shard files, each shard file, of the plurality of shard files, corresponding to a partition, of the plurality of partitions; and
distributing each shard file, in the plurality of shard files, to a respective node, the distributing of the plurality of shard files being related to the plurality of partitions.
8. The system of claim 7 , wherein the plurality of partitions includes a first partition and a second partition, wherein the plurality of shard files includes a first shard file and a second shard file, wherein the distributing includes distributing the first shard file to the first partition of the filesystem and distributing the second shard file to the second partition of the filesystem.
9. The system of claim 7 , wherein sharding the file into the plurality of shard files includes creating the plurality of shard files on a fourth extended (EXT4) filesystem under a Linux operating system.
10. The system of claim 7 , wherein each of the partitions, in the plurality of partitions, respectively corresponds to a shard file, in the plurality of shard files.
11. The system of claim 7 , wherein the operations further comprise scanning at least one file in the filesystem to generate the filesystem metadata, wherein the scanning is performed by partitioning logic.
12. The system of claim 7 , wherein the plurality of shard files includes a first shard file and wherein the first shard file is appended with a random hash.
13. A non-transitory machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform operations in a method, the operations comprising at least:
sharding a file in a filesystem into a plurality of shard files based on filesystem metadata;
partitioning the filesystem into a plurality of partitions based on the plurality of shard files, each shard file, of the plurality of shard files, corresponding to a partition, of the plurality of partitions; and
distributing each shard file, in the plurality of shard files, to a respective node, the distributing of the plurality of shard files being related to the plurality of partitions.
14. The medium of claim 13 , wherein the plurality of partitions includes a first partition and a second partition, wherein the plurality of shard files includes a first shard file and a second shard file, wherein the distributing includes distributing the first shard file to the first partition of the filesystem and distributing the second shard file to the second partition of the filesystem.
15. The medium of claim 13 , wherein sharding the file into the plurality of shard files includes creating the plurality of shard files on a fourth extended (EXT4) filesystem under a Linux operating system.
16. The medium of claim 13 , wherein each of the partitions in the plurality of partitions respectively corresponds to a shard file in the plurality of shard files.
17. The medium of claim 13 , wherein the operations further comprise scanning at least one file in the filesystem to generate the filesystem metadata, wherein the scanning is performed by partitioning logic.
18. The medium of claim 13 , wherein the plurality of shard files includes a first shard file and wherein the first shard file is appended with a random hash.