Sharding of full and incremental snapshots
Methods and systems for backing up and restoring different point in time versions of a virtual machine, an application, a database, or an electronic file using independently managed snapshot chains are described. In some cases, different point in time versions of a virtual machine may be captured and stored using one or more snapshot chains. Each snapshot chain may correspond with a base image (e.g., a full image snapshot) and one or more incremental files (e.g., two forward incremental files) that derive from the base image. A snapshot chain may be split into a plurality of shards or sub-chains such that the width of each shard or the maximum data size of the files in each shard is less than a threshold data size (e.g., is less than 1 TB). Data operations (e.g., consolidation and reverse operations) may be performed on the individual shards or sub-chains in parallel.
1. A method for operating a data management system, comprising:
acquiring a snapshot chain corresponding with a first set of versions of a virtual machine, the snapshot chain including a first set of files, the first set of files including a base image and a set of incremental files;
upon detecting a triggering event including determining an amount of available disk space and detecting that the amount of available disk space is less than a threshold amount of disk space, splitting the first set of files of the snapshot chain into a first snapshot sub-chain and a second snapshot sub-chain, the first snapshot sub-chain including a second set of files corresponding with the first snapshot sub-chain, the second snapshot sub-chain including a third, separate set of files corresponding with the second snapshot sub-chain, the first snapshot sub-chain and the second snapshot sub-chain including a uniform width or maximum file size;
generating a first base image for the first snapshot sub-chain using the base image of the snapshot chain in response to detecting the triggering event, the first base image corresponding to a first version of the virtual machine;
generating a second base image for the second snapshot sub-chain using the base image of the snapshot chain in response to detecting the triggering event, the second base image corresponding to a second version different from the first version of the virtual machine; and
performing a first operation on the first snapshot sub-chain while independently performing a second operation on the second snapshot sub-chain, the performing of the first operation including a consolidation operation, the performing of the second operation including a rebasing operation.
2. The method of claim 1 , wherein:
the detecting the triggering event includes determining a snapshot frequency for the virtual machine and detecting that the snapshot frequency is greater than a threshold snapshot frequency.
3. The method of claim 1 , further comprising:
storing the first base image on a first storage device; and
storing the second base image on a second storage device different from the first storage device.
4. The method of claim 1 , further comprising:
determining an amount of available disk space;
determining a threshold number of snapshot sub-chains based on the amount of available disk space; and
generating a set of base images each corresponding with the number of snapshot sub-chains, the set of base images includes the first base image and the second base image.
5. The method of claim 1 , wherein:
the first base image has a first file size and the base image has a second file size that is greater than the first file size;
the first snapshot sub-chain has a first maximum file size; and
the second snapshot sub-chain has a second maximum file size different from the first maximum file size.
6. The method of claim 1 , further comprising:
detecting a second triggering event to consolidate the first snapshot sub-chain and the second snapshot sub-chain into a third snapshot chain corresponding with the first set of versions of the virtual machine.
7. The method of claim 6 , wherein:
the detecting the second triggering event includes detecting that an amount of available disk space is greater than a threshold amount of disk space.
8. The method of claim 6 , wherein:
the detecting the second triggering event includes detecting that a snapshot frequency for the virtual machine is less than a threshold snapshot frequency.
9. A data management system, comprising:
a memory configured to store a snapshot chain corresponding with a first set of versions of a virtual machine, the snapshot chain including a first set of files, the first set of files including a base image and a set of incremental files; and
one or more processors configured to determine an amount of available disk space and detect a triggering event if the amount of available disk space is less than a threshold amount of disk space and to split the first set of files of the snapshot chain into a first snapshot sub-chain and a second snapshot sub-chain upon detecting the triggering event, the first snapshot sub-chain including a second set of files corresponding with the first snapshot sub-chain, the second snapshot sub-chain including a third, separate set of files corresponding with the second snapshot sub-chain, the first snapshot sub-chain and the second snapshot sub-chain including a uniform width or maximum file size,
the one or more processors configured to generate a first base image for the first snapshot sub-chain using the base image of the snapshot chain, the first base image corresponding to a first version of the virtual machine, and generate a second base image for the second snapshot sub-chain using the base image of the snapshot chain in response to detection of the triggering event, the second base image corresponding to a second version different from the first version of the virtual machine,
the one or more processors configured to cause a first operation to be performed on the first snapshot sub-chain while a second operation is performed independently on the second snapshot sub-chain, the first operation including a consolidation operation, the performing of the second operation including a rebasing operation.
10. The data management system of claim 9 , wherein:
the one or more processors configured to determine a snapshot frequency for the virtual machine and detect the triggering event if the snapshot frequency is greater than a threshold snapshot frequency.
11. The data management system of claim 9 , wherein:
the first base image is stored using a first storage device; and
the second base image is stored using a second storage device different from the first storage device.
12. The data management system of claim 9 , wherein:
the one or more processors configured to determine a threshold number of snapshot sub-chains based on an amount of available disk space and generate a set of base images each corresponding with the number of snapshot sub-chains, the set of base images includes the first base image and the second base image.
13. The data management system of claim 9 , wherein:
the first snapshot sub-chain has a first maximum file size; and
the second snapshot sub-chain has a second maximum file size greater than the first maximum file size.
14. The data management system of claim 9 , wherein:
the one or more processors configured to detect a second triggering event to consolidate the first snapshot sub-chain and the second snapshot sub-chain into a third snapshot chain corresponding with the first set of versions of the virtual machine, the one or more processors configured to generate a third base image for the third snapshot chain in response to detection of the second triggering event.
15. The data management system of claim 14 , wherein:
the second triggering event comprises detection that an amount of available disk space is greater than a threshold amount of disk space.
16. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a data management system, the processor readable code comprising:
processor readable code configured to acquire a snapshot chain corresponding with a first set of versions of a virtual machine, the snapshot chain including a first set of files, the first set of files including a base image and a set of incremental files;
processor readable code configured to determine an amount of available disk space for a cluster of data storage nodes;
processor readable code configured to determine an amount of available disk space and detect a triggering event if the amount of available disk space is less than a threshold amount of disk space and to detect the triggering event to split the first set of files of the snapshot chain into a first snapshot sub-chain and a second snapshot sub-chain, the first snapshot sub-chain including a second set of files corresponding with the first snapshot sub-chain, the second snapshot sub-chain including a third, separate set of files corresponding with the second snapshot sub-chain, the first snapshot sub-chain and the second snapshot sub-chain including a uniform width or maximum file size;
processor readable code configured to generate a first base image for the first snapshot sub-chain using the base image in response to a detection of the triggering event, the first base image corresponding to a first version of the virtual machine;
processor readable code configured to generate a second base image for the second snapshot sub-chain using the base image in response to a detection of the triggering event, the second base image corresponding to a version different from the first version of the virtual machine;
processor readable code configured to perform a first operation on the first snapshot sub-chain while independently performing a second operation on the second snapshot sub-chain, the first operation including a consolidation operation, the performing of the second operation including a rebasing operation.