Blobstore system for the management of large data objects
Methods and systems for backing up and restoring different point in time versions of a virtual machine, an application, a database, or an electronic file using independently managed snapshot chains are described. In some cases, different point in time versions of a virtual machine may be captured and stored using one or more snapshot chains. Each snapshot chain may correspond with a base image (e.g., a full image snapshot) and one or more incremental files (e.g., two forward incremental files) that derive from the base image. A snapshot chain may be split into a plurality of shards or sub-chains such that the width of each shard or the maximum data size of the files in each shard is less than a threshold data size (e.g., is less than 1 TB). Data operations (e.g., consolidation and reverse operations) may be performed on the individual shards or sub-chains in parallel.
1. A method for operating a data management system, the method comprising:
acquiring a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain including at least a first base image;
detecting a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine;
generating a second base image for the second snapshot chain using at least the first base image;
storing the second base image;
performing a first operation on the first snapshot chain while performing a second operation on the second snapshot chain, the performing the first operation including reading the first base image, the performing the second operation including reading the second base image;
detecting a reduction in a snapshot frequency for capturing snapshots of the virtual machine;
detecting a second triggering event to consolidate the second snapshot chain into the first snapshot chain into the first snapshot chain in response to detecting the reduction in the snapshot frequency;
generating a third set of incremental files that derives from the first base image, the third set of incremental files corresponding with the second set of versions of the virtual machine; and
storing the third set of incremental files.
2. The method of claim 1 , wherein:
the detecting the triggering event includes detecting that a number of incremental files within the first snapshot chain is greater than a threshold number of incremental files.
3. The method of claim 1 , wherein:
the detecting the triggering event includes detecting that a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files.
4. The method of claim 1 , wherein:
the detecting the triggering event includes determining an amount of available disk space and detecting that the amount of available disk space is greater than a threshold amount of disk space.
5. The method of claim 1 , wherein:
the first snapshot chain has a first snapshot chain length and the second snapshot chain has a second snapshot chain length different from the first snapshot chain length.
6. The method of claim 1 , wherein:
the first operation comprises a consolidation operation; and
the second operation comprises a reverse operation.
7. The method of claim 1 , wherein:
the first operation comprises a first consolidation operation; and
the second operation comprises a second consolidation operation.
8. The method of claim 1 , wherein:
the second operation comprises a reverse operation that generates a third base image for the second snapshot chain.
9. A data management system, comprising:
a memory configured to store a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain including at least a first base image; and
one or more processors configured to detect a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine;
generate a second base image for the second snapshot chain using at least the first base mage;
store the second base image;
perform a first operation on the first snapshot chain while performing a second operation on the second snapshot chain, the performing the first operation including reading the first base image, the performing the second operation including reading the second base image;
detecting a reduction in a snapshot frequency for capturing snapshots of the virtual machine;
detect a second triggering event to consolidate the second snapshot chain into the first snapshot chain into the first snapshot chain in response to detecting the reduction in the snapshot frequency;
generate a third set of incremental files that derives from the first base image, the third set of incremental files corresponding with the second set of versions of the virtual machine; and
store the third set of incremental files.
10. The data management system of claim 9 , wherein:
the one or more processors are configured to detect the triggering event if a number of incremental files within the first snapshot chain is greater than a threshold number of incremental files.
11. The data management system of claim 9 , wherein:
the one or more processors are configured to detect the triggering event if a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files.
12. The data management system of claim 9 , wherein:
the one or more processors are configured to determine an amount of available disk space and detect the triggering event if the amount of available disk space is greater than a threshold amount of disk space.
13. The data management system of claim 9 , wherein:
the first operation comprises a consolidation operation; and
the second operation comprises a reverse operation.
14. The data management system of claim 9 , wherein:
the first operation comprises a first consolidation operation; and
the second operation comprises a second consolidation operation.
15. The data management system of claim 9 , wherein:
the second operation comprises a reverse operation that generates a third base image for the second snapshot chain.
16. A machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform in operations comprising:
acquiring a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain including at least a first base image;
detecting a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine;
generating a second base image for the second snapshot chain using at least the first base image;
storing the second base image;
performing a first operation on the first snapshot chain while performing a second operation on the second snapshot chain, the performing the first operation including reading the first base image, the performing the second operation including reading the second base image;
detecting a reduction in a snapshot frequency for capturing snapshots of the virtual machine;
detecting a second triggering event to consolidate the second snapshot chain into the first snapshot chain into the first snapshot chain in response to detecting the reduction in the snapshot frequency;
generating a third set of incremental files that derives from the first base image, the third set of incremental files corresponding with the second set of versions of the virtual machine; and
storing the third set of incremental files.
17. The medium of claim 16 , wherein:
the detecting the triggering event includes detecting that a number of incremental files within the first snapshot chain is greater than a threshold number of incremental files.
18. The medium of claim 16 , wherein:
the detecting the triggering event includes detecting that a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files.
19. The medium of claim 16 , wherein:
the detecting the triggering event includes deter mining an amount of available disk space and detecting that the amount of available disk space is greater than a threshold amount of disk space.
20. The medium of claim 16 , wherein:
the first snapshot chain has a first snapshot chain length and the second snapshot chain has a second snapshot chain length different from the first snapshot chain length.