Blobstore system for the management of large data objects
Methods and systems for backing up and restoring different point in time versions of a virtual machine, an application, a database, or an electronic file using independently managed snapshot chains are described. In some cases, different point in time versions of a virtual machine may be captured and stored using one or more snapshot chains. Each snapshot chain may correspond with a base image (e.g., a full image snapshot) and one or more incremental files (e.g., two forward incremental files) that derive from the base image. A snapshot chain may be split into a plurality of shards or sub-chains such that the width of each shard or the maximum data size of the files in each shard is less than a threshold data size (e.g., is less than 1 TB). Data operations (e.g., consolidation and reverse operations) may be performed on the individual shards or sub-chains in parallel.
1. A method for operating a data management system, comprising:
acquiring a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain includes a first base image and a first set of incremental files;
detecting a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine;
generating a second base image for the second snapshot chain using the first base image and the first set of incremental files;
storing the second base image;
performing a first operation on the first snapshot chain while performing a second operation different from the first operation on the second snapshot chain, the performing the first operation includes reading the first base image, the performing the second operation includes reading the second base image;
detecting a second triggering event to consolidate the second snapshot chain into the first snapshot chain, the detecting the second triggering event includes detecting that a number of incremental files within the first snapshot chain is less than a threshold number of incremental files;
generating a third set of incremental files that derives from the first base image the third set of incremental files correspond with the second set of versions of the virtual machine; and
storing the third set of incremental files.
2. The method of claim 1 , wherein:
the detecting the triggering event includes detecting that a number of incremental files within the first snapshot chain is greater than a threshold number of incremental files.
3. The method of claim 1 , wherein:
the detecting the triggering event includes detecting that a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files.
4. The method of claim 1 , wherein:
the detecting the triggering event includes determining an amount of available disk space and detecting that the amount of available disk space is greater than a threshold amount of disk space.
5. The method of claim 1 , wherein:
the first snapshot chain has a first snapshot chain length and the second snapshot chain has a second snapshot chain length different from the first snapshot chain length.
6. The method of claim 1 , wherein:
the first operation comprises a consolidation operation; and
the second operation comprises a reverse operation.
7. The method of claim 1 , wherein:
the first operation comprises a first consolidation operation; and
the second operation comprises a second consolidation operation.
8. The method of claim 1 , wherein:
the second operation comprises a reverse operation that generates a third base image for the second snapshot chain.
9. The method of claim 1 , further comprising:
detecting a reduction in a snapshot frequency for capturing snapshots of the virtual machine; and
consolidating the second snapshot chain into the first snapshot chain in response to detecting the reduction in the snapshot frequency.
10. A data management system, comprising:
a memory configured to store a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain includes a first base image and a first set of incremental files; and
one or more processors configured to detect a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine and generate a second base image for the second snapshot chain using the first base image and the first set of incremental files, the one or more processors configured to cause a first operation to be performed on the first snapshot chain while a second operation different from the first operation is performed on the second snapshot chain, the first operation includes reading the first base image and the second operation includes reading the second base image, the one or more processors configured to detect that a number of incremental files within the first snapshot chain is less than a threshold number of incremental files and consolidate the second snapshot chain into the first snapshot chain in response to detection that the number of incremental files within the first snapshot chain is less than the threshold number of incremental files.
11. The data management system of claim 10 , wherein:
the one or more processors configured to detect the triggering event if a number of incremental files within the first snapshot chain is greater than a threshold number of incremental files.
12. The data management system of claim 10 , wherein:
the one or more processors configured to detect the triggering event if a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files.
13. The data management system of claim 10 , wherein:
the one or more processors configured to determine an amount of available disk space and detect the triggering event if the amount of available disk space is greater than a threshold amount of disk space.
14. The data management system of claim 10 , wherein:
the first operation comprises a consolidation operation; and
the second operation comprises a reverse operation.
15. The data management system of claim 10 , wherein:
the first operation comprises a first consolidation operation; and
the second operation comprises a second consolidation operation.
16. The data management system of claim 10 , wherein:
the second operation comprises a reverse operation that generates a third base image for the second snapshot chain.
17. The data management system of claim 10 , wherein:
the one or more processors configured to detect a reduction in a snapshot frequency for capturing snapshots of the virtual machine and consolidate the second snapshot chain into the first snapshot chain in response to detection of the reduction in the snapshot frequency.
18. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a data management system, the processor readable code comprising:
processor readable code configured to acquire a first snapshot chain corresponding with a first set of versions of a virtual machine, the first snapshot chain includes a first base image and a first set of incremental files;
processor readable code configured to detect that a number of forward incremental files within the first snapshot chain is greater than a threshold number of forward incremental files;
processor readable code configured to detect a triggering event to generate a second snapshot chain corresponding with a second set of versions of the virtual machine in response to detection that the number of forward incremental files within the first snapshot chain is greater than the threshold number of forward incremental files;
processor readable code configured to generate a second base image for the second snapshot chain using the first base image and the first set of incremental files;
processor readable code configured to store the second base image; and
processor readable code configured to perform a first operation on the first snapshot chain while a second operation different from the first operation is performed on the second snapshot chain, the first operation includes reading the first base image and the second operation includes reading the second base image.