Storage volume snapshot object management
Systems and methods for durable storage of storage volume “snapshots” are provided. Snapshots are stored as collections of snapshot data objects. To improve the durability of snapshot storage, physical deletion of snapshot data objects may be delayed for a period of time after the snapshot data objects are marked for deletion. Lists of the stored snapshot data objects and the snapshot data objects that make up active snapshots may be periodically analyzed. If there are any snapshot data objects that are part of active snapshots and are not present in the list of stored snapshot data objects, the snapshot data objects may be recovered before they are physically deleted.
1. A system comprising:
an object data store storing an original snapshot of a data volume and an incremental snapshot of the data volume, wherein the original snapshot comprises a first plurality of snapshot data objects, wherein the incremental snapshot comprises a second plurality of snapshot data objects, and wherein a shared snapshot data object is in both the first plurality of snapshot data objects and the second plurality of snapshot data objects; and
a snapshot durability subsystem comprising computer-readable memory and one or more processors, wherein the snapshot durability subsystem is configured to at least:
in response to a request to delete the original snapshot:
modify metadata associated with the first plurality of snapshot data objects to indicate that each of the first plurality of snapshot data objects, including the shared snapshot data object, is to be deleted after a threshold period of time; and
delete an original snapshot manifest referencing the first plurality of snapshot data objects, wherein an incremental snapshot manifest referencing the second plurality of snapshot data objects is retained; and prior to passage of the threshold period of time:
generate a first list of stored snapshot data objects in the object data store, wherein the shared snapshot data object is excluded from the first list based at least partly on metadata associated with the shared snapshot data object indicating the shared snapshot data object is to be deleted;
generate a second list of active snapshot data objects based at least partly on a set of active snapshot manifests including the incremental snapshot manifest, wherein the shared snapshot data object is included in the second list based at least partly on the incremental snapshot manifest; and
modify the metadata associated with the shared snapshot data object to indicate that the shared snapshot data object is not to be deleted based at least partly on the shared snapshot data object being absent from the first list and present in the second list.
2. The system of claim 1 , further comprising a garbage collection subsystem comprising computer-readable memory and one or more processors, wherein the garbage collection subsystem is configured to at least:
determine that the threshold period of time has passed; and
delete a first snapshot data object of the first plurality of snapshot data objects based on metadata associated with the first snapshot data object indicating the first snapshot data object is to be deleted after the threshold period of time, wherein the first snapshot data object is absent from the second list of active snapshot data objects.
3. The system of claim 1 , further comprising:
a substrate network, wherein the substrate network comprises the object data store and the snapshot durability subsystem; and
a network extension comprising one or more computing devices external to the substrate network, wherein the network extension is configured to:
restore the data volume using the original snapshot;
modify a first portion of the data volume, wherein a second portion of the data volume remains unmodified;
generate the incremental snapshot, wherein a first data object of the second plurality of snapshot data objects represents the first portion of the data volume that has been modified, and wherein the shared snapshot data object represents the second portion of the data volume that remains unmodified; and
send the incremental snapshot to the substrate network.
4. The system of claim 1 , further comprising:
a substrate network comprising a second object data store and a second snapshot durability subsystem; and
a network extension external to the substrate network and in communication with the substrate network via a network connection, wherein the network extension comprises the object data store and the snapshot durability subsystem.
5. A computer-implemented method comprising:
under control of a snapshot durability system comprising one or more computing devices configured to execute specific instructions,
storing, in an object data store, an original snapshot of a data volume and an incremental snapshot of the data volume, wherein the original snapshot comprises a first plurality of snapshot data objects, wherein the incremental snapshot comprises a second plurality of snapshot data objects, and wherein a shared snapshot data object is in both the first plurality of snapshot data objects and the second plurality of snapshot data objects; and
in response to a request to delete the original snapshot:
modifying metadata associated with the first plurality of snapshot data objects to indicate that each of the first plurality of snapshot data objects, including the shared snapshot data object, is to be deleted after a threshold period of time; and
deleting an original snapshot manifest referencing the first plurality of snapshot data objects, wherein an incremental snapshot manifest referencing the second plurality of snapshot data objects is retained; and prior to passage of the threshold period of time:
generating a first list of stored snapshot data objects in the object data store, wherein the shared snapshot data object is excluded from the first list based at least partly on metadata associated with the shared snapshot data object indicating the shared snapshot data object is to be deleted;
generating a second list of active snapshot data objects based at least partly on a set of active snapshot manifests including the incremental snapshot manifest, wherein the shared snapshot data object is included in the second list based at least partly on the incremental snapshot manifest; and
modifying the metadata associated with the shared snapshot data object to indicate that the shared snapshot data object is not to be deleted based at least partly on the shared snapshot data object being absent from the first list and present in the second list.
6. The computer-implemented method of claim 5 , further comprising:
determining that the threshold period of time has passed; and
deleting a first snapshot data object of the first plurality of snapshot data objects based on metadata associated with the first snapshot data object indicating the first snapshot data object is to be deleted after the threshold period of time, wherein the first snapshot data object is absent from the second list of active snapshot data objects.
7. The computer-implemented method of claim 5 , further comprising:
restoring, by a network extension comprising one or more computing devices, the data volume using the original snapshot, wherein the network extension is external to a substrate network comprising the object data store and the snapshot durability system;
modifying, by the network extension, a first portion of the data volume, wherein a second portion of the data volume remains unmodified;
generating, by the network extension, the incremental snapshot, wherein a first data object of the second plurality of snapshot data objects represents the first portion of the data volume that has been modified, and wherein the shared snapshot data object represents the second portion of the data volume that remains unmodified; and
sending, by the network extension, the incremental snapshot to the substrate network.
8. The computer-implemented method of claim 5 , wherein generating the first list of stored snapshot data objects comprises:
executing an application programming interface (“API”) call to the object data store, wherein the API call comprises a request for a list of snapshot data objects stored in the object data store and available for restoring one or more data volumes; and
receiving, from the object data store, the list of snapshot data objects stored in the object data store and available for restoring one or more data volumes, wherein each snapshot data object of the second plurality of snapshot data objects are marked for deletion from the object data store.
9. The computer-implemented method of claim 8 , wherein generating the first list of stored snapshot data objects further comprises generating a compressed columnar representation of the list of snapshot data objects stored in the object data store and available for restoring one or more data volumes, wherein the first list of stored snapshot data objects comprises the compressed columnar representation.
10. The computer-implemented method of claim 5 , wherein generating the second list of active snapshot data objects comprises:
obtaining the set of active snapshot manifests, wherein individual snapshot manifests of the set of active snapshot manifests comprise data referencing one or more snapshot data objects associated with an active data volume snapshot of a plurality of active data volume snapshots, and wherein a first snapshot manifest and a second snapshot manifest of the set of active snapshot manifests both comprise data referencing a same active snapshot data object; and
generating a de-duplicated list of the second list of active snapshot data objects based on the set of active snapshot manifests.
11. The computer-implemented method of claim 10 , wherein generating the second list of active snapshot data objects further comprises generating a compressed columnar representation of the de-duplicated list of the second list of active snapshot data objects, wherein the second list of active snapshot data objects comprises the compressed columnar representation.
12. The computer-implemented method of claim 5 , wherein modifying the metadata associated with the shared snapshot data object comprises one of: removing data marking the shared snapshot data object for deletion, or adding the shared snapshot data object to an index of the object data store.
13. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed on a computing system, cause the computing system to perform operations comprising:
storing, in an object data store, an original snapshot of a data volume and an incremental snapshot of the data volume, wherein the original snapshot comprises a first plurality of snapshot data objects, wherein the incremental snapshot comprises a second plurality of snapshot data objects, and wherein a shared snapshot data object is in both the first plurality of snapshot data objects and the second plurality of snapshot data objects; and
in response to a request to delete the original snapshot:
modifying metadata associated with the first plurality of snapshot data objects to indicate that each of the first plurality of snapshot data objects, including the shared snapshot data object, is to be deleted after a threshold period of time; and
deleting an original snapshot manifest referencing the first plurality of snapshot data objects, wherein an incremental snapshot manifest referencing the second plurality of snapshot data objects is retained; and prior to passage of the threshold period of time:
generating a first list of stored snapshot data objects in the object data store, wherein the shared snapshot data object is excluded from the first list based at least partly on metadata associated with the shared snapshot data object indicating the shared snapshot data object is to be deleted;
generating a second list of active snapshot data objects based at least partly on a set of active snapshot manifests including the incremental snapshot manifest, wherein the shared snapshot data object is included in the second list based at least partly on the incremental snapshot manifest; and
modifying the metadata associated with the shared snapshot data object to indicate that the shared snapshot data object is not to be deleted based at least partly on the shared snapshot data object being absent from the first list and present in the second list.
14. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
determining that the threshold period of time has passed; and
deleting a first snapshot data object of the first plurality of snapshot data objects based on metadata associated with the first snapshot data object indicating the first snapshot data object is to be deleted after the threshold period of time, wherein the first snapshot data object is absent from the second list of active snapshot data objects.
15. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
restoring, by a network extension comprising one or more computing devices external to a substrate network, the data volume using the original snapshot, wherein the substrate network comprises the object data store and a the computing system;
modifying, by the network extension, a first portion of the data volume, wherein a second portion of the data volume remains unmodified;
generating, by the network extension, the incremental snapshot, wherein a first data object of the second plurality of snapshot data objects represents the first portion of the data volume that has been modified, and wherein the shared snapshot data object represents the second portion of the data volume that remains unmodified; and
sending, by the network extension, the incremental snapshot to the substrate network.
16. The non-transitory computer-readable medium of claim 13 , wherein to generate the first list of stored snapshot data objects, the operations further comprise:
executing an application programming interface (“API”) call to the object data store, wherein the API call comprises a request for a list of snapshot data objects stored in the object data store and available for restoring one or more data volumes; and
receiving, from the object data store, the list of snapshot data objects stored in the object data store and available for restoring one or more data volumes, wherein each snapshot data object of the second plurality of snapshot data objects are marked for deletion from the object data store.
17. The non-transitory computer-readable medium of claim 16 , wherein to generate the first list of stored snapshot data objects, the operations further comprise generating a compressed columnar representation of the list of snapshot data objects stored in the object data store and available for restoring one or more data volumes, wherein the first list of stored snapshot data objects comprises the compressed columnar representation.
18. The non-transitory computer-readable medium of claim 13 , wherein to generate the second list of active snapshot data objects, the operations further comprise:
obtaining the set of active snapshot manifests, wherein individual snapshot manifests of the set of active snapshot manifests comprise data referencing one or more snapshot data objects associated with an active data volume snapshot of a plurality of active data volume snapshots, and wherein a first snapshot manifest and a second snapshot manifest of the set of active snapshot manifests both comprise data referencing a same active snapshot data object; and
generating a de-duplicated list of the second list of active snapshot data objects based on the set of active snapshot manifests.
19. The non-transitory computer-readable medium of claim 18 , wherein to generate the second list of active snapshot data objects, the operations further comprise generating a compressed columnar representation of the de-duplicated list of the second list of active snapshot data objects, wherein the second list of active snapshot data objects comprises the compressed columnar representation.
20. The non-transitory computer-readable medium of claim 13 , wherein to modify the metadata associated with the shared snapshot data object, the operations further comprise one of: removing data marking the shared snapshot data object for deletion, or adding the shared snapshot data object to an index of the object data store.