Data equivalency system using co-pair manifests for identifying duplicates in generations of resources

1 . A system for identifying data duplication in object-based snapshots of block-storage volumes, each object-based snapshot representing data of a corresponding block-storage volume at a point-in-time, the system comprising:

a metadata store storing metadata identifying individual object-based snapshots as encrypted or unencrypted, wherein an encrypted object-based snapshot corresponds to an encrypted block-storage volume, and wherein an unencrypted object-based snapshot corresponds to an unencrypted block-storage volume;

an object data store including:

snapshot objects, each snapshot object representing one or more blocks of data;

main manifests, each main manifest corresponding to an object-based snapshot and including a full listing of snapshot objects within the object data store that collectively represent data of the corresponding block-storage volume at the point-in-time of the object-based snapshot; and

one or more co-pair manifests, each co-pair manifest corresponding to an object-based snapshot, and including a listing of encrypted snapshot objects with equivalencies to unencrypted snapshot objects, wherein the encrypted snapshot objects are distinct from the unencrypted snapshot objects and wherein an equivalency indicates that an encrypted snapshot object represents duplicate storage, in encrypted form, as block-storage volume source data stored in unencrypted form as an unencrypted snapshot object; and

one or more processors configured to:

generate a deduplicated listing of snapshot objects within the object data store by:

identifying a set of object-based snapshots for deduplication;

retrieving from the object data store a set of main manifests and a set of co-pair manifests corresponding to the set of object-based snapshots;

identifying data duplication in the set of main manifests based on listings of encrypted snapshot objects with equivalencies to unencrypted snapshot objects found in the set of co-pair manifests; and

removing identified data duplication in the set of main manifests.

2 . The system of claim 1 , wherein the deduplicated listing of snapshot objects provides accurate snapshot usage data for a usage metering service.

3 . The system of claim 1 , wherein a block-storage volume represents a virtual storage disk drive hosted by a block storage system distinct from the object data store.

4 . The system of claim 1 , wherein the one or more processors represent a distributed computing system collectively configured to generate the deduplicated listing of snapshot objects.

5 . A computer-implemented method comprising:

generating a listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots, each object-based snapshot representing data of a corresponding block-storage volume at a point-in-time as a collection of snapshot objects that provide a copy of data of the block-storage volume at the point-in-time, wherein generating the listing comprises:

retrieving from an object data store a set of main manifests corresponding to the set of object-based snapshots, each main manifest including a full listing of snapshot objects within the object data store that collectively represent data of the corresponding block-storage volume at the point-in-time of the object-based snapshot;

retrieving from the object data store a set of co-pair manifests corresponding to the set of object-based snapshots, each co-pair manifest corresponding to an object-based snapshot, and including a listing of encrypted snapshot objects with equivalencies to unencrypted snapshot objects, wherein the encrypted snapshot objects are distinct from the unencrypted snapshot objects, and wherein an equivalency indicates that an encrypted snapshot object represents duplicate storage, in encrypted form, as block-storage volume source data stored in an unencrypted form as an unencrypted snapshot object;

identifying data duplication in the set of main manifests based on listings of encrypted snapshot objects with equivalencies to unencrypted snapshot objects found in the set of co-pair manifests; and

removing identified data duplication in the set of main manifests.

6 . The computer-implemented method of claim 5 , wherein identifying data duplication in the set of main manifests based on listings of encrypted snapshot objects with equivalencies to unencrypted snapshot objects found in the set of co-pair manifests comprises implementing a MapReduce operation on a distributed computing system, the MapReduce operation including a reshuffling of information between devices of the distributed computing system during generation of the listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots.

7 . The computer-implemented method of claim 5 , further comprising identifying individual object-based snapshots as incremental or non-incremental, wherein an incremental object-based snapshot is associated with a parent object-based snapshot that represents an earlier state of the block-storage volume corresponding to the incremental object-based snapshot, and wherein a non-incremental object-based snapshot is not associated with a parent object-based snapshot.

8 . The computer-implemented method of claim 7 , wherein individual object-based snapshots can be inactive or active, and wherein identifying an individual object-based snapshot as incremental comprises identifying that the individual object-based snapshot is associated with an active parent object-based snapshot at a given point-in-time.

9 . The computer-implemented method of claim 8 , wherein identifying an individual object-based snapshot as non-incremental comprises inspecting metadata of the individual object-based snapshot, the metadata identifying a state of a parent of individual object-based snapshot as null.

10 . The computer-implemented method of claim 9 , wherein the metadata identifies an object-based snapshot as incremental or non-incremental by identifying whether the object-based snapshot depends on a prior object-based snapshot and identifying an active time period of the prior object-based snapshot.

11 . The computer-implemented method of claim 9 , wherein the metadata identifies each of the object-based snapshots as encrypted or unencrypted.

12 . The computer-implemented method of claim 5 , wherein the listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots is provided as input to a resource usage metering service.

13 . The computer-implemented method of claim 5 , wherein the set of main manifests and the set of co-pair manifests is stored on a network data store, and wherein retrieving the set of main manifests and the set of co-pair manifests comprises transferring the set of main manifests and the set of co-pair manifests over a network.

14 . One or more non-transitory computer-readable media comprising instructions executable by one or more processors to generate a listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots, each object-based snapshot representing data of a corresponding block-storage volume at a point-in-time as a collection of snapshot objects that provide a copy of data of the block-storage volume at the point-in-time, wherein the instructions, when executed, cause the one or more processors to:

retrieve from an object data store a set of main manifests corresponding to the set of object-based snapshots, each main manifest including a full listing of snapshot objects within the object data store that collectively represent data of the corresponding block-storage volume at the point-in-time of the object-based snapshot;

retrieve from the object data store a set of co-pair manifests corresponding to the set of object-based snapshots, each co-pair manifest corresponding to an object-based snapshot, and including a listing of encrypted snapshot objects with equivalencies to unencrypted snapshot objects, wherein the encrypted snapshot objects are distinct from the unencrypted snapshot objects and wherein an equivalency indicates that an encrypted snapshot object represents duplicate storage, in encrypted form, as block-storage volume source data stored in unencrypted form as an unencrypted snapshot object;

identify data duplication in the set of main manifests based on listings of encrypted snapshot objects with equivalencies to unencrypted snapshot objects found in the set of co-pair manifests; and

remove identified data duplication in the set of main manifests.

15 . The one or more non-transitory computer-readable media of claim 14 , wherein identifying data duplication in the set of main manifests based on listings of encrypted snapshot objects with equivalencies to unencrypted snapshot objects found in the set of co-pair manifests comprises implementing a MapReduce operation on a distributed computing system, the MapReduce operation including a reshuffling of information between devices of the distributed computing system during generation of the listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots.

16 . The one or more non-transitory computer-readable media of claim 14 , wherein the instructions, when executed, further cause the one or more processors to identify individual object-based snapshots as incremental or non-incremental, wherein an incremental object-based snapshot is associated with a parent object-based snapshot that represents an earlier state of the block-storage volume corresponding to the incremental object-based snapshot, and wherein a non-incremental object-based snapshot is not associated with a parent object-based snapshot.

17 . The one or more non-transitory computer-readable media of claim 16 , wherein individual object-based snapshots can be inactive or active, and wherein identifying an individual object-based snapshot as incremental comprises identifying that the individual object-based snapshot is associated with an active parent object-based snapshot at a given point-in-time.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein identifying an individual object-based snapshot as non-incremental comprises inspecting metadata of the individual object-based snapshot, the metadata identifying a state of a parent of individual object-based snapshot as null.

19 . The one or more non-transitory computer-readable media of claim 14 , wherein the listing of equivalent snapshot objects that are referenced by one or more of a set of object-based snapshots is provided as input to a resource usage metering service.

20 . The one or more non-transitory computer-readable media of claim 14 , wherein the set of main manifests and the set of co-pair manifests is stored on a network data store, and wherein retrieving the set of main manifests and the set of co-pair manifests comprises transferring the set of main manifests and the set of co-pair manifests over a network.