Protection groups for backing up cloud-based key-value stores
Techniques are disclosed relating to providing protection groups and rapid determination of expired objects and associated storage resources in a cloud-based backup storage context. In various embodiments, the disclosed techniques include generating, by a computing system, incremental backup data for a backup based on protection group information, including a set of cloud-based key-value data store buckets, filter information, and policy information. Disclosed techniques also include rapid determination of expired objects and associated storage resources. Disclosed techniques may advantageously provide a centralized view across buckets, accounts, and regions and allow users to target specific subsets of their data stores for backup, which may improve performance and reduce backup storage costs, relative to traditional techniques.
1 . A method performed by a computer system operating in a cloud computing environment, wherein the cloud computing environment comprises a cloud storage service, and
wherein the computer system comprises a processor subsystem coupled to non-transitory media storing computer program instructions executed by the processor subsystem, the method comprising:
responsive to a request for an incremental backup of data objects of a protection group,
wherein the protection group is associated with a first plurality of buckets of the cloud storage service,
wherein the data objects of the protection group are stored among the first plurality of buckets, and
wherein each bucket among the first plurality of buckets is organized as a key-value data store,
determining:
the first plurality of buckets associated with the protection group,
filters associated with the protection group, wherein the filters indicate first data objects to select for backup from among the data objects of the protection group, and
policy parameters for backing up the first data objects;
based on the filters and the policy parameters, generating incremental backup data objects, wherein the incremental backup data objects correspond to second data objects among the first data objects that changed since a preceding backup of the protection group;
after storing the incremental backup data objects in a second plurality of buckets, querying the second plurality of buckets that comprise the incremental backup data objects; and
based on the querying, providing first incremental backup data objects that satisfy query criteria from among the incremental backup data objects stored among the second plurality of buckets.
2 . The method of claim 1 , wherein the first plurality of buckets are configured in a first service account of the cloud computing environment, and wherein the second plurality of buckets are configured in a second service account of the cloud computing environment, wherein the second service account is distinct from the first service account.
3 . The method of claim 1 , wherein the filters comprise one or more key fields that comprise one or more prefixes for identifying the first data objects within the first plurality of buckets.
4 . The method of claim 1 , wherein the filters comprise one or more key fields that comprise one or more prefixes for excluding some of the first data objects.
5 . The method of claim 1 , wherein the filters indicate one or more storage classes for identifying the first data objects within the first plurality of buckets.
6 . The method of claim 1 , wherein the filters include version criteria for identifying the first data objects within the first plurality of buckets.
7 . The method of claim 1 , wherein the policy parameters indicate a target storage tier for the second plurality of buckets.
8 . The method of claim 1 , wherein querying the second plurality of buckets includes an indication of whether to return a latest version of an incremental backup data object or multiple versions thereof.
9 . The method of claim 1 , wherein the first plurality of buckets additionally comprises a bucket that is configured in a second cloud computing environment that is distinct from the cloud computing environment that comprises the cloud storage service.
10 . The method of claim 1 , wherein each data object among the data objects of the protection group comprises a key and version identifier that, collectively, uniquely identify the respective data object within a bucket, and wherein the filters indicate version identifiers to include in the incremental backup.
11 . A computer system operating in a cloud computing environment, wherein the cloud computing environment comprises a cloud storage service, and
wherein the computer system comprises a processor subsystem coupled to non-transitory media storing computer program instructions, which, when executed by the processor subsystem, cause the computer system to:
responsive to a request for an incremental backup of data objects of a protection group,
wherein the protection group is associated with a first plurality of buckets configured in the cloud storage service,
wherein the data objects of the protection group are stored among the first plurality of buckets, and
wherein each bucket among the first plurality of buckets is organized as a key-value data store,
determine:
the first plurality of buckets associated with the protection group,
filters associated with the protection group, wherein the filters indicate first data objects to select for backup from among the data objects of the protection group, and
policy parameters for backing up the first data objects;
based on the filters and the policy parameters, generate incremental backup data objects, wherein the incremental backup data objects correspond to second data objects among the first data objects that changed since a preceding backup of the protection group;
cause the incremental backup data objects to be stored in a second plurality of buckets;
query the second plurality of buckets that comprise the incremental backup data objects; and
provide first incremental backup data objects that satisfy query criteria from among the incremental backup data objects stored among the second plurality of buckets.
12 . The computer system of claim 11 , wherein the first plurality of buckets are configured in a first service account of the cloud computing environment, and wherein the second plurality of buckets are configured in a second service account of the cloud computing environment, wherein the second service account is distinct from the first service account.
13 . The computer system of claim 11 , wherein the filters comprise one or more key fields that comprise one or more prefixes for identifying the first data objects within the first plurality of buckets.
14 . The computer system of claim 11 , wherein the filters comprise one or more key fields that comprise one or more prefixes for excluding some of the first data objects.
15 . The computer system of claim 11 , wherein the filters indicate one or more storage classes for identifying the first data objects within the first plurality of buckets.
16 . The computer system of claim 11 , wherein the filters include version criteria for identifying the first data objects within the first plurality of buckets.
17 . The computer system of claim 11 , wherein the policy parameters indicate a target storage tier for the second plurality of buckets.
18 . The computer system of claim 11 , wherein querying the second plurality of buckets includes an indication of whether to return a latest version of an incremental backup data object or multiple versions thereof.
19 . The computer system of claim 11 , wherein the first plurality of buckets comprises a bucket that is configured in a second cloud computing environment that is distinct from the cloud computing environment that comprises the cloud storage service.
20 . The computer system of claim 11 , wherein each data object among the data objects of the protection group comprises a key and version identifier that, collectively, uniquely identify the respective data object within a bucket, and wherein the filters indicate version identifiers to include in the incremental backup.