Time and value ordered data objects for a backup of a data set
Time and value ordering may be applied for items stored in data backups. A change log that persists changes to a data set may be updated with changes and used to update an in-memory table for the data set, which describes changes to items up to a current time. An event may be detected to seal the in-memory table from subsequent updates and a persistent data object that orders the items in the in-memory according to both keys of the respective items and the respective time values of the items, as stored in the change log, may be generated and stored as part of a backup for the data set.
1 . A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a database system, the database system configured to:
detect a compaction event for a backup of a data set;
identify persistent data objects of the backup of the data set to merge;
generate a merged persistent data object that includes items from the persistent data objects sorted according to values of the items and respective time values of the items, the values of the items being included if the respective time values of the items are within a retention threshold; and
generate a current item version of the merged persistent data object that includes the values of those items with a latest time value for the item.
2 . The system of claim 1 , wherein the detection of the compaction event is based, at least in part, on availability of computing resources to perform compaction.
3 . The system of claim 1 , wherein the detection of the compaction event is based, at least in part, on a number of the persistent data objects.
4 . The system of claim 1 , wherein the detection of the compaction event is based, at least in part, on a periodic occurrence of time.
5 . The system of claim 1 , wherein the detection of the compaction event is based, at least in part, on a request received from a component monitoring backup storage utilization in order to induce more efficient read performance or reduce storage utilization.
6 . The system of claim 1 , wherein to generate the merged persistent data object, the database system is configured to encrypt data of the merged persistent data object.
7 . The system of claim 1 , wherein to generate the merged persistent data object, the database system is configured to compress data of the merged persistent data object.
8 . A method, comprising:
detecting a compaction event for a backup of a data set;
identifying persistent data objects of the backup of the data set to merge;
generating a merged persistent data object that includes items from the persistent data objects sorted according to values of the items and respective time values of the items, the values of the items being included if the respective time values of the items are within a retention threshold; and
generating a current item version of the merged persistent data object that includes the values of those items with a latest time value for the item.
9 . The method of claim 8 , wherein the detection of the compaction event is based, at least in part, on availability of computing resources to perform compaction.
10 . The method of claim 8 , wherein the detection of the compaction event is based, at least in part, on a number of the persistent data objects.
11 . The method of claim 8 , wherein the detection of the compaction event is based, at least in part, on a periodic occurrence of time.
12 . The method of claim 8 , wherein the detection of the compaction event is based, at least in part, on a request received from a component monitoring backup storage utilization in order to induce more efficient read performance or reduce storage utilization.
13 . The method of claim 8 , wherein generating the merged persistent data object comprises encrypting data of the merged persistent data object.
14 . The method of claim 8 , wherein generating the merged persistent data object comprises compressing data of the merged persistent data object.
15 . One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:
detecting a compaction event for a backup of a data set;
identifying persistent data objects of the backup of the data set to merge;
generating a merged persistent data object that includes items from the persistent data objects sorted according to values of the items and respective time values of the items, the values of the items being included if the respective time values of the items are within a retention threshold; and
generating a current item version of the merged persistent data object that includes the values of those items with a latest time value for the item.
16 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the detection of the compaction event is based, at least in part, on a number of the persistent data objects.
17 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the detection of the compaction event is based, at least in part, on a number of the persistent data objects.
18 . The one or more non-transitory computer-readable storage media of claim 15 , wherein the detection of the compaction event is based, at least in part, on a request received from a component monitoring backup storage utilization in order to induce more efficient read performance or reduce storage utilization.
19 . The one or more non-transitory computer-readable storage media of claim 15 , wherein, in generating the merged persistent data object, the program instructions cause the one or more computing devices to implement encrypting data of the merged persistent data object.
20 . The one or more non-transitory computer-readable storage media of claim 15 , wherein, in generating the merged persistent data object,, the program instructions cause the one or more computing devices to implement compressing data of the merged persistent data object.