Parallelization of incremental backups
An assigned subgroup that includes a plurality of entries is traversed by a prefetcher. It is determined that an expected number of entries associated with the assigned subgroup have been traversed. In response to determining that expected number of entries associated with the assigned subgroup have been traversed, it is determined that a last read entry associated with the assigned subgroup does not correspond to a last entry associated with the assigned subgroup. The prefetcher is preempted by stopping the prefetcher from obtaining a list of entries associated with a remaining portion of the assigned subgroup.
1 . A method, comprising:
invoking an application programming interface to obtain a list of objects associated with a subgroup assigned to a prefetcher;
updating, in response to determining that an entry of the obtained list of objects includes a partition identifier associated with a subsequent subgroup, a checkpoint file to include entries corresponding to the list of objects except the entry corresponding to the partition identifier associated with the subsequent subgroup and any entries that are sequentially after the entry that includes the partition identifier associated with the subsequent subgroup;
receiving, by a prefetcher, an indication that the checkpoint file has been updated;
determining, by the prefetcher, a plurality of differences between the updated checkpoint file and a checkpoint file associated with a previous backup; and
ingesting, by the prefetcher and concurrent to determining at least one of the plurality of differences, the plurality of determined differences.
2 . The method of claim 1 , further comprising discarding any entries that are sequentially after the entry corresponding to the partition identifier associated with the subsequent subgroup.
3 . The method of claim 1 , wherein determining the one or more differences comprises:
obtaining the checkpoint file associated with the previous backup;
reading a portion of the checkpoint file associated with the previous backup corresponding to objects assigned to the prefetcher; and
comparing the objects included in the read portion of the checkpoint file associated with the previous backup to objects included in an updated portion of the updated checkpoint file associated with a current backup to determine the one or more differences.
4 . The method of claim 3 , wherein the objects included in the checkpoint file associated with the previous backup and the updated checkpoint file associated with the current backup are listed in lexicographically sorted order.
5 . The method of claim 3 , wherein determining the plurality of differences comprises identifying one or more of the plurality of differences between a current state of an object storage bucket and a previous state of the object storage bucket corresponding to the previous backup.
6 . The method of claim 1 , wherein data associated with the one or more determined differences includes one or more object metadata and object data content.
7 . The method of claim 1 , wherein data associated with the plurality of differences indicates that an object was deleted after the previous backup.
8 . The method of claim 1 , wherein data associated with the one or more determined differences includes one or more of metadata associated with an object that changed after the previous backup, changed data associated with an object after the previous backup, object metadata and object content data associated with an object that was created after the previous backup.
9 . A system device, comprising:
a memory configured to store instructions that implement a prefetcher,
one or more processors configured to execute the instructions to implement the prefetcher, the prefetcher configured to:
invoke an application programming interface to obtain a list of objects associated with a subgroup assigned to the prefetcher;
update, in response to determining that an entry of the obtained list of objects includes a partition identifier associated with a subsequent subgroup, a checkpoint file to include entries corresponding to the list of objects except the entry corresponding to the partition identifier associated with the subsequent subgroup an any entries that are sequentially after the entry that includes the partition identifier associated with the subsequent subgroup;
receive an indication that the checkpoint file has been updated;
determine a plurality of differences between the updated checkpoint file and a checkpoint file associated with a previous backup; and
ingesting, concurrent to determining at least one of the plurality of differences, the plurality of determined differences.
10 . The computing system of claim 9 , wherein the one or more processors are configured to discard any entries that are sequentially after the entry corresponding to the partition identifier associated with the subsequent subgroup.
11 . The computing system of claim 9 , wherein the prefetcher is configured to:
obtain the checkpoint file associated with the previous backup;
read a portion of the checkpoint file associated with the previous backup corresponding to objects assigned to the prefetcher; and
compare the objects included in the read portion of the checkpoint file associated with the previous backup to objects included in an updated portion of the updated checkpoint file associated with a current backup to determine the one or more differences.
12 . The computing system of claim 11 , wherein the objects included in the checkpoint file associated with the previous backup and the updated checkpoint file associated with the current backup are listed in lexicographically sorted order.
13 . The computing system of claim 11 , wherein the prefetcher is configured to identify one or more of the plurality of differences between a current state of an object storage bucket and a previous state of the object storage bucket corresponding to the previous backup.
14 . The computing system of claim 9 , wherein data associated with the one or more determined differences includes one or more object metadata and object data content.
15 . The computing system of claim 9 , wherein data associated with the plurality of differences indicates that an object was deleted after the previous backup.
16 . The computing system of claim 9 , wherein data associated with the one or more determined differences includes one or more of metadata associated with an object that changed after the previous backup, changed data associated with an object after the previous backup, object metadata and object content data associated with an object that was created after the previous backup.
17 . Non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to execute a prefetcher configured to:
invoke an application programming interface to obtain a list of objects associated with a subgroup assigned to the prefetcher;
update, in response to determining that an entry of the obtained list of objects includes a partition identifier associated with a subsequent subgroup, a checkpoint file to include entries corresponding to the list of objects except the entry corresponding to the partition identifier associated with the subsequent subgroup an any entries that are sequentially after the entry that includes the partition identifier associated with the subsequent subgroup;
receive an indication that the checkpoint file has been updated;
determine a plurality of differences between the updated checkpoint file and a checkpoint file associated with a previous backup; and
ingest, concurrent to determining at least one of the plurality of differences, the plurality of determined differences.