Cloud data archiving using latest snapshot metadata

1. A method of operating a data processing system to archive local snapshots of a primary storage object to cloud storage, comprising:

dividing an address space of the local snapshots into sequential chunks each including a predetermined number of allocation units;

organizing the cloud storage into archived snapshots corresponding to the local snapshots, the archived snapshots including respective sets of cloud objects created from respective modified chunks of respective local snapshots, the archived snapshots organized into families each having a full snapshot and one or more incremental snapshots referenced to preceding archived snapshots of the family, a given local snapshot being restored by sequentially applying the incremental snapshots to the full snapshot in a corresponding given family;

starting a new family of archived snapshots by creating a synthesized full snapshot from the archived snapshots of a preceding family, including (1) for chunks whose data is fully contained within a respective cloud object of the preceding family, logically incorporating the cloud object into the synthesized full snapshot, and (2) for chunks whose data is contained within multiple respective cloud objects of the preceding family, coalescing the data of the cloud objects into a new cloud object and logically incorporating the new cloud object into the synthesized full snapshot; and

continually maintaining latest snapshot metadata as the archived snapshots are created over time, the latest snapshot metadata used to quickly identify chunks to be retrieved to restore the latest snapshot, the latest snapshot metadata including an allocated list and a deallocated list, the deallocated list identifying all chunks previously allocated in the family but currently deallocated, the allocated list identifying all currently allocated chunks and, for each chunk whose contents are stored in a corresponding single cloud object, an identifier of the cloud object.

2. The method of claim 1 , wherein the synthesized full snapshot serves as a reference snapshot for the new family, the new family subsequently being populated with incremental snapshots referenced to the reference snapshot of the new family.

3. The method of claim 1 , wherein the data processing system enforces a limit on a number of families retained in the cloud storage, and further including, in connection with the starting of the new family, deleting an oldest family from the cloud storage, the oldest family being deleted by deleting all snapshots of the oldest family and only those cloud objects of the oldest family that are owned by oldest family.

4. The method of claim 1 , further including persisting snapshot metadata as an object that forms part of an archived snapshot for a chunk in the cloud storage.

5. The method of claim 1 used in connection with restoring a snapshot previously archived to the cloud storage, the restoring including:

identifying chunk addresses of all chunks for the snapshot;

for each chunk address:

a) utilizing a single object from one of the snapshots of the family when there is only one object at that address and when a latest object is a superset of all earlier objects; and

b) otherwise, coalescing objects starting from latest object until all blocks in the chunk are retrieved.

6. The method of claim 5 , wherein coalescing an object includes starting at a latest snapshot and successively applying modified and deleted objects from earlier snapshots.

7. The method of claim 1 , wherein maintaining the latest snapshot metadata includes:

when a chunk first appears for a family, as indicated by there being no entry for the chunk in the allocated list, (1) adding a new entry in the allocated list and setting the object identifier for the corresponding cloud object, and (2) deleting any existing entry for this chunk in the deallocated list; and

when an existing chunk is updated by modification or deallocation, as indicated by an entry already existing in the allocated list, (1) incorporating metadata bits of the chunk into existing the entry, (2) if the resulting bitmap is a superset for the updated chunk, then updating the object identifier to point to the single superset object, and otherwise clearing the object identifier, and (3) if the resulting bitmap has no bit set, indicating entire deallocation of the chunk, then removing the entry from the allocated list and adding a corresponding entry to the deallocated list.

8. The method of claim 1 , further including use of a per-family promoted object list to track the transfer of ownership of objects from a given family to another family, a respective promoted object list existing at any given moment for all families except for the latest family and being created for each family when a synthetic full snapshot is created therefor, the promoted object for a given family being used to prevent deletion of promoted objects when the family is deleted, the use of the promoted object being used in lieu of explicitly track ownership of chunks by family.

9. The method of claim 8 , further including, in connection with the starting of the new family, deleting an oldest family from the cloud storage, including:

reading all snapshot metadata in the family and identifying a unique list of objects for modified chunks of the family;

from the unique list, excluding objects that have been promoted to a next family, according to the promoted objects list;

deleting objects identified in the list after the promoted objects have been excluded; and

deleting all metadata for the family.

10. A data processing system including local storage and an interface to cloud storage, the data processing system including computer processing circuitry configured and operative to execute computer program instructions to perform a method of archiving local snapshots of a primary storage object to cloud storage, the method including:

dividing an address space of the local snapshots into sequential chunks each including a predetermined number of allocation units;

organizing the cloud storage into archived snapshots corresponding to the local snapshots, the archived snapshots including respective sets of cloud objects created from respective modified chunks of respective local snapshots, the archived snapshots organized into families each having a full snapshot and one or more incremental snapshots referenced to preceding archived snapshots of the family, a given local snapshot being restored by sequentially applying the incremental snapshots to the full snapshot in a corresponding given family;

starting a new family of archived snapshots by creating a synthesized full snapshot from the archived snapshots of a preceding family, including (1) for chunks whose data is fully contained within a respective cloud object of the preceding family, logically incorporating the cloud object into the synthesized full snapshot, and (2) for chunks whose data is contained within multiple respective cloud objects of the preceding family, coalescing the data of the cloud objects into a new cloud object and logically incorporating the new cloud object into the synthesized full snapshot; and

continually maintaining latest snapshot metadata as the archived snapshots are created over time, the latest snapshot metadata used to quickly identify chunks to be retrieved to restore the latest snapshot, the latest snapshot metadata including an allocated list and a deallocated list, the deallocated list identifying all chunks previously allocated in the family but currently deallocated, the allocated list identifying all currently allocated chunks and, for each chunk whose contents are stored in a corresponding single cloud object, an identifier of the cloud object.

11. The data processing system of claim 10 , wherein the synthesized full snapshot serves as a reference snapshot for the new family, the new family subsequently being populated with incremental snapshots referenced to the reference snapshot of the new family.

12. The data processing system of claim 10 , wherein the method further includes enforcing a limit on a number of families retained in the cloud storage, and, in connection with the starting of the new family, deleting an oldest family from the cloud storage, the oldest family being deleted by deleting all snapshots of the oldest family and only those cloud objects of the oldest family that are owned by oldest family.

13. The data processing system of claim 10 , wherein the method further includes persisting snapshot metadata as an object that forms part of an archived snapshot for a chunk in the cloud storage.

14. The data processing system of claim 10 , wherein the method is used in connection with restoring a snapshot previously archived to the cloud storage, the restoring including:

identifying chunk addresses of all chunks for the snapshot;

for each chunk address:

a) utilizing a single object from one of the snapshots of the family when there is only one object at that address and when a latest object is a superset of all earlier objects; and

b) otherwise, coalescing objects starting from latest object until all blocks in the chunk are retrieved.

15. The data processing system of claim 14 , wherein coalescing an object includes starting at a latest snapshot and successively applying modified and deleted objects from earlier snapshots.

16. The data processing system of claim 10 , wherein maintaining the latest snapshot metadata includes:

when a chunk first appears for a family, as indicated by there being no entry for the chunk in the allocated list, (1) adding a new entry in the allocated list and setting the object identifier for the corresponding cloud object, and (2) deleting any existing entry for this chunk in the deallocated list; and

when an existing chunk is updated by modification or deallocation, as indicated by an entry already existing in the allocated list, (1) incorporating metadata bits of the chunk into existing the entry, (2) if the resulting bitmap is a superset for the updated chunk, then updating the object identifier to point to the single superset object, and otherwise clearing the object identifier, and (3) if the resulting bitmap has no bit set, indicating entire deallocation of the chunk, then removing the entry from the allocated list and adding a corresponding entry to the deallocated list.

17. The data processing system of claim 10 , wherein the method further includes use of a per-family promoted object list to track the transfer of ownership of objects from a given family to another family, a respective promoted object list existing at any given moment for all families except for the latest family and being created for each family when a synthetic full snapshot is created therefor, the promoted object for a given family being used to prevent deletion of promoted objects when the family is deleted, the use of the promoted object being used in lieu of explicitly track ownership of chunks by family.

18. The data processing system of claim 17 , wherein the method further includes, in connection with the starting of the new family, deleting an oldest family from the cloud storage, including:

reading all snapshot metadata in the family and identifying a unique list of objects for modified chunks of the family;

from the unique list, excluding objects that have been promoted to a next family, according to the promoted objects list;

deleting objects identified in the list after the promoted objects have been excluded; and

deleting all metadata for the family.