Method to perform crash and failure recovery for a virtualized checkpoint protected storage system

1. A method for recovering content, comprising:

isolating a data object from processes other than a manager process;

backing out uncommitted changes of the data object based on one or more undo logs;

determining a most recent checkpoint for the data object;

reinitializing the data object from the most recent checkpoint to produce a reinitialized data object; and

populating the reinitialized data object to produce a recovered data object by replaying, from a write-ahead log, a sequence of operations that occurred after a time at which the most recent checkpoint was generated, wherein replaying the sequence of operations recovers data and metadata of the data object.

2. The method of claim 1 , wherein the sequence of operations comprises operations corresponding to uncommitted log entries of the write-ahead log.

3. The method of claim 1 , further comprising: checkpointing the data object to persistent storage after a subset of the sequence of operations has replayed.

4. The method of claim 1 , wherein replaying the sequence of operations comprises:

releasing a first amount of storage space for the data object based on a comparison of entries of the write-ahead log to entries of a delete log, wherein releasing comprises indicating, on a free list, space available for writing;

releasing a second storage space for the data object by traversing the write-ahead log for uncommitted write entries and confirming, against a confirmation log, or checksum-verifying, against checksums of the uncommitted write entries and of referenced storage space of the uncommitted write entries, the uncommitted write entries, wherein the uncommitted write entries occurred after a time at which the most recent checkpoint was generated, and wherein confirmed and checksum-verified uncommitted write entries are to be replayed; and

calculating freed storage space based on the released first storage space and the released second storage space for the confirmed and checksum-verified uncommitted write entries.

5. The method of claim 4 , wherein the comparison of the write-ahead log to the delete log comprises correlating entries of freed physical block addresses listed on a free list to entries of the delete log comprising entries of freed storage space from checkpoint delete operations.

6. The method of claim 4 , wherein releasing the first amount of storage space comprises releasing storage space corresponding to a physical block address indicated in either the delete log or the write-ahead log but not both the delete log and the write-ahead log.

7. The method of claim 1 , further comprising: checkpointing the data object when a number of uncommitted log entries of the write-ahead log exceeds one or more thresholds.

8. A non-transitory computer-readable storage medium embodying computer program instructions for recovering content for an object, the computer program instructions implementing a method, the method comprising:

isolating a data object from other processes other than a manager process;

backing out uncommitted changes of the data object based on one or more undo logs;

determining a most recent checkpoint for the data object;

reinitializing the data object from the most recent checkpoint to produce a reinitialized data object; and

populating the reinitialized data object to produce a recovered data object by replaying, from a write-ahead log, a sequence of operations that occurred after a time at which the most recent checkpoint was generated, wherein replaying the sequence of operations recovers data and metadata of the data object.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the sequence of operations comprises operations corresponding to uncommitted log entries of the write-ahead log.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the method further comprises: checkpointing the data object to persistent storage after a subset of the sequence of operations has replayed.

11. The non-transitory computer-readable storage medium of claim 8 , wherein replaying the sequence of operations comprises:

releasing a first amount of storage space for the data object based on a comparison of entries of the write-ahead log to entries of a delete log, wherein releasing comprises indicating, on a free list, space available for writing;

releasing a second storage space for the data object by traversing the write-ahead log for uncommitted write entries and confirming, against a confirmation log, or checksum-verifying, against checksums of the uncommitted write entries and of referenced storage space of the uncommitted write entries, the uncommitted write entries, wherein the uncommitted write entries occurred after a time at which the most recent checkpoint was generated, and wherein confirmed and checksum-verified uncommitted write entries are to be replayed; and

calculating freed storage space based on the released first storage space and the released second storage space for the confirmed and checksum-verified uncommitted write entries.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the comparison of the write-ahead log to the delete log comprises correlating entries of freed physical block addresses listed on a free list to entries of the delete log comprising entries of freed storage space from checkpoint delete operations.

13. The non-transitory computer-readable storage medium of claim 11 , wherein releasing the first amount of storage space comprises releasing storage space corresponding to a physical block address indicated in either the delete log or the write-ahead log but not both the delete log and the write-ahead log.

14. The non-transitory computer-readable storage medium of claim 8 , wherein the method further comprises: checkpointing the data object when a number of uncommitted log entries of the write-ahead log exceeds one or more thresholds.

15. A system, comprising:

a processor; and

non-transitory computer-readable storage medium embodying computer program instructions for recovering content for an object, the computer program instructions implementing a method, the method comprising:

isolating a data object from other processes other than a manager process;

backing out uncommitted changes of the data object based on one or more undo logs;

determining a most recent checkpoint for the data object;

reinitializing the data object from the most recent checkpoint to produce a reinitialized data object; and

populating the reinitialized data object to produce a recovered data object by replaying, from a write-ahead log, a sequence of operations that occurred after a time at which the most recent checkpoint was generated, wherein replaying the sequence of operations recovers data and metadata of the data object.

16. The system of claim 15 , wherein the sequence of operations comprises operations corresponding to uncommitted log entries of the write-ahead log.

17. The system of claim 15 , wherein the method further comprises: checkpointing the data object to persistent storage after a subset of the sequence of operations has replayed.

18. The system of claim 15 , wherein replaying the sequence of operations comprises:

releasing a first amount of storage space for the data object based on a comparison of entries of the write-ahead log to entries of a delete log, wherein releasing comprises indicating, on a free list, space available for writing;

releasing a second storage space for the data object by traversing the write-ahead log for uncommitted write entries and confirming, against a confirmation log, or checksum-verifying, against checksums of the uncommitted write entries and of referenced storage space of the uncommitted write entries, the uncommitted write entries, wherein the uncommitted write entries occurred after a time at which the most recent checkpoint was generated, and wherein confirmed and checksum-verified uncommitted write entries are to be replayed; and

calculating freed storage space based on the released first storage space and the released second storage space for the confirmed and checksum-verified uncommitted write entries.

19. The system of claim 18 , wherein the comparison of the write-ahead log to the delete log comprises correlating entries of freed physical block addresses listed on a free list to entries of the delete log comprising entries of freed storage space from checkpoint delete operations.

20. The system of claim 18 , wherein releasing the first amount of storage space comprises releasing storage space corresponding to a physical block address indicated in either the delete log or the write-ahead log but not both the delete log and the write-ahead log.