Epoch based snapshot summary
Epoch-based management of file system data is disclosed. In various embodiments, for each of a plurality of snapshots, data that associates with the snapshot a corresponding last chunk identifier assigned as of a time at which the snapshot is or was created is stored a in a first data structure. For each of a plurality of chunks of file system data no longer referenced by any live file system object, an epoch identifier associated with an epoch in which the chunk became no longer referenced by any live file system object is stored in a second data structure. A chunk identifier and associated epoch comprising an entry in the second data structure are compared with one or more entries in the first data structure to determine whether a chunk associated with the chunk identifier is associated with a snapshot that continues to be retained.
1. A method, comprising:
storing, by one or more processors, a mapping of snapshots to chunk identifiers;
storing, by one or more processors, a mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion, an epoch identifier being associated with an epoch for which the corresponding chunk was determined to no longer be referenced by a file system object that has not been subject to deletion; and
determining, by one or more processors, whether to delete a chunk based at least in part on a determination whether the chunk is referenced by the file system object, and a determination whether the chunk is associated with a snapshot that continues to be retained, wherein:
the determination whether the chunk is referenced by the file system object is based at least in part on the mapping of epoch identifiers to chunks of file system data no longer referenced by the file system object; and
the determination whether the chunk is associated with the snapshot that continues to be retained is based at least in part on the mapping of snapshots to chunk identifiers,
wherein the chunk is further determined not to be included in a snapshot by virtue of the chunk having become no longer referenced by a the file system object that has not been subject to deletion at a time prior a creation time when the snapshot was created, and the determination is made at least in part by comparing an epoch associated with the chunk identifier in a corresponding entry in the mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion to an epoch or other identifier associated with the snapshot and concluding that the epoch in which the chunk became no longer referenced is prior to or the same as an epoch immediately preceding creation of the snapshot.
2. The method of claim 1 , wherein the determining whether to delete the chunk comprises:
comparing, by one or more processors, a chunk identifier and associated epoch with one or more entries in the mapping of snapshots to chunk identifiers to determine whether the chunk associated with the chunk identifier is associated with a snapshot that continues to be retained; and
determining, by one or more processors, a status with respect to deletion of the chunk corresponding to the chunk identifiers, wherein the status with respect to deletion of the chunk is based at least in part on the comparison of the chunk identifier and associated epoch comprising an entry in a second data structure with one or more entries in a first data structure.
3. The method of claim 2 , wherein the chunk is determined not to be included in a snapshot by virtue of the chunk having been created and stored subsequent to the snapshot having been created, and the determination is made at least in part by comparing a chunk identifier of the chunk.
4. The method of claim 3 , further comprising:
deleting the chunk based at least in part on a determination made based on least in part on the comparison that the chunk associated with the chunk identifier is not associated with any snapshot that continues to be retained.
5. The method of claim 1 , wherein the chunk is determined not to be included in a snapshot by virtue of the chunk having been created and stored subsequent to the snapshot having been created, and the determination is made at least in part by comparing a chunk identifier of the chunk with a last chunk identifier associated with the snapshot in a first data structure and concluding that the chunk identifier is greater than the last chunk identifier associated with the snapshot.
6. The method of claim 5 , further comprising:
deleting the chunk based at least in part on a determination made based on least in part on the comparison that the chunk associated with the chunk identifier is not associated with any snapshot that continues to be retained.
7. The method of claim 1 , wherein the chunk identifier mapped to a corresponding snapshot is assigned as of a time at which the corresponding snapshot is or was created.
8. The method of claim 1 , wherein determining whether to delete the chunk comprises:
determining to delete the chunk in response to determining that an epoch for which the chunk was determined to no longer be referenced by a file system object that has not been subject to deletion is different from an epoch for which the chunk was created.
9. The method of claim 1 , further comprising assigning a corresponding chunk identifier to chunks of file system data, as each chunk is created and stored in the file system.
10. The method of claim 9 , wherein the chunk identifiers are monotonically increasing over time.
11. The method of claim 10 , wherein a birth epoch of a chunk is determined based on chunk identifier.
12. The method of claim 1 , wherein chunks as stored in the file system are immutable.
13. The method of claim 1 , further comprising receiving an indication that a chunk is no longer referenced by a file system object that has not been subject to deletion.
14. The method of claim 13 , further comprising adding to a delete queue data associated with the chunk.
15. The method of claim 13 , further comprising deleting the chunk immediately based at least in part on a determination that the chunk was created and became no longer referenced within a same epoch.
16. The method of claim 1 , wherein an epoch is defined with reference to a snapshot that marks either the beginning or the end of the epoch.
17. The method of claim 16 , wherein the snapshot marks the end of the epoch and the epoch extends to a next previous snapshot to the snapshot that marks the end of the epoch.
18. The method of claim 1 , wherein the snapshot corresponds to one or more of file system data and metadata.
19. The method of claim 1 , wherein each epoch is associated with one or more snapshots, and the one or more snapshots are associated with respective snapshot creation times which define one or both of a beginning and an end of the epoch.
20. A computer system, comprising:
a storage device; and
one or more processors coupled to the storage device and configured to:
store a mapping of snapshots to chunk identifiers;
store a mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion, an epoch identifier being associated with an epoch for which the corresponding chunk was determined to no longer be referenced by a file system object that has not been subject to deletion; and
determine whether to delete a chunk based at least in part a determination whether the chunk is referenced by the file system object, and a determination whether the chunk is associated with a snapshot that continues to be retained, wherein:
the determination whether the chunk is referenced by the file system object is based at least in part on the mapping of epoch identifiers to chunks of file system data no longer referenced by the file system object; and
the determination whether the chunk is associated with the snapshot that continues to be retained is based at least in part on the mapping of snapshots to chunk identifiers,
wherein the chunk is further determined not to be included in a snapshot by virtue of the chunk having become no longer referenced by a the file system object that has not been subject to deletion at a time prior a creation time when the snapshot was created, and the determination is made at least in part by comparing an epoch associated with the chunk identifier in a corresponding entry in the mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion to an epoch or other identifier associated with the snapshot and concluding that the epoch in which the chunk became no longer referenced is prior to or the same as an epoch immediately preceding creation of the snapshot.
21. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
storing, by one or more processors, a mapping of snapshots to chunk identifiers;
storing, by one or more processors, a mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion, an epoch identifier being associated with an epoch for which the corresponding chunk was determined to no longer be referenced by a file system object that has not been subject to deletion; and
determining, by one or more processors, whether to delete a chunk based at least in part on a determination whether the chunk is referenced by the file system object, and a determination whether the chunk is associated with a snapshot that continues to be retained, wherein:
the determination whether the chunk is referenced by the file system object is based at least in part on the mapping of epoch identifiers to chunks of file system data no longer referenced by the file system object; and
the determination whether the chunk is associated with the snapshot that continues to be retained is based at least in part on the mapping of snapshots to chunk identifiers,
wherein the chunk is further determined not to be included in a snapshot by virtue of the chunk having become no longer referenced by a the file system object that has not been subject to deletion at a time prior a creation time when the snapshot was created, and the determination is made at least in part by comparing an epoch associated with the chunk identifier in a corresponding entry in the mapping of epoch identifiers to chunks of file system data no longer referenced by a file system object that has not been subject to deletion to an epoch or other identifier associated with the snapshot and concluding that the epoch in which the chunk became no longer referenced is prior to or the same as an epoch immediately preceding creation of the snapshot.