Snapshot and restoration of distributed file system
In some examples, a data management system processes snapshots of a distributed file system, the distributed file system having files, each file comprising multiple data chunks. The data management system performs operations including storing file-to-chunk mapping in file system metadata; creating, for each chunk, a chunk generation ID by associating each chunk with a file system generation ID; in a next-generation snapshot of the distributed file system, incrementing, for all chunks in the next-generation snapshot, the respective chunk generation IDs; and taking a snapshot of the file system metadata and storing an updated file-to-chunk mapping associated with the next-generation snapshot.
1. A method comprising:
storing a file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps a file of a distributed file system to a first set of data chunks and a first file system generation identifier (ID) associated with a first time at which a first snapshot of the distributed file system was obtained;
detecting, after the first time, a trigger event that indicates that the first snapshot is set to expire; and
deleting, based at least in part on the trigger event, a second set of data chunks from the distributed file system based at least in part on the trigger event and further based at least in part on the second set of data chunks being associated with the first file system generation ID and being excluded from the file-to-chunk mapping in the file system metadata, wherein the first set of data chunks remain in the distributed file system after the first snapshot expires based at least in part on the first set of data chunks being included in the file-to-chunk mapping in the file system metadata.
2. The method of claim 1 , further comprising:
obtaining, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first set of data chunks and the second set of data chunks at the first time, and wherein, based at least in part on the file including the first set of data chunks and the second set of data chunks at the first time, the file-to-chunk mapping maps the file to a first entry associated with the first set of data chunks and the first file system generation ID and a second entry associated with the second set of data chunks and the first file system generation ID.
3. The method of claim 2 , further comprising:
copying, based at least in part on an update to physical data after the first time and before a second time associated with a second snapshot of the distributed file system, the physical data from the second set of data chunks to a third set of data chunks; and
removing the second entry associated with the second set of data chunks and the first file system generation ID from the file-to-chunk mapping based at least in part on copying the physical data from the second set of data chunks to the third set of data chunks, wherein the second set of data chunks are associated with the first file system generation ID and excluded from the file-to-chunk mapping based at least in part on removing the second entry.
4. The method of claim 2 , further comprising:
obtaining, at a second time that is after the first time, a second snapshot of the distributed file system, wherein the second time is associated with a second file system generation ID, and wherein the file includes at least a third set of data chunks at the second time; and
modifying, based at least in part on obtaining the second snapshot, the file-to-chunk mapping to map the file to the first entry associated with the first set of data chunks and the first file system generation ID and a third entry associated with the third set of data chunks and the second file system generation ID.
5. The method of claim 4 , wherein:
the trigger event is detected after modifying the file-to-chunk mapping to map the file to the first entry and the third entry; and
the third set of data chunks remain in the distributed file system after the first snapshot expires based at least in part on the third set of data chunks being included in the file-to-chunk mapping in the file system metadata after the first snapshot expires.
6. The method of claim 1 , further comprising:
obtaining, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first set of data chunks and one or more other data chunks at the first time; and
maintaining, based at least in part on a state of the first set of data chunks being maintained between the first time and the trigger event, the file-to-chunk mapping that maps the file to the first set of data chunks and the first file system generation ID.
7. The method of claim 1 , wherein detecting the trigger event comprises:
receiving, via an interface, a request to delete the first snapshot.
8. The method of claim 1 , wherein detecting the trigger event comprises:
detecting that a periodicity associated with garbage collection operations for the distributed file system.
9. An apparatus, comprising:
one or more memories storing processor-executable code; and
one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
store a file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps a file of a distributed file system to a first set of data chunks and a first file system generation identifier (ID) associated with a first time at which a first snapshot of the distributed file system was obtained;
detect, after the first time, a trigger event that indicates that the first snapshot is set to expire; and
delete, based at least in part on the trigger event, a second set of data chunks from the distributed file system based at least in part on the trigger event and further based at least in part on the second set of data chunks being associated with the first file system generation ID and being excluded from the file-to-chunk mapping in the file system metadata, wherein the one or more processors are individually or collectively operable to execute the code to cause the apparatus to maintain the first set of data chunks in the distributed file system after the first snapshot expires based at least in part on the first set of data chunks being included in the file-to-chunk mapping in the file system metadata.
10. The apparatus of claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
obtain, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first set of data chunks and the second set of data chunks at the first time, and wherein, based at least in part on the file including the first set of data chunks and the second set of data chunks at the first time, the file-to-chunk mapping maps the file to a first entry associated with the first set of data chunks and the first file system generation ID and a second entry associated with the second set of data chunks and the first file system generation ID.
11. The apparatus of claim 10 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
copy, based at least in part on an update to physical data after the first time and before a second time associated with a second snapshot of the distributed file system, the physical data from the second set of data chunks to a third set of data chunks; and
remove the second entry associated with the second set of data chunks and the first file system generation ID from the file-to-chunk mapping based at least in part on copying the physical data from the second set of data chunks to the third set of data chunks, wherein the second set of data chunks are associated with the first file system generation ID and excluded from the file-to-chunk mapping based at least in part on removing the second entry.
12. The apparatus of claim 10 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
obtain, at a second time that is after the first time, a second snapshot of the distributed file system, wherein the second time is associated with a second file system generation ID, and wherein the file includes at least a third set of data chunks at the second time; and
modify, based at least in part on obtaining the second snapshot, the file-to-chunk mapping to map the file to the first entry associated with the first set of data chunks and the first file system generation ID and a third entry associated with the third set of data chunks and the second file system generation ID.
13. The apparatus of claim 12 , wherein:
the one or more processors are individually or collectively operable to execute the code to cause the apparatus to detect the trigger event after modifying the file-to-chunk mapping to map the file to the first entry and the third entry; and
the one or more processors are individually or collectively operable to execute the code to cause the apparatus to maintain the third set of data chunks in the distributed file system after the first snapshot expires based at least in part on the third set of data chunks being included in the file-to-chunk mapping in the file system metadata after the first snapshot expires.
14. The apparatus of claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
obtain, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first set of data chunks and one or more other data chunks at the first time; and
maintain, based at least in part on a state of the first set of data chunks being maintained between the first time and the trigger event, the file-to-chunk mapping that maps the file to the first set of data chunks and the first file system generation ID.
15. The apparatus of claim 9 , wherein, to detect the trigger event, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
receive, via an interface associated with the apparatus, a request to delete the first snapshot.
16. The apparatus of claim 9 , wherein, to detect the trigger event, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
detect that a periodicity associated with garbage collection operations for the distributed file system.
17. A non-transitory computer-readable medium storing code, the code comprising instructions executable by at one or more processors to:
store a file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps a file of a distributed file system to a first set of data chunks and a first file system generation identifier (ID) associated with a first time at which a first snapshot of the distributed file system was obtained;
detect, after the first time, a trigger event that indicates that the first snapshot is set to expire; and
delete, based at least in part on the trigger event, a second set of data chunks from the distributed file system based at least in part on the trigger event and further based at least in part on the second set of data chunks being associated with the first file system generation ID and being excluded from the file-to-chunk mapping in the file system metadata, wherein the instructions are further executable by the one or more processors to maintain the first set of data chunks in the distributed file system after the first snapshot expires based at least in part on the first set of data chunks being included in the file-to-chunk mapping in the file system metadata.
18. The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the one or more processors to:
obtain, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first set of data chunks and the second set of data chunks at the first time, and wherein, based at least in part on the file including the first set of data chunks and the second set of data chunks at the first time, the file-to-chunk mapping maps the file to a first entry associated with the first set of data chunks and the first file system generation ID and a second entry associated with the second set of data chunks and the first file system generation ID.
19. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable by the one or more processors to:
copy, based at least in part on an update to physical data after the first time and before a second time associated with a second snapshot of the distributed file system, the physical data from the second set of data chunks to a third set of data chunks; and
remove the second entry associated with the second set of data chunks and the first file system generation ID from the file-to-chunk mapping based at least in part on copying the physical data from the second set of data chunks to the third set of data chunks, wherein the second set of data chunks are associated with the first file system generation ID and excluded from the file-to-chunk mapping based at least in part on removing the second entry.
20. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable by the one or more processors to:
obtain, at a second time that is after the first time, a second snapshot of the distributed file system, wherein the second time is associated with a second file system generation ID, and wherein the file includes at least a third set of data chunks at the second time; and
modify, based at least in part on obtaining the second snapshot, the file-to-chunk mapping to map the file to the first entry associated with the first set of data chunks and the first file system generation ID and a third entry associated with the third set of data chunks and the second file system generation ID.