Snapshot and restoration of distributed file system
In some examples, a data management system processes snapshots of a distributed file system, the distributed file system having files, each file comprising multiple data chunks. The data management system performs operations including storing file-to-chunk mapping in file system metadata; creating, for each chunk, a chunk generation ID by associating each chunk with a file system generation ID; in a next-generation snapshot of the distributed file system, incrementing, for all chunks in the next-generation snapshot, the respective chunk generation IDs; and taking a snapshot of the file system metadata and storing an updated file-to-chunk mapping associated with the next-generation snapshot.
1 . A method comprising:
updating, based at least in part on obtaining a second snapshot of a distributed file system at a second time, a file-to-chunk mapping that maps a file of the distributed file system to one or more data chunks and one or more file system generation identifiers (IDs) associated with the one or more data chunks, wherein updating the file-to-chunk mapping comprises:
changing, in the file-to-chunk mapping, metadata from a first mapping between the file, a first data chunk, and a first generation ID associated with a first time at which a first snapshot of the distributed file system is obtained to a second mapping between the file, a second data chunk, and a second generation ID associated with the second time at which the second snapshot of the distributed file system is obtained, the changing from the first mapping to the second mapping being based at least in part on the second data chunk comprising changes to the first data chunk that occurred between the first time and the second time;
receiving a request to restore the file to the first time associated with the first generation ID; and
updating the file-to-chunk mapping based at least in part on the request, wherein updating the file-to-chunk mapping comprises:
reverting, in the file-to-chunk mapping, the metadata from the second mapping back to the first mapping between the file, the first data chunk, and the first generation ID based at least in part on the request to restore the file to the first time associated with the first generation ID, wherein, after reverting the metadata to the second mapping, the file-to-chunk mapping excludes the second data chunk comprising the changes to the first data chunk that occurred between the first time and the second time.
2 . The method of claim 1 , further comprising:
obtaining, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first data chunk at the first time; and
storing, based at least in part on obtaining the first snapshot, the file-to-chunk mapping comprising the first mapping between the file, the first data chunk, and the first generation ID associated with the first time based at least in part on the file comprising the first data chunk at the first time.
3 . The method of claim 1 , further comprising:
obtaining, at the second time that is after the first time, the second snapshot of the distributed file system, wherein the file includes the second data chunk at the second time, and wherein changing the metadata from the first mapping to the second mapping is based at least in part on obtaining the second snapshot at the second time.
4 . The method of claim 1 , further comprising:
generating, after the first time and in accordance with the changes to the first data chunk, the second data chunk associated with the second generation ID;
copying physical data stored in the first data chunk at the first time to the second data chunk based at least in part on generating the second data chunk; and
writing, to the second data chunk, data associated with the changes to the first data chunk that occurred between the first time and the second time, wherein the changes are not included in the file-to-chunk mapping after reverting the metadata from the second mapping back to the first mapping based at least in part on the request.
5 . The method of claim 1 , wherein updating the file-to-chunk mapping based at least in part on obtaining the second snapshot of the distributed file system at the second time comprises:
writing, to the file-to-chunk mapping, second metadata associated with a third mapping between the file, a third data chunk, and the second generation ID based at least in part on the file comprising the third data chunk at the second time.
6 . The method of claim 5 , wherein updating the file-to-chunk mapping based at least in part on the request comprises:
removing, from the file-to-chunk mapping, the second metadata associated with the third mapping between the file, the third data chunk, and the second generation ID based at least in part on the third data chunk being mapped to the second generation ID and being absent from the file at the first time.
7 . The method of claim 1 , wherein updating the file-to-chunk mapping based at least in part on obtaining the second snapshot of the distributed file system at the second time comprises:
maintaining, in the file-to-chunk mapping, a third mapping between the file, a third data chunk, and the first generation ID based at least in part on the file comprising the third data chunk at the first time and the second time and based at least in part on a state of the third data chunk being maintained between the first time and the second time.
8 . The method of claim 7 , wherein the third mapping is further maintained in the file-to-chunk mapping after reverting the metadata from the second mapping back to the first mapping based at least in part on the request.
9 . An apparatus, comprising:
one or more memories storing processor-executable code; and
one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
update, based at least in part on obtaining a second snapshot of a distributed file system at a second time, a file-to-chunk mapping that maps a file of the distributed file system to one or more data chunks and one or more file system generation identifiers (IDs) associated with the one or more data chunks, wherein, to update the file-to-chunk mapping, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
change, in the file-to-chunk mapping, metadata from a first mapping between the file, a first data chunk, and a first generation ID associated with a first time at which a first snapshot of the distributed file system is obtained to a second mapping between the file, a second data chunk, and a second generation ID associated with the second time at which the second snapshot of the distributed file system is obtained, the changing from the first mapping to the second mapping being based at least in part on the second data chunk comprising changes to the first data chunk that occurred between the first time and the second time;
receive a request to restore the file to the first time associated with the first generation ID; and
update the file-to-chunk mapping based at least in part on the request, wherein, to update the file-to-chunk mapping, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
revert, in the file-to-chunk mapping, the metadata from the second mapping back to the first mapping between the file, the first data chunk, and the first generation ID based at least in part on the request to restore the file to the first time associated with the first generation ID, wherein, after reverting the metadata to the second mapping, the file-to-chunk mapping excludes the second data chunk comprising the changes to the first data chunk that occurred between the first time and the second time.
10 . The apparatus of claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
obtain, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first data chunk at the first time; and
store, based at least in part on obtaining the first snapshot, the file-to-chunk mapping comprising the first mapping between the file, the first data chunk, and the first generation ID associated with the first time based at least in part on the file comprising the first data chunk at the first time.
11 . The apparatus of claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
obtain, at the second time that is after the first time, the second snapshot of the distributed file system, wherein the file includes the second data chunk at the second time, and wherein changing the metadata from the first mapping to the second mapping is based at least in part on obtaining the second snapshot at the second time.
12 . The apparatus of claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
generate, after the first time and in accordance with the changes to the first data chunk, the second data chunk associated with the second generation ID;
copy physical data stored in the first data chunk at the first time to the second data chunk based at least in part on generating the second data chunk; and
write, to the second data chunk, data associated with the changes to the first data chunk that occurred between the first time and the second time, wherein the changes are not included in the file-to-chunk mapping after reverting the metadata from the second mapping back to the first mapping based at least in part on the request.
13 . The apparatus of claim 9 , wherein, to update the file-to-chunk mapping based at least in part on obtaining the second snapshot of the distributed file system at the second time, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
write, to the file-to-chunk mapping, second metadata associated with a third mapping between the file, a third data chunk, and the second generation ID based at least in part on the file comprising the third data chunk at the second time.
14 . The apparatus of claim 13 , wherein, to update the file-to-chunk mapping based at least in part on the request, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
remove, from the file-to-chunk mapping, the second metadata associated with the third mapping between the file, the third data chunk, and the second generation ID based at least in part on the third data chunk being mapped to the second generation ID and being absent from the file at the first time.
15 . The apparatus of claim 9 , wherein, to update the file-to-chunk mapping based at least in part on obtaining the second snapshot of the distributed file system at the second time, the one or more processors are individually or collectively operable to execute the code to cause the apparatus to:
maintain, in the file-to-chunk mapping, a third mapping between the file, a third data chunk, and the first generation ID based at least in part on the file comprising the third data chunk at the first time and the second time and based at least in part on a state of the third data chunk being maintained between the first time and the second time.
16 . The apparatus of claim 15 , wherein the third mapping is further maintained in the file-to-chunk mapping after reverting the metadata from the second mapping back to the first mapping based at least in part on the request.
17 . A non-transitory computer-readable medium storing code, the code comprising instructions executable by one or more processors to:
update, based at least in part on obtaining a second snapshot of a distributed file system at a second time, a file-to-chunk mapping that maps a file of the distributed file system to one or more data chunks and one or more file system generation identifiers (IDs) associated with the one or more data chunks, wherein, to update the file-to-chunk mapping, the instructions are executable by the one or more processors to:
change, in the file-to-chunk mapping, metadata from a first mapping between the file, a first data chunk, and a first generation ID associated with a first time at which a first snapshot of the distributed file system is obtained to a second mapping between the file, a second data chunk, and a second generation ID associated with the second time at which the second snapshot of the distributed file system is obtained, the changing from the first mapping to the second mapping being based at least in part on the second data chunk comprising changes to the first data chunk that occurred between the first time and the second time;
receive a request to restore the file to the first time associated with the first generation ID; and
update the file-to-chunk mapping based at least in part on the request, wherein, to update the file-to-chunk mapping, the instructions are executable by the one or more processors to:
revert, in the file-to-chunk mapping, the metadata from the second mapping back to the first mapping between the file, the first data chunk, and the first generation ID based at least in part on the request to restore the file to the first time associated with the first generation ID, wherein, after reverting the metadata to the second mapping, the file-to-chunk mapping excludes the second data chunk comprising the changes to the first data chunk that occurred between the first time and the second time.
18 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the one or more processors to:
obtain, at the first time, the first snapshot of the distributed file system, wherein the file comprises the first data chunk at the first time; and
store, based at least in part on obtaining the first snapshot, the file-to-chunk mapping comprising the first mapping between the file, the first data chunk, and the first generation ID associated with the first time based at least in part on the file comprising the first data chunk at the first time.
19 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the one or more processors to:
obtain, at the second time that is after the first time, the second snapshot of the distributed file system, wherein the file includes the second data chunk at the second time, and wherein changing the metadata from the first mapping to the second mapping is based at least in part on obtaining the second snapshot at the second time.
20 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the one or more processors to:
generate, after the first time and in accordance with the changes to the first data chunk, the second data chunk associated with the second generation ID;
copy physical data stored in the first data chunk at the first time to the second data chunk based at least in part on generating the second data chunk; and
write, to the second data chunk, data associated with the changes to the first data chunk that occurred between the first time and the second time, wherein the changes are not included in the file-to-chunk mapping after reverting the metadata from the second mapping back to the first mapping based at least in part on the request.