Snapshot and restoration of distributed file system
In some examples, a data management system processes snapshots of a distributed file system, the distributed file system having files, each file comprising multiple data chunks. The data management system performs operations including storing file-to-chunk mapping in file system metadata; creating, for each chunk, a chunk generation ID by associating each chunk with a file system generation ID; in a next-generation snapshot of the distributed file system, incrementing, for all chunks in the next-generation snapshot, the respective chunk generation IDs; and taking a snapshot of the file system metadata and storing an updated file-to-chunk mapping associated with the next-generation snapshot.
1. A method of processing snapshots in a distributed file system, including:
storing file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps at least a file of the distributed file system to a respective set of data chunks that are included in the file at a first time associated with a first file system generation identifier (ID), and wherein each file of the distributed file system comprises multiple data chunks;
creating, for each data chunk included in the distributed file system, a chunk generation ID by associating each data chunk with a respective file system generation ID, wherein the respective file system generation ID for a data chunk corresponds to a time at which an earliest snapshot that includes the data chunk was obtained;
taking, at a second time associated with a second file system generation ID, a next-generation snapshot of the distributed file system, the next-generation snapshot comprising one or more data chunks of the distributed file system;
incrementing, for all data chunks of the one or more data chunks included in the next-generation snapshot, the respective chunk generation IDs based at least in part on taking the next-generation snapshot; and
updating, based at least in part on taking the next-generation snapshot of the distributed file system, the file-to-chunk mapping in the file system metadata, wherein the updated file-to-chunk mapping maps the file of the distributed file system to a second set of data chunks that are included in the file at the second time associated with the second file system generation ID.
2. The method of claim 1 , wherein a first snapshot associated with a first file system generation ID is set to expire, the method further comprising:
identifying a plurality of first data chunks included in the first snapshot based at least in part on the plurality of first data chunks comprising chunk generation IDs associated with the first file system generation ID, wherein one or more first data chunks of the plurality of first data chunks are included in the updated file-to-chunk mapping in the file system metadata; and
garbage-collecting remaining first data chunks of the plurality of first data chunks that are included in the first snapshot based at least in part on the first snapshot being set to expire and the remaining first data chunks being excluded from the file-to- chunk mapping in the file system metadata.
3. The method of claim 1 , further comprising:
rolling back the distributed file system to a previous snapshot of the distributed file system obtained before the next-generation snapshot of the distributed file system based at least in part on a chunk generation ID associated with the previous snapshot and the file system metadata comprising the file-to-chunk mapping associated with the previous snapshot.
4. The method of claim 3 , further comprising:
garbage-collecting unreachable data chunks after the rollback of the distributed file system to the previous snapshot.
5. A data management system to process snapshots of a distributed file system, comprising at least one processor configured to perform operations including:
storing file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps at least a file of the distributed file system to a respective set of data chunks that are included in the file at a first time associated with a first file system generation identifier (ID), and wherein each file of the distributed file system comprises multiple data chunks;
creating, for each data chunk included in the distributed file system, a chunk generation ID by associating each data chunk with a respective file system generation ID, wherein the respective file system generation ID for a data chunk corresponds to a time at which an earliest snapshot that includes the data chunk was obtained;
taking, at a second time associated with a second file system generation ID, a next-generation snapshot of the distributed file system, the next-generation snapshot comprising one or more data chunks of the distributed file system;
incrementing, for all data chunks of the one or more data chunks included in the next-generation snapshot, the respective chunk generation IDs based at least in part on taking the next-generation snapshot; and
updating, based at least in part on taking the next-generation snapshot of the distributed file system, the file-to-chunk mapping in the file system metadata, wherein the updated file-to-chunk mapping maps the file of the distributed file system to a second set of data chunks that are included in the file at the second time associated with the second file system generation ID.
6. The data management system of claim 5 , wherein a first snapshot associated with a first file system generation ID is set to expire, the operations further comprising:
identifying a plurality of first data chunks included in the first snapshot based at least in part on the plurality of first data chunks comprising chunk generation IDs associated with the first file system generation ID, wherein one or more first data chunks of the plurality of first data chunks are included in the updated file-to-chunk mapping in the file system metadata; and
garbage-collecting remaining first data chunks of the plurality of first data chunks that are included in the first snapshot based at least in part on the first snapshot being set to expire and the remaining first data chunks being excluded from the file-to- chunk mapping in the file system metadata.
7. The data management system of claim 5 , the operations further comprising:
rolling back the distributed file system to a previous snapshot of the distributed file system obtained before the next-generation snapshot of the distributed file system based at least in part on a chunk generation ID associated with the previous snapshot and the file system metadata comprising the file-to-chunk mapping associated with the previous snapshot.
8. The data management system of claim 7 , the operations further comprising:
garbage-collecting unreachable data chunks after the rollback of the distributed file system to the previous snapshot.
9. A tangible machine-readable medium including instructions which, when read by a machine, cause a machine to perform operations in a method of processing snapshots in a distributed file system, the operations including:
storing file-to-chunk mapping in file system metadata, wherein the file-to-chunk mapping maps at least a file of the distributed file system to a respective set of data chunks that are included in the file at a first time associated with a first file system generation identifier (ID), and wherein each file of the distributed file system comprises multiple data chunks;
creating, for each data chunk included in the distributed file system, a chunk generation ID by associating each data chunk with a respective file system generation ID, wherein the respective file system generation ID for a data chunk corresponds to a time at which an earliest snapshot that includes the data chunk was obtained;
taking, at a second time associated with a second file system generation ID, a next-generation snapshot of the distributed file system, the next-generation snapshot comprising one or more data chunks of the distributed file system;
incrementing, for all data chunks of the one or more data chunks included in the next-generation snapshot, the respective chunk generation IDs based at least in part on taking the next-generation snapshot; and
updating, based at least in part on taking the next-generation snapshot of the distributed file system, the file-to-chunk mapping in the file system metadata, wherein the updated file-to-chunk mapping maps the file of the distributed file system to a second set of data chunks that are included in the file at the second time associated with the second file system generation ID.
10. The medium of claim 9 , wherein a first snapshot associated with a first file system generation ID is set to expire, the operations further comprising:
identifying a plurality of first data chunks included in the first snapshot based at least in part on the plurality of first data chunks comprising chunk generation IDs associated with the first file system generation ID, wherein one or more first data chunks of the plurality of first data chunks are included in the updated file-to-chunk mapping in the file system metadata; and
garbage-collecting remaining first data chunks of the plurality of first data chunks that are included in the first snapshot based at least in part on the first snapshot being set to expire and the remaining first data chunks being excluded from the file-to- chunk mapping in the file system metadata.
11. The medium of claim 9 , the operations further comprising:
rolling back the distributed file system to a previous snapshot of the distributed file system obtained before the next-generation snapshot of the distributed file system based at least in part on a chunk generation ID associated with the previous snapshot and the file system metadata comprising the file-to-chunk mapping associated with the previous snapshot.
12. The medium of claim 11 , the operations further comprising:
garbage-collecting unreachable data chunks after the rollback of the distributed file system to the previous snapshot.