Distributed write journals that support fast snapshotting for a distributed file system
Embodiments presented herein disclose techniques for capturing a snapshot of a file system object (e.g., a file or a directory) that is associated with a write journal having outstanding data. A bridge process in a storage server receives a request to capture a snapshot of a file system object. The snapshot is a backup of a state of the file system object in a given point in time. Upon determining that the file system object has one or more outstanding updates recorded in a write journal, the bridge process generates a copy of the write journal. The bridge process captures the snapshot of the file system object. The bridge process also associates the copy of the write journal with the snapshot of the file system object.
1. A method, comprising:
receiving a request to capture a snapshot of a file system object, wherein the snapshot is a backup of a state of the file system object in a given point in time;
requesting from a lock service a lock on the file system object;
receiving from the lock service a response indicating that the lock is unavailable;
in response to receiving the response indicating that the lock is unavailable, generating a copy of a write journal, wherein the write journal includes a metadata journal and a data journal, wherein the metadata journal is configured to store one or more changes to a file system associated with the file system object, wherein the data journal is configured to store one or more changes to a file or directory, wherein the write journal includes one or more outstanding updates associated with the metadata journal or the data journal, wherein the copy of the write journal includes the one or more outstanding updates associated with the metadata journal or the data journal, wherein the one or more outstanding updates are updates to the file system object that are received before the request to capture the snapshot;
capturing the snapshot of the file system object to disk;
associating the copy of the write journal with the snapshot of the file system object; and
executing a background maintenance process, wherein executing the background maintenance process includes flushing to disk the one or more outstanding updates included in the copy of the write journal.
2. The method of claim 1 , further comprising applying the one or more outstanding updates associated with the metadata journal or the data journal in the copy of the write journal to the snapshot.
3. The method of claim 1 , wherein the file system object is a directory having a plurality of files and a plurality of sub-directories.
4. The method of claim 3 , further comprising:
generating a copy of each write journal associated with each of the plurality of files and sub-directories within the directory; and
associating the generated copies of each write journal with the snapshot of the directory.
5. The method of claim 1 , wherein the write journal captures a plurality of updates to one of at least data associated with a file or metadata associated with the file.
6. The method of claim 1 , wherein the file system object is an object of a file system, wherein the file system is a distributed file system, and wherein the write journal is a distributed write journal including at least a first journal and a second journal that maintains updates for the file system object.
7. The method of claim 6 , wherein generating a copy of the write journal comprises generating a copy of at least the first journal.
8. The method of claim 1 , wherein the one or more outstanding updates associated with the metadata journal or the data journal included in the write journal and the copy of the write journal have a corresponding sequence number.
9. The method of claim 8 , wherein the background maintenance process is configured to sequentially flush to disk the one or more outstanding updates associated with the metadata journal or the data journal included in the copy of the write journal.
10. The method of claim 1 , further comprising:
receiving one or more write operations associated with the file system object; and
storing the one or more write operations associated with the file system object in the write journal, wherein the one or more write operations associated with the file system object have a corresponding sequence number.
11. The method of claim 1 , further comprising determining whether the metadata journal or the data journal are storing any outstanding updates.
12. A non-transitory computer-readable storage medium having instructions, which, when executed on a processor, causes the processor to perform the steps of:
receiving a request to capture a snapshot of a file system object, wherein the snapshot is a backup of a state of the file system object in a given point in time;
requesting from a lock service a lock on the file system object;
receiving from the lock service a response indicating that the lock is unavailable;
in response to receiving the response indicating that the lock is unavailable, generating a copy of a write journal, wherein the write journal includes a metadata journal and a data journal, wherein the metadata journal is configured to store one or more changes to a file system associated with the file system object, wherein the data journal is configured to store one or more changes to a file or directory, wherein the write journal includes one or more outstanding updates associated with the metadata journal or the data journal, wherein the copy of the write journal includes the one or more outstanding updates associated with the metadata journal or the data journal wherein the one or more outstanding updates are updates to the file system object that are received before the request to capture the snapshot;
capturing the snapshot of the file system object to disk; and
associating the copy of the write journal with the snapshot of the file system object; and
executing a background maintenance process, wherein executing the background maintenance process includes flushing to disk the one or more outstanding updates included in the copy of the write journal.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the instructions, which, when executed on a processor, further causes the processor to perform the step of applying the one or more outstanding updates associated with the metadata journal or the data journal in the copy of the write journal to the snapshot.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the file system object is a directory having a plurality of files and a plurality of sub-directories.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the processor to perform the steps of:
generating a copy of each write journal associated with each of the plurality of files and sub-directories within the directory; and
associating the generated copies of each write journal with the snapshot of the directory.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the write journal captures a plurality of updates to one of at least data associated with a file or metadata associated with the file.
17. The non-transitory computer-readable storage medium of claim 12 , wherein the file system is a distributed file system, and wherein the write journal is a distributed write journal including at least a first journal and a second journal that maintains updates for the file system object.
18. A system, comprising:
a processor configured to:
receive a request to capture a snapshot of a file system object, wherein the snapshot is a backup of a state of the file system object in a given point in time;
request from a lock service a lock on the file system object;
receive from the lock service a response indicating that the lock is unavailable;
in response to receiving the response indicating that the lock is unavailable, generate a copy of a write journal, wherein the write journal includes a metadata journal and a data journal, wherein the metadata journal is configured to store one or more changes to a file system associated with the file system object, wherein the data journal is configured to store one or more changes to a file or directory, wherein the write journal includes one or more outstanding updates associated with the metadata journal or the data journal, wherein the copy of the write journal includes the one or more outstanding updates associated with the metadata journal or the data journal, wherein the one or more outstanding updates are updates to the file system object that are received before the request to capture the snapshot;
capture the snapshot of the file system object to disk; and
associate the copy of the write journal with the snapshot of the file system object; and
execute a background maintenance process, wherein executing the background maintenance process includes flushing to disk the one or more outstanding updates included in the copy of the write journal; and
a memory coupled to the processor and configured to provide the processor with instructions.
19. The system of claim 18 , wherein the processor is further configured to apply the one or more outstanding updates associated with the metadata journal or the data journal in the copy of the write journal to the snapshot.
20. The system of claim 18 , wherein the file system object is a directory having a plurality of files and a plurality of sub-directories.
21. The system of claim 20 , wherein the processor is further configured to:
generate a copy of each write journal associated with each of the plurality of files and sub-directories within the directory; and
associate the generated copies of each write journal with the snapshot of the directory.