Distributed write journals that support fast snapshotting for a distributed file system
Embodiments presented herein disclose techniques for capturing a snapshot of a file system object (e.g., a file or a directory) that is associated with a write journal having outstanding data. A bridge process in a storage server receives a request to capture a snapshot of a file system object. The snapshot is a backup of a state of the file system object in a given point in time. Upon determining that the file system object has one or more outstanding updates recorded in a write journal, the bridge process generates a copy of the write journal. The bridge process captures the snapshot of the file system object. The bridge process also associates the copy of the write journal with the snapshot of the file system object.
1. A method, comprising:
receiving at a first storage node of a cluster a request to capture a snapshot of a file system object;
identifying one or more write journals associated with the file system object;
determining whether the one or more identified write journals associated with the file system object include one or more corresponding updates;
in response to determining that the one or more identified write journals associated with the file system object include one or more corresponding outstanding updates, generating at the first storage node of the cluster one or more on-disk copies of the one or more identified write journals, wherein the one or more of identified write journals includes one or more corresponding outstanding updates to the file system object that are received before the request to capture the snapshot;
capturing the snapshot of the file system object to a disk of the first storage node;
associating the one or more on-disk copies of the one or more identified write journals with the snapshot of the file system object; and
executing a background maintenance process, wherein executing the background maintenance process includes flushing to the disk of the first storage node the one or more corresponding outstanding updates included in the one or more on-disk copies of the one or more identified write journals.
2. The method of claim 1 , further comprising:
evaluating one or more files and one or more directories associated with the file system object; and
determining that the one or more identified write journals associated with the file system object include one or more corresponding outstanding updates associated with the one or more files and the one or more directories associated with the file system object.
3. The method of claim 2 , wherein the one or more on-disk copies of the one or more identified write journals include the one or more corresponding outstanding updates associated with the one or more files and the one or more directories associated with the file system object.
4. The method of claim 2 , wherein one of the identified write journals associated with the file system object corresponds to one of the one or more files associated with the file system object.
5. The method of claim 2 , wherein one of the identified write journals associated with the file system object corresponds to one of the one or more directories associated with the file system object.
6. The method of claim 1 , wherein a plurality of write journals are stored across a plurality of storage nodes of the cluster, wherein the one or more write journals associated with the file system object are a subset of the plurality of write journals, wherein the one or more write journals associated with the file system object are stored across the plurality of storage nodes.
7. The method of claim 6 , wherein a storage node of the plurality of storage nodes includes one or more hard disk drives and one or more solid state drives.
8. The method of claim 7 , wherein at least one of the plurality of write journals is stored in the one or more solid state drives of the storage node.
9. The method of claim 1 , wherein an identified write journal of the one or more write journals includes one or more entries, wherein each of the one or more entries includes a corresponding sequence number and a corresponding timestamp.
10. The method of claim 1 , wherein an identified write journal of the one or more write journals includes a metadata journal.
11. The method of claim 10 , wherein the metadata journal includes one or more recorded changes related to a structure of the file system object.
12. The method of claim 1 , wherein an identified write journal of the one or more write journals includes a data journal.
13. The method of claim 12 , wherein the data journal includes one or more recorded changes to data associated with a file or a directory.
14. The method of claim 1 , wherein executing the background maintenance process further comprises sequentially flushing the one or more corresponding outstanding updates included in the one or more on-disk copies of the one or more identified write journals.
15. The method of claim 1 , wherein the background maintenance process is scheduled based on current resources and activity associated with the system.
16. A computer program product, the computer program product being embodied in a non-transitory computer readable medium and comprising instructions for:
receiving at a first node of a cluster a request to capture a snapshot of a file system object;
identifying one or more write journals associated with the file system object;
determining whether the one or more identified write journals associated with the file system object include one or more corresponding updates;
in response to determining that the one or more identified write journals associated with the file system object include one or more corresponding outstanding updates, generating at the first node of the cluster one or more on-disk copies of the one or more identified write journals, wherein the one or more of identified write journals includes one or more corresponding outstanding updates to the file system object that are received before the request to capture the snapshot;
capturing the snapshot of the file system object to a disk of the first node;
associating the one or more on-disk copies of the one or more identified write journals with the snapshot of the file system object; and
executing a background maintenance process, wherein executing the background maintenance process includes flushing to the disk of the first node the one or more corresponding outstanding updates included in the one or more on-disk copies of the one or more identified write journals.
17. The computer program product of claim 16 , wherein executing the background maintenance process comprises sequentially flushing the one or more corresponding outstanding updates included in the one or more on-disk copies of the one or more identified write journals.
18. The computer program product of claim 17 , wherein the background maintenance process is scheduled based on current resources and activity associated with a secondary storage system.
19. The computer program product of claim 16 , further comprising:
evaluating one or more files and one or more directories associated with the file system object; and
determining that the one or more identified write journals associated with the file system object include one or more corresponding outstanding updates associated with the one or more files and the one or more directories associated with the file system object.
20. A system, comprising:
a processor configured to:
receive at a first node of a cluster a request to capture a snapshot of a file system object;
identify one or more write journals associated with the file system object;
determine whether the one or more identified write journals associated with the file system object include one or more corresponding updates;
in response to a determination that the one or more identified write journals associated with the file system object include one or more corresponding outstanding updates, generate at the first node of the cluster one or more on-disk copies of the one or more identified write journals, wherein the one or more of identified write journals includes one or more corresponding outstanding updates to the file system object that are received before the request to capture the snapshot;
capture the snapshot of the file system object to a disk of the first node;
associate the one or more on-disk copies of the one or more identified write journals with the snapshot of the file system object; and
execute a background maintenance process, wherein executing the background maintenance process includes flushing to the disk of the first node the one or more corresponding outstanding updates included in the one or more on-disk copies of the one or more identified write journals; and
a memory coupled to the processor and configured to provide the processor with instructions.