Concurrent access and transactions in a distributed file system
According to one embodiment of the present disclosure, a first set of file system objects included in performing a requested file system operation is identified in response to a request to perform a file system operation. An update intent corresponding to the requested file system operation is inserted into a data structure associated with each identified file system object. Each file system object corresponding to the corresponding data structure is modified as specified by the update intent in that data structure. After modifying the file system object corresponding to the corresponding data structure, the update intent is removed from that data structure.
1. A method, comprising:
identifying a plurality of file system data structures to modify to perform a file system operation;
obtaining, by a computing node of a distributed file system, a corresponding node-level lock for each of the plurality of file system data structures;
obtaining, by the computing node of the distributed file system, a corresponding process-wide lock for each of the plurality of file system data structures; and
performing the file system operation, comprising:
determining that the file system operation corresponds to creating a file in a directory:
storing a first update intent in a first inode associated with the directory of a file system data structure of the plurality of file system data structures;
storing a second update intent in a second inode associated with the file after the storing of the first update intent; and
in the event that the second inode crashes after the first update intent is stored and before the second update intent is stored, determining that the file system operation has not been performed successfully.
2. The method of claim 1 , wherein the plurality of file system data structures are a plurality of inodes.
3. The method of claim 1 , further comprising receiving a request to perform the file system operation.
4. The method of claim 3 , wherein the request specifies a source directory, a destination directory, and a file handle.
5. The method of claim 1 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures includes requesting, from a distributed lock service, the corresponding node-level lock for each of the plurality of file system data structures.
6. The method of claim 5 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures further includes receiving from the distributed lock service, the corresponding node-level lock for each of the plurality of file system data structures.
7. The method of claim 5 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures further includes waiting for the corresponding node-level lock for each of the plurality of file system data structures in the event at least one of the corresponding node-level lock for each of the plurality of file system data structures is being used by another computing node of the distributed file system.
8. The method of claim 1 , wherein obtaining the corresponding process-wide lock for each of the plurality of file system data structures includes requesting from a process-wide lock service the corresponding process-wide lock for each of the plurality of file system data structures.
9. The method of claim 1 , wherein performing the file system operation includes generating a corresponding update intent for each of the plurality of file system data structures.
10. The method of claim 9 , wherein performing the file system operation further includes storing the corresponding generated update intent into each of the plurality of file system data structures.
11. The method of claim 10 , wherein the corresponding generated update intent are stored into each of the plurality of file system data structures in a particular order.
12. The method of claim 10 , wherein performing the file system operation further includes modifying each of the plurality of file system data structures based on the corresponding generated update intent.
13. The method of claim 12 , wherein performing the file system operation further includes removing the corresponding generated update intent after modifying a file system data structure.
14. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
identifying a plurality of file system data structures to modify to perform a file system operation;
obtaining, by a computing node of a distributed file system, a corresponding node-level lock for each of the plurality of file system data structures;
obtaining, by the computing node of the distributed file system, a corresponding process- wide lock for each of the plurality of file system data structures; and
performing the file system operation, comprising:
performing one or more of the following:
A) in the event that the file system operation corresponds to creating a first file in a first directory:
storing a first update intent in a first inode associated with the first directory of a file system data structure of the plurality of file system data structures;
storing a second update intent in a second inode associated with the first file after the storing of the first update intent; and
in the event that the second inode crashes after the first update intent is stored and before the second update intent is stored, determining that the file system operation has not been performed successfully; and/or
B) in the event that the file system operation corresponds to deleting a second file from a second directory:
storing a first update intent in a first inode associated with the second file;
storing a second update intent in a second inode associated with the second directory after the storing of the first update intent; and
in the event that the second inode crashes after the first update intent is stored and before the second update intent is stored, determining that the file system operation has not been performed successfully.
15. The computer program product of claim 14 , wherein the plurality of file system data structures are a plurality of inodes.
16. The computer program product of claim 14 , further comprising computer instructions for receiving a request to perform the file system operation.
17. The computer program product of claim 14 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures includes requesting, from a distributed lock service, the corresponding node-level lock for each of the plurality of file system data structures.
18. The computer program product of claim 17 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures further includes receiving from the distributed lock service, the corresponding node-level lock for each of the plurality of file system data structures.
19. The computer program product of claim 14 , wherein obtaining the corresponding node-level lock for each of the plurality of file system data structures further includes waiting for the corresponding node-level lock for each of the plurality of file system data structures in the event at least one of the corresponding node-level lock for each of the plurality of file system data structures is being used by another computing node of the distributed file system.
20. A method, comprising:
identifying a plurality of file system data structures to modify to perform a file system operation;
obtaining, by a computing node of a distributed file system, a corresponding node-level lock for each of the plurality of file system data structures;
obtaining, by the computing node of the distributed file system, a corresponding process-wide lock for each of the plurality of file system data structures; and
performing the file system operation, comprising:
determining that the file system operation corresponds to deleting a file from a directory:
storing a first update intent in a first inode associated with the file;
storing a second update intent in a second inode associated with the directory after the storing of the first update intent; and
in the event that the second inode crashes after the first update intent is stored and before the second update intent is stored, determining that the file system operation has not been performed successfully.