File system operation cancellation
Cancelling file system operations at a file system client of a distributed file system is disclosed. In various embodiments, an indication is received to send locally-stored file system operations to a remote file system metadata server. A dependency graph or other data structure reflecting dependencies between the locally-stored file system operations is used to identify a set of locally-stored file system operations the members of which are fully dominated by a create-delete pair of operations comprising the set. The locally-stored file system operations comprising the set are cancelled, without being sent to the remote file system metadata server.
1. A method of processing file system operations, comprising:
receiving, by a client device connected to a distributed file system including a remote file system metadata server, an indication of a lease break with respect to a lease held by the client device, wherein the lease is associated with metadata stored by the remote file system metadata server; and
in response to receiving the indication of the lease break,
determining, by the client device, a first subset of locally-stored file system operations that collectively correspond to a non-event, wherein:
the locally-stored file system operations are stored in a buffer at the client device;
the locally-stored file system operations correspond to one or more file system calls made by an application running on the client device that are intercepted and redirected to the buffer at the client device;
the locally-stored file system operations are associated with a portion of file system metadata corresponding to the lease; and
the first subset is determined based at least in part on a dependency graph or other data structure reflecting dependencies of the locally-stored file system operations;
cancelling, by the client device, the locally-stored file system operations comprised in the first subset;
determining, by the client device, a second subset of the locally-stored file system operations to communicate to the remote file system metadata server in a single package, wherein the second subset is determined based at least in part on the dependency graph or other data structure reflecting dependencies of the locally-stored file system operations; and
communicating, by the client device, the second subset to the remote file system metadata server, wherein communicating an operation of the second subset includes ensuring that the operation of the second subset is not sent to the remote file system metadata server unless all its dependent operations have already been communicated to the remote file system metadata server or are being communicated in connection with the operation, and
wherein cancelling the locally-stored file system operations comprised in the first subset results in the locally-stored file system operations comprised in the first subset not being sent to the remote file system metadata server.
2. The method of claim 1 , wherein the locally-stored file system operations comprise file system operations affecting file system metadata.
3. The method of claim 1 , wherein the locally-stored file system operations comprise file system operations that have been buffered locally to permit asynchronous transmission to the remote file system metadata server.
4. The method of claim 3 , wherein the portion of file system metadata corresponding to the lease corresponds to a portion of file system metadata with respect to which a metadata write lease is held.
5. The method of claim 3 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the buffer without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
6. The method of claim 3 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the dependency graph without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
7. A system, comprising:
a memory or other storage device configured to store file system operations locally at the system prior to such file system operations being sent to a remote file system metadata server; and
one or more processors of a client device connected to a distributed file system including a remote file system metadata server, the one or more processors coupled to the memory or other storage device and configured to:
receive an indication of a lease break with respect to a lease held by the client device, wherein the lease is associated with metadata stored by the remote file system metadata server; and
in response to receiving the indication of the lease break,
determine a first subset of locally-stored file system operations that collectively correspond to a non-event, wherein:
the locally-stored file system operations are stored in a buffer at the client device;
the locally-stored file system operations correspond to one or more file system calls made by an application running on the client device that are intercepted and redirected to the buffer at the client device;
the locally-stored file system operations are associated with a portion of file system metadata corresponding to the lease; and
the first subset is determined based at least in part on a dependency graph or other data structure reflecting dependencies of the locally-stored file system operations;
cancel the locally-stored file system operations comprised in the first subset;
determine a second subset of the locally-stored file system operations to communicate to the remote file system metadata server in a single package, wherein the second subset is determined based at least in part on the dependency graph or other data structure reflecting dependencies of the locally-stored file system operations; and
communicate, to the remote file system metadata server, the second subset, wherein communicating an operation of the second subset includes ensuring that the operation of the second subset is not sent to the remote file system metadata server unless all its dependent operations have already been communicated to the remote file system metadata server or are being communicated in connection with the operation, and
wherein cancelling the locally-stored file system operations comprised in the first subset results in the locally-stored file system operations comprised in the first subset not being sent to the remote file system metadata server.
8. The system of claim 7 , wherein the locally-stored file system operations comprise file system operations affecting file system metadata.
9. The system of claim 7 , wherein the locally-stored file system operations comprise file system operations that have been buffered locally to permit asynchronous transmission to the remote file system metadata server.
10. The system of claim 9 , wherein the portion of file system metadata corresponding to the lease corresponds to a portion of file system metadata with respect to which a metadata write lease is held.
11. The system of claim 9 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the buffer without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
12. The system of claim 9 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the dependency graph without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
13. A computer program product to process file system operations, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, by a client device connected to a distributed file system including a remote file system metadata server, an indication of a lease break with respect to a lease held by the client device, wherein the lease is associated with metadata stored by the remote file system metadata server; and
in response to receiving the indication of the lease break,
determining, by the client device, a first subset of locally-stored file system operations that collectively correspond to a non-event, wherein:
the locally-stored file system operations are stored in a buffer at the client device;
the locally-stored file system operations correspond to one or more file system calls made by an application running on the client device that are intercepted and redirected to the buffer at the client device;
the locally-stored file system operations are associated with a portion of file system metadata corresponding to the lease; and
the first subset is determined based at least in part on a dependency graph or other data structure reflecting dependencies of the locally-stored file system operations;
cancelling, by the client device, the locally-stored file system operations comprised in the first subset;
determining, by the client device, a second subset of the locally-stored file system operations to communicate to the remote file system metadata server in a single package, wherein the second subset is determined based at least in part on the dependency graph or other data structure reflecting dependencies of the locally-stored file system operations; and
communicating, by the client device, the second subset to the remote file system metadata server, wherein communicating an operation of the second subset includes ensuring that the operation of the second subset is not sent to the remote file system metadata server unless all its dependent operations have already been communicated to the remote file system metadata server or are being communicated in connection with the operation, and
wherein cancelling the locally-stored file system operations comprised in the first subset results in the locally-stored file system operations comprised in the first subset not being sent to the remote file system metadata server.
14. The computer program product of claim 13 , wherein the locally-stored file system operations comprise file system operations that have been buffered locally to permit asynchronous transmission to the remote file system metadata server.
15. The computer program product of claim 14 , wherein the portion of file system metadata corresponding to the lease corresponds to a portion of file system metadata with respect to which a metadata write lease is held.
16. The computer program product of claim 15 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the buffer without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
17. The computer program product of claim 15 , wherein cancelling the locally-stored file system operations comprised in the first subset includes removing the locally-stored file system operations comprised in the first subset from the dependency graph without sending the locally-stored file system operations comprised in the first subset to the remote file system metadata server.
18. The method of claim 1 , wherein members of the first subset of the locally-stored file system operations are bounded by a create-delete pair of operations comprised in the first subset.
19. The method of claim 1 , wherein the determining the second subset of the locally-stored file system operations to communicate to the remote file system metadata server in the single package comprises determining linear chains of operations.
20. The method of claim 19 , wherein a set of operations corresponding to a linear chain of operations is communicated to the remote file system metadata server in a single communication.
21. The method of claim 20 , wherein the linear chain of operations comprise a sequence of operations each of which has at most a single dependency on an immediately prior operation in the linear chain.
22. The method of claim 21 , wherein the locally-stored file system operations were locally-stored in the buffer of the client device in response to corresponding operations being intercepted by a process running in a kernel of an operating system of the client device, and redirecting, by the process running in the kernel, the corresponding operations to a file system client that stores the locally-stored file system operations in the buffer.
23. The method of claim 1 , wherein the second subset is communicated in connection with one or more uniform resource identifiers (URIs) corresponding to one or more file system objects affected by the second subset, and the remote file system metadata server stores payload data associated with the second subset in a location associated with the one or more URIs in response to determining that the one or more URIs are valid.
24. The method of claim 21 , wherein the second subset is communicated to the remote file system metadata server in the single communication.
25. The method of claim 22 , wherein the corresponding operations intercepted by the process running the kernel are communicated by the client device to the remote file system metadata server so as to be asynchronous with respect to when corresponding file system calls are intercepted by the process.
26. The method of claim 1 , wherein a plurality of operations for changing an end of file attribute of a file that are comprised in a linear chain of the dependency graph from which the second subset is determined are merged into a single operation to set the attribute to an ending value indicated by a last operation in the linear chain.