RPC-less locking mechanism based on RDMA CAW for storage cluster with active-active architecture
Techniques for providing an RPC-less locking mechanism based on RDMA CAW for a storage cluster with an active-active architecture. The techniques include, in response to receipt at a storage node of a lock request for accessing an LBA of a storage object, synchronizing, between the storage node and a remote node, a state of a lock for the LBA. The synchronizing of the lock state includes locating an ALS object having a field containing the LBA, updating a field of the ALS object to include the storage node, and setting a field of the ALS object to a type of lock specified in the lock request. If the storage node locates the ALS object in an array maintained within visible memory on the remote node, then the storage node updates/sets corresponding fields of the ALS object by performing atomic CAW operations supported by an RDMA channel between the respective nodes.
1. A method of locking an entity of a storage object in a storage cluster with an active-active architecture, comprising:
maintaining a lock state object on a memory of a first storage node, the lock state object including a first field for a lock state of an entity of a storage object, and the memory being registered for remote direct memory access (RDMA);
in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, accessing, by the second storage node, the lock state object by performing an atomic RDMA compare-and-write (CAW) operation against a mutex value associated with a lock state object bucket that includes the lock state object; and
updating, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node.
2. The method of claim 1 wherein the performing of the second RDMA operation between the first storage node and the second storage node includes setting the first field of the lock state object to a value indicating that at least the second storage node is an owner of the lock.
3. The method of claim 2 wherein the accessing of the lock state object includes allocating the lock state object on the memory of the first storage node by performing a third RDMA operation between the first storage node and the second storage node.
4. The method of claim 3 wherein the lock state object includes a second field, and wherein the performing of the third RDMA operation between the first storage node and the second storage node includes setting the second field of the lock state object to a value of the entity of the storage object.
5. The method of claim 4 wherein the lock state object includes a third field, and wherein the accessing of the lock state object includes performing a fourth RDMA operation between the first storage node and the second storage node, the fourth RDMA operation including setting the third field of the lock state object to a value indicating a type of lock specified by the request.
6. The method of claim 2 wherein the lock state object includes a second field, and wherein the accessing of the lock state object includes locating the lock state object on the memory of the first storage node by performing a third RDMA operation between the first storage node and the second storage node.
7. The method of claim 6 wherein the performing of the third RDMA operation between the first storage node and the second storage node includes inspecting the first field of the lock state object to determine that at least the second storage node is the owner of the lock.
8. The method of claim 7 wherein the inspecting of the first field of the lock state object includes determining that the second storage node is the sole owner of the lock.
9. The method of claim 8 further comprising:
releasing, by the second storage node, the lock by performing a fourth RDMA operation between the first storage node and the second storage node.
10. The method of claim 9 wherein the lock state object includes a second field having a value of the entity of the storage object, and wherein the performing of the fourth RDMA operation between the first storage node and the second storage node includes replacing the value of the entity in the second field of the lock state object with a predetermined unspecified entity value.
11. The method of claim 10 wherein each of the first storage node and the second storage node is the owner of the lock, wherein the releasing of the lock includes performing a fifth RDMA operation between the first storage node and the second storage node, and wherein the performing of the fifth RDMA operation includes updating the first field of the lock state object to a value indicating that the first storage node is the sole owner of the lock.
12. The method of claim 11 wherein the lock includes a waiting queue and a reference to the lock state object, wherein the waiting queue is on the memory of the first storage node, wherein the waiting queue includes a process thread waiting for the lock, and wherein the method further comprises:
registering the lock state object with a poller on the first storage node; and
periodically polling, by the poller, the lock state object to detect the updating of the first field of the lock state object to the value indicating that the first storage node is the sole owner of the lock.
13. The method of claim 12 further comprising:
having detected the updating of the first field of the lock state object, providing a notification of the updating to the process thread in the waiting queue.
14. The method of claim 1 wherein the performing of the atomic RDMA CAW operation includes executing a compare operation and a write operation atomically, wherein the executing of the compare operation and the write operation atomically comprises:
in the compare operation, comparing the mutex value with a first predetermined value to produce a comparison result, the mutex value being contained in a mutex field; and
in the write operation, in response to the comparison result indicating that the mutex value is equal to the first predetermined value, writing, atomically with the compare operation, a second predetermined value to the mutex field.
15. A system for locking an entity of a storage object in a storage cluster with an active-active architecture, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
maintain a lock state object on a memory of a first storage node, the lock state object including a first field for a lock state of an entity of a storage object, and the memory being registered for remote direct memory access (RDMA);
in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, access, by the second storage node, the lock state object by performing a first RDMA operation between the first storage node and the second storage node; and
update, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node,
wherein the processing circuitry is further configured to execute the program instructions out of the memory to perform the first RDMA operation including performing an atomic RDMA compare-and-write (CAW) operation against a mutex value associated with a lock state object bucket that includes the lock state object.
16. The system of claim 15 wherein the processing circuitry is further configured to execute the program instructions out of the memory to perform the second RDMA operation between the first storage node and the second storage node including setting the first field of the lock state object to a value indicating that at least the second storage node is an owner of the lock.
17. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of locking an entity of a storage object in a storage cluster with an active-active architecture, the method comprising:
maintaining a lock state object on a memory of a first storage node, the lock state object including a first field for a lock state of an entity of a storage object, and the memory being registered for remote direct memory access (RDMA);
in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, accessing, by the second storage node, the lock state object by performing an atomic RDMA compare-and-write (CAW) operation against a mutex value associated with a lock state object bucket that includes the lock state object; and
updating, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node.
18. The computer program product of claim 17 wherein the performing of the second RDMA operation between the first storage node and the second storage node includes setting the first field of the lock state object to a value indicating that at least the second storage node is an owner of the lock.
19. The method of claim 14 wherein the executing of the write operation atomically with the compare operation further comprises:
in response to the comparison result indicating that the mutex value is not equal to the first predetermined value, avoiding writing the second predetermined value to the mutex field.
20. The method of claim 19 further comprising:
having avoided the writing of the second predetermined value to the mutex field, repeating the executing of the compare operation and the write operation atomically.