Performing fencing operations in multi-node distributed storage systems
Various systems and methods for managing node connectivity in distributed storage systems are disclosed. For example, one method involves detecting a communication fault between two nodes. A first node is configured to communicate input/output (I/O) requests to the second node as in-flight I/O requests. The second node is configured to communicate locally generated I/O requests and the received I/O requests to storage devices. Once the communication fault is detected, a fencing operation is performed. The fencing operation can include processing some of the received I/O requests by the second node, and rejecting any additional in-flight I/O requests received from the first node.
1. A computer-implemented method comprising:
detecting a communication fault between a first node and a second node, wherein
the first node and the second node each issue one or more input/output (I/O) requests to a storage device;
performing an arbitration process to determine to which of a winner node group or a loser node group the first node and the second node belong, respectively; and
if as a result of the arbitration process, either of the first node or the second is determined to be a node in the loser node group, fencing the node from the storage device by performing a fencing operation, wherein
the fencing operation prevents the node from communicating with the storage device.
2. The computer-implemented method of claim 1 , wherein
the arbitration process is performed by the first node and the second node.
3. The computer-implemented method of claim 2 , further comprising:
if the arbitration process is successful, performing the fencing operation; and
if the arbitration process is unsuccessful, retrying the arbitration process using one or more other nodes.
4. The computer-implemented method of claim 1 , wherein
the fencing operation further comprises instructing the storage device to not accept any communication from the node.
5. The computer-implemented method of claim 1 , wherein
the node uses a proxy node to issue the one or more I/O requests to the storage device.
6. The computer-implemented method of claim 5 , wherein
the proxy node is part of the winner node group, and
as a result of the fencing operation, the proxy node is prevented from processing the one or more I/O requests.
7. The computer-implemented method of claim 1 , further comprising:
de-registering the node with the storage device.
8. The computer-implemented method of claim 7 , wherein
the de-registering comprises deleting a key of the node from the storage device.
9. The computer-implemented method of claim 1 , further comprising:
detecting a network partition prior to performing the arbitration process.
10. A system comprising:
one or more processors; and
a control module configured to be executed using the one or more processors to detect a communication fault between a first node and a second node, wherein
the first node and the second node each issue one or more input/output (I/O) requests to a storage device;
perform an arbitration process to determine to which of a winner node group or a loser node group the first node and the second node belong, respectively; and
if as a result of the arbitration process, either of the first node or the second is determined to be a node in the loser node group, fence the node from the storage device by performing a fencing operation, wherein
the fencing operation prevents the node from communicating with the storage device.
11. The system of claim 10 , wherein
the arbitration process is performed by the first node and the second node, and
a network partition is detected prior to performing the arbitration process.
12. The system of claim 11 , further comprising:
if the arbitration process is successful, performing the fencing operation; and
if the arbitration process is unsuccessful, retrying the arbitration process using one or more other nodes.
13. The system of claim 10 , wherein
the fencing operation further comprises instructing the storage device to not accept any communication from the node.
14. The system of claim 10 , wherein
the node uses a proxy node to issue the one or more I/O requests to the storage device,
the proxy node is part of the winner node group, and
as a result of the fencing operation, the proxy node is prevented from processing the one or more I/O requests.
15. The system of claim 10 , further comprising:
de-registering the node with the storage device, wherein
the de-registering comprises deleting a key of the node from the storage device.
16. A non-transitory computer readable storage medium storing program instructions executable to:
detect a communication fault between a first node and a second node, wherein
the first node and the second node each issue one or more input/output (I/O) requests to a storage device;
perform an arbitration process to determine to which of a winner node group or a loser node group the first node and the second node belong, respectively; and
if as a result of the arbitration process, either of the first node or the second is determined to be a node in the loser node group, fence the node from the storage device by performing a fencing operation, wherein
the fencing operation prevents the node from communicating with the storage device.
17. The non-transitory computer readable storage medium of claim 16 , wherein
the arbitration process is performed by the first node and the second node, and
a network partition is detected prior to performing the arbitration process.
18. The non-transitory computer readable storage medium of claim 17 , further comprising:
if the arbitration process is successful, performing the fencing operation; and
if the arbitration process is unsuccessful, retrying the arbitration process using one or more other nodes.
19. The non-transitory computer readable storage medium of claim 16 , wherein
the node uses a proxy node to issue the one or more I/O requests to the storage device,
the proxy node is part of the winner node group, and
as a result of the fencing operation, the proxy node is prevented from processing the one or more I/O requests.
20. The non-transitory computer readable storage medium of claim 16 , further comprising: de-registering the node with the storage device, wherein
the de-registering comprises deleting a key of the node from the storage device; and
instructing the storage device to not accept any communication from the node.