Determining the health of other nodes in a same cluster based on physical link information
The embodiments disclosed herein relate to using SNMP functionality for fast and accurate detection of node failure in a computer cluster. When cluster nodes are connected via a switch that provides SNMP capability, a trap mechanism can be used to immediately and proactively notify cluster nodes that a particular cluster node is disconnected from the network, potentially even before the remaining cluster nodes have noticed a lack of response. In addition, when a cluster node notices a lack of response from a particular cluster node, the switch can be queried for link status and traffic information that can be used for fast determination of node status.
1. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising:
determining, by a first node, that a second node is in a same node cluster as the first node;
based at least on the second node being in the same node cluster as the first node:
subscribing, by the first node, to events indicating failure of a direct physical link between the second node and a network device, wherein subscribing to events indicating failure of the direct physical link between the second node and the network device comprises sending a physical link identifier of a direct physical link between the second node and the network device;
receiving, by the first node from the network device, an event indicating failure of the direct physical link between the second node and the network device; and
initiating, by the first node, a node eviction process to remove the second node from the node cluster.
2. The medium of claim 1 , wherein:
the network device is an SNMP-enabled switch; and
subscribing by the first node comprises transmitting an SNMP request to the SNMP-enabled switch.
3. The medium of claim 1 , wherein:
subscribing to events indicating failure of a direct physical link between the second node and a network device comprises sending an IP address or a physical network address of the second node to the network device.
4. The medium of claim 1 , wherein the operations further comprise:
prior to subscribing to events indicating failure of the direct physical link, sending an identifier of the second node with a query to the network device, and receiving in response to the query a physical link identifier of the direct physical link between the second node and the network device.
5. The medium of claim 1 , wherein a node cluster comprises a set of communicatively coupled nodes that conduct tasks based on a consensus among the set of nodes, wherein each node of the set of nodes is a computer.
6. The medium of claim 1 , wherein the event indicating failure of the direct physical link between the second node received by the first node from the network device is an SNMP trap notification.
7. A method comprising:
determining, by a first node, that a second node is in a same node cluster as the first node;
based at least on the second node being in the same node cluster as the first node:
subscribing, by the first node, to events indicating failure of a direct physical link between the second node and a network device, wherein subscribing to events indicating failure of the direct physical link between the second node and the network device comprises sending a physical link identifier of a direct physical link between the second node and the network device;
receiving, by the first node from the network device, an event indicating failure of the direct physical link between the second node and the network device; and
initiating, by the first node, a node eviction process to remove the second node from the node cluster.
8. The method of claim 7 , wherein:
the network device is an SNMP-enabled switch; and
subscribing by the first node comprises transmitting an SNMP request to the SNMP-enabled switch.
9. The method of claim 7 , wherein:
subscribing to events indicating failure of a direct physical link between the second node and a network device comprises sending an IP address or a physical network address of the second node to the network device.
10. The method of claim 7 , further comprising:
prior to subscribing to events indicating failure of the direct physical link, sending an identifier of the second node with a query to the network device, and receiving in response to the query a physical link identifier of the direct physical link between the second node and the network device.
11. The method of claim 7 , wherein the first node receives a trap notification from the network device responsive to the network device experiencing a network failure.
12. The method of claim 7 , wherein a node cluster comprises a set of communicatively coupled nodes that conduct tasks based on a consensus among the set of nodes, wherein each node of the set of nodes is a computer.
13. The method of claim 7 , wherein the event indicating failure of the direct physical link between the second node received by the first node from the network device is an SNMP trap notification.
14. A system comprising:
at least one hardware device including a processor; and
the system configured to perform operations comprising:
determining, by a first node, that a second node is in a same node cluster as the first node;
based at least on the second node being in the same node cluster as the first node:
subscribing, by the first node, to events indicating failure of a direct physical link between the second node and a network device, wherein subscribing to events indicating failure of the direct physical link between the second node and the network device comprises sending a physical link identifier of a direct physical link between the second node and the network device;
receiving, by the first node from the network device, an event indicating failure of the direct physical link between the second node and the network device; and
initiating, by the first node, a node eviction process to remove the second node from the node cluster.
15. The system of claim 14 , wherein:
the network device is an SNMP-enabled switch; and
subscribing by the first node comprises transmitting an SNMP request to the SNMP-enabled switch.
16. The system of claim 14 , wherein:
subscribing to events indicating failure of a direct physical link between the second node and a network device comprises sending an IP address or a physical network address of the second node to the network device.
17. The system of claim 14 , wherein the operations further comprise:
prior to subscribing to events indicating failure of the direct physical link, sending an identifier of the second node with a query to the network device, and receiving in response to the query a physical link identifier of the direct physical link between the second node and the network device.
18. The system of claim 14 , wherein a node cluster comprises a set of communicatively coupled nodes that conduct tasks based on a consensus among the set of nodes, wherein each node of the set of nodes is a computer.
19. The system of claim 14 , wherein the event indicating failure of the direct physical link between the second node received by the first node from the network device is an SNMP trap notification.
20. The system of claim 14 , wherein the first node receives a trap notification from the network device responsive to the network device experiencing a network failure.