Safely recovering workloads within a finite timeframe from unhealthy cluster nodes

1. A method comprising:

monitoring, by a polling process of a first node among a plurality of nodes, for a polling signal response from a control plane at periodic intervals, wherein the polling signal response indicates a health status of the first node as healthy and wherein the control plane is hosted on a first computing device and each of the plurality of nodes is hosted on a respective second computing device;

in response to not receiving the polling signal response for a threshold number of consecutive intervals:

identifying, by the first node, a subset of peer nodes among the plurality of nodes;

instructing each peer node in the subset to inquire as to the health status of the first node from the control plane and forward a response received from the control plane to the first node; and

performing, by the first node, an action to reset itself or continue operating based on a response received by each of the subset of peer nodes after inquiring as to the health status of the first node from the control plane.

2. The method of claim 1 , wherein a heartbeat signal is received from the polling process at periodic intervals by a watchdog device of the first node, the watchdog device to reset the first node based on a failure to receive a threshold number of consecutive heartbeat signals.

3. The method of claim 2 , wherein the control plane includes a cluster state store that includes, for each of the plurality of nodes, a record indicating a health status of the node, the health status of each node based, at least in part, on one or more of: whether the control plane is in communication with the node, a memory capacity of the node, and a processing capacity of the node.

4. The method of claim 3 , wherein a response received from a peer node after inquiring as to the health status of the first node indicates that the first node is healthy, the first node is unhealthy, the peer node is unable to communicate with the cluster state store on the control plane, or no response was received from the peer node.

5. The method of claim 4 , wherein performing, by the first node, an action to reset itself or continue operating comprises:

transmitting a heartbeat signal to the watchdog device if the response of any of the peer nodes in the subset indicate that the first node is healthy; and

using the watchdog device to reset itself if the response of at least one peer node in the subset indicates that the first node is unhealthy, and the responses from the remaining peer nodes in the subset indicate that the peer node is unable to communicate with the control plane, or no response was received from the peer node.

6. The method of claim 4 , wherein performing, by the first node, an action to reset itself or continue operating comprises:

transmitting a heartbeat signal to the watchdog device if more than a threshold number of the peer nodes in the subset report that they are unable to communicate with the control plane; and

using the watchdog device to reset the first node if there is no response from a majority of the peer nodes in the subset.

7. The method of claim 3 , wherein for each node, the control plane periodically updates a record indicating a health status of the node based at least in part on a polling signal received from the polling process of the first node, wherein the polling signal indicates whether the control plane is in communication with the node, a memory capacity of the node, and a processing capacity of the node.

8. A system comprising:

a first computing device hosting a control plane; and

a plurality of second computing devices, each of the plurality of second computing devices hosting a respective node of a plurality of nodes, wherein a first node of the plurality of nodes is to:

monitor, by a polling process of the first node, for a polling signal response from the control plane at periodic intervals, wherein the polling signal response indicates a health status of the first node as healthy;

in response to not receiving the polling signal response for a threshold number of consecutive intervals:

identify a subset of peer nodes among the plurality of nodes;

instruct each peer node in the subset to inquire as to the health status of the first node from the control plane and forward a response received from the control plane to the first node; and

perform an action to reset itself or continue operating based on a response received by each of the subset of peer nodes after inquiring as to the health status of the first node from the control plane.

9. The system of claim 8 , wherein the first node comprises a watchdog device that receives a heartbeat signal from the polling process at periodic intervals, the watchdog device to reset the first node based on a failure to receive a threshold number of consecutive heartbeat signals.

10. The system of claim 9 , wherein the control plane includes a cluster state store that includes, for each of the plurality of nodes, a record indicating a health status of the node, the health status of each node based, at least in part, on one or more of: whether the control plane is in communication with the node, a memory capacity of the node, and a processing capacity of the node.

11. The system of claim 10 , wherein a response received from a peer node after inquiring as to the health status of the first node indicates that the first node is healthy, the first node is unhealthy, the peer node is unable to communicate with the cluster state store on the control plane, or no response was received from the peer node.

12. The system of claim 11 , wherein to perform an action to reset itself or continue operating, the first node is to:

transmit a heartbeat signal to the watchdog device if the response of any of the peer nodes in the subset indicate that the first node is healthy; and

use the watchdog device to reset itself if the response of at least one peer node in the subset indicates that the first node is unhealthy, and the responses from the remaining peer nodes in the subset indicate that the peer node is unable to communicate with the control plane, or no response was received from the peer node.

13. The system of claim 11 , wherein to perform an action to reset itself or continue operating, the first node is to:

transmit a heartbeat signal to the watchdog device if more than a threshold number of the peer nodes in the subset report that they are unable to communicate with the control plane; and

use the watchdog device to reset the first node if there is no response from a majority of the peer nodes in the subset.

14. The system of claim 10 , wherein for each node, the control plane periodically updates a record indicating a health status of the node based at least in part on a polling signal received from the polling process of the first node, wherein the polling signal indicates whether the control plane is in communication with the node, a memory capacity of the node, and a processing capacity of the node.

15. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device of a computing device on which a first node of a plurality of nodes is hosted, cause the processing device to:

monitor, by a polling process of the first node, for a polling signal response from a control plane at periodic intervals, wherein the polling signal response indicates a health status of the first node as healthy and wherein the control plane is hosted on a first computing device and each of the plurality of nodes is hosted on a respective second computing device;

in response to not receiving the polling signal response for a threshold number of consecutive intervals:

identify a subset of peer nodes among the plurality of nodes;

instruct each peer node in the subset to inquire as to the health status of the first node from the control plane and forward a response received from the control plane to the first node; and

perform, by the first node, an action to reset itself or continue operating based on a response received by each of the subset of peer nodes after inquiring as to the health status of the first node from the control plane.

16. The non-transitory computer-readable medium of claim 15 , wherein the processing device receives a heartbeat signal from the polling process at periodic intervals using a watchdog device of the first node, the watchdog device to reset the first node based on a failure to receive a threshold number of consecutive heartbeat signals.

17. The non-transitory computer-readable medium of claim 16 , wherein the control plane includes a cluster state store that includes, for each of the plurality of nodes, a record indicating a health status of the node, the health status of each node based, at least in part, on one or more of: whether the control plane is in communication with the node, a memory capacity of the node, and a processing capacity of the node.

18. The non-transitory computer-readable medium of claim 17 , wherein a response received from a peer node after inquiring as to the health status of the first node indicates that the first node is healthy, the first node is unhealthy, the peer node is unable to communicate with the cluster state store on the control plane, or no response was received from the peer node.

19. The non-transitory computer-readable medium of claim 18 , wherein to perform, by the first node, an action to reset itself or continue operating, the processing device is to:

transmit a heartbeat signal to the watchdog device if the response of any of the peer nodes in the subset indicate that the first node is healthy; and

use the watchdog device to reset itself if the response of at least one peer node in the subset indicates that the first node is unhealthy, and the responses from the remaining peer nodes in the subset indicate that the peer node is unable to communicate with the control plane, or no response was received from the peer node.

20. The non-transitory computer-readable medium of claim 18 , wherein to perform, by the first node, an action to reset itself or continue operating, the processing device is to:

transmit a heartbeat signal to the watchdog device if more than a threshold number of the peer nodes in the subset report that they are unable to communicate with the control plane; and

use the watchdog device to reset the first node if there is no response from a majority of the peer nodes in the subset.