Distributed monitoring in clusters with self-healing

1. A computer-implemented method, comprising:

determining, by a first instance of a monitoring agent of a first node of a cluster, at least one self-monitoring check to perform for the first node, wherein the first node is among multiple, other nodes included in the cluster, including a second node;

performing, by the first instance of the monitoring agent, the at least one self-monitoring check;

in response to receiving a successful status for the at least one self-monitoring check, updating a first registry in the first node with the successful status, the first registry including node statuses for each node in the cluster;

in response to receiving an unsuccessful status for the at least one self-monitoring check, performing, by the first instance of the monitoring agent, at least one first corrective action on the first node and updating the first registry in the first node with a result of the at least one corrective action;

broadcasting the first registry in the first node to each of the other nodes in the cluster as an updated registry;

receiving, by the second node, the updated registry;

determining, by a second instance of the monitoring agent running in the second node, an error condition for the first node in the updated registry; and

performing, by the second instance of the monitoring agent, at least one second corrective action on the first node.

2. The computer-implemented method of claim 1 , wherein the second node is designated as a friend node to the first node.

3. The computer-implemented method of claim 1 , further comprising:

determining, by the second instance of the monitoring agent, that the at least one second corrective action was unsuccessful;

updating, in a second registry in the second node, node status for the first node to indicate the error condition in the first node; and

broadcasting the second registry to each of the other nodes in the cluster other than the second node.

4. The computer-implemented method of claim 3 , further comprising sending, by the second instance of the monitoring agent, an alert message to an administrator regarding the error condition in the first node.

5. The computer-implemented method of claim 3 , wherein the second registry indicates that the first node is no longer included in the cluster due to the error condition in the first node.

6. The computer-implemented method of claim 5 , further comprising assigning the second node as a friend to a third node in response to determining that the first node is no longer included in the cluster.

7. The computer-implemented method of claim 5 , further comprising:

receiving, at the second node, a request to re-add the first node to the cluster;

updating the second registry to indicate that the first node has been re-added to the cluster;

adjusting friendship assignments in the second registry so that each node, including the first node, is assigned as a friend node to another node; and

broadcasting the second registry to each of the other nodes in the cluster other than the second node.

8. The computer-implemented method of claim 7 , further comprising:

determining that an amount of time since the first node has sent status information is more than a threshold; and

performing, by the second instance of the monitoring agent, the at least one second corrective action on the first node in response to determining that the amount of time since the first node has sent status information is more than the threshold.

9. A system comprising:

one or more computers; and

a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

determining, by a first instance of a monitoring agent of a first node of a cluster, at least one self-monitoring check to perform for the first node, wherein the first node is among multiple, other nodes included in the cluster, including a second node;

performing, by the first instance of the monitoring agent, the at least one self-monitoring check;

in response to receiving a successful status for the at least one self-monitoring check, updating a first registry in the first node with the successful status, the first registry including node statuses for each node in the cluster;

in response to receiving an unsuccessful status for the at least one self-monitoring check, performing, by the first instance of the monitoring agent, at least one first corrective action on the first node and updating the first registry in the first node with a result of the at least one corrective action;

broadcasting the first registry in the first node to each of the other nodes in the cluster as an updated registry;

receiving, by the second node, the updated registry;

determining, by a second instance of the monitoring agent running in the second node, an error condition for the first node in the updated registry; and

performing, by the second instance of the monitoring agent, at least one second corrective action on the first node.

10. The system of claim 9 , wherein the second node is designated as a friend node to the first node.

11. The system of claim 9 , wherein the operations further comprise:

determining, by the second instance of the monitoring agent, that the at least one second corrective action was unsuccessful;

updating, in a second registry in the second node, node status for the first node to indicate the error condition in the first node; and

broadcasting the second registry to each of the other nodes in the cluster other than the second node.

12. The system of claim 11 , wherein the operations further comprise sending, by the second instance of the monitoring agent, an alert message to an administrator regarding the error condition in the first node.

13. The system of claim 11 , wherein the second registry indicates that the first node is no longer included in the cluster due to the error condition in the first node.

14. The system of claim 13 , wherein the operations further comprise assigning the second node as a friend to a third node in response to determining that the first node is no longer included in the cluster.

15. A computer program product encoded on a non-transitory storage medium, the product comprising non-transitory, computer readable instructions for causing one or more processors to perform operations comprising:

determining, by a first instance of a monitoring agent of a first node of a cluster, at least one self-monitoring check to perform for the first node, wherein the first node is among multiple, other nodes included in the cluster, including a second node;

performing, by the first instance of the monitoring agent, the at least one self-monitoring check;

in response to receiving a successful status for the at least one self-monitoring check, updating a first registry in the first node with the successful status, the first registry including node statuses for each node in the cluster;

in response to receiving an unsuccessful status for the at least one self-monitoring check, performing, by the first instance of the monitoring agent, at least one first corrective action on the first node and updating the first registry in the first node with a result of the at least one corrective action;

broadcasting the first registry in the first node to each of the other nodes in the cluster as an updated registry;

receiving, by the second node, the updated registry;

determining, by a second instance of the monitoring agent running in the second node, an error condition for the first node in the updated registry; and

performing, by the second instance of the monitoring agent, at least one second corrective action on the first node.

16. The computer program product of claim 15 , wherein the second node is designated as a friend node to the first node.

17. The computer program product of claim 15 , wherein the operations further comprise:

determining, by the second instance of the monitoring agent, that the at least one second corrective action was unsuccessful;

updating, in a second registry in the second node, node status for the first node to indicate the error condition in the first node; and

broadcasting the second registry to each of the other nodes in the cluster other than the second node.

18. The computer program product of claim 17 , wherein the operations further comprise sending, by the second instance of the monitoring agent, an alert message to an administrator regarding the error condition in the first node.

19. The computer program product of claim 17 , wherein the second registry indicates that the first node is no longer included in the cluster due to the error condition in the first node.

20. The computer program product of claim 19 , wherein the operations further comprise assigning the second node as a friend to a third node in response to determining that the first node is no longer included in the cluster.