Dynamically changing members of a consensus group in a distributed self-healing coordination service

1. A method for managing a consensus group of nodes in a distributed computing cluster, the method comprising:

reaching a first consensus, based on at least two members of the consensus group of nodes in the distributed computing cluster, to elect an instance of an authority module executing on a first node of the consensus group of nodes, to serve as a master instance of the authority module for the consensus group of nodes, wherein each member of the consensus group of nodes executes a respective instance of the authority module, and the master instance of the authority module is configured to assign locks to processes executing in the distributed computing cluster;

reaching a second consensus, based on at least two members of the consensus group of nodes, that the instance of the authority module executing on the first node of the consensus group of nodes, has failed;

reaching a third consensus, based on at least two not failed members of the consensus group of nodes, to elect the instance of the authority module executing on a second node of the consensus group of nodes to serve as the master instance of the authority module for the consensus group;

receiving, by the master authority module on the second node, a communication from a new node in the distributed computing cluster, wherein the new node comprises a node in the distributed computing cluster that was not a member of the consensus group at a time the at least two members of the consensus group of nodes reached the second consensus that the instance of the authority module executing on the first node failed;

adding, by the master authority module on the second node, the new node to the consensus group; and

sending to the new node, an identification of a set of nodes in the distributed computing cluster eligible to join the consensus group.

2. The method of claim 1 , wherein the communication from the new node in the computing cluster advertises the new node in the computing cluster.

3. The method of claim 1 , further comprising sending to the new node, a state of a set of locks issued to nodes in the distributed computing cluster.

4. The method of claim 1 , further comprising sending, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group, wherein the locks are required to access data in a storage device and the distributed computing cluster comprises a secondary storage environment.

5. The method of claim 1 , further comprising:

adding, by the master authority module on second node, the new node to a set of available nodes in the distributed computing cluster; and

sharing the set of available nodes with members of the consensus group.

6. The method of claim 1 , further comprising removing, by the second node, the first node from the consensus group.

7. A system for managing a consensus group of node in a distributed computing cluster, comprising:

one or more processors configured to:

reach a first consensus, based on at least two members of the consensus group of nodes in the distributed computing cluster, to elect an instance of an authority module executing on a first node of the consensus group of nodes, to serve as a master instance of the authority module for the consensus group of nodes, wherein each member of the consensus group of nodes executes a respective instance of the authority module, and the master instance of the authority module is configured to assign locks to processes executing in the distributed computing cluster;

reach a second consensus, based on at least two members of the consensus group of nodes, that the instance of the authority module executing on the first node of the consensus group of nodes, has failed;

reach a third consensus, based on at least two not failed members of the consensus group of nodes, to elect the instance of the authority module executing on a second node of the consensus group of nodes to serve as the master instance of the authority module for the consensus group;

receive, by the master authority module on the second node, a communication from a new node in the distributed computing cluster, wherein the new node comprises a node in the distributed computing cluster that was not a member of the consensus group at a time the at least two members of the consensus group of nodes reached the second consensus that the instance of the authority module executing on the first node failed; and

add, by the master authority module on the second node, the new node to the consensus group;

send to the new node, an identification of a set of nodes in the distributed computing cluster eligible to join the consensus group; and

one or more memory coupled to the one or more processors and configured to provide the one or more processors with instructions.

8. The system of claim 7 , wherein the communication from the new node in the distributed computing cluster advertises the new node in the distributed computing cluster.

9. The system of claim 7 , wherein the one or more processors are further configured to send to the new node, a state of a set of locks issued to nodes in the distributed computing cluster.

10. The system of claim 7 , wherein the one or more processors are further configured to send, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group, wherein the locks are required to access data in a storage device and the distributed computing cluster comprises a secondary storage environment.

11. The system of claim 8 , wherein the one or more processors are further configured to:

add, by the master authority module on second node, the new node to a set of available nodes in the distributed computing cluster; and

share the set of available nodes with members of the consensus group.

12. The system of claim 7 , wherein the one or more processors are further configured to remove the first node from the consensus group.

13. A computer program product for managing a consensus group of nodes in a distributed computing cluster, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

reaching a first consensus, based on at least two members of the consensus group of nodes in the distributed computing cluster, to elect an instance of an authority module executing on a first node of the consensus group of nodes, to serve as a master instance of the authority module for the consensus group of nodes, wherein each member of the consensus group of nodes executes a respective instance of the authority module, and the master instance of the authority module is configured to assign locks to processes executing in the distributed computing cluster;

reaching a second consensus, based on at least two members of the consensus group of nodes, that the instance of the authority module executing on the first node of the consensus group of nodes, has failed;

reaching a third consensus, based on at least two not failed members of the consensus group of nodes, to elect the instance of the authority module executing on a second node of the consensus group of nodes to serve as the master instance of the authority module for the consensus group;

receiving, by the master authority module on the second node, a communication from a new node in the distributed computing cluster, wherein the new node comprises a node in the distributed computing cluster that was not a member of the consensus group at a time the at least two members of the consensus group of nodes reached the second consensus that the instance of the authority module executing on the first node failed;

adding, by the master authority module on the second node, the new node to the consensus group; and

sending to the new node, an identification of a set of nodes in the distributed computing cluster eligible to join the consensus group.

14. The computer program product of claim 13 , wherein the communication from the new node in the distributed computing cluster advertises the new node in the distributed computing cluster.

15. The computer program product of claim 13 , further comprising computer instructions for sending to the new node, a state of a set of locks issued to nodes in the distributed computing cluster.

16. The computer program product of claim 13 , further comprising computer instructions for sending, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group, wherein the locks are required to access data in a storage device and the distributed computing cluster comprises a secondary storage environment.

17. The computer program product of claim 13 , further comprising computer instructions for:

adding, by the master authority module on second node, the new node to a set of available nodes in the distributed computing cluster; and

sharing the set of available nodes with members of the consensus group.

18. The computer program product of claim 13 , further comprising computer instructions for removing, by the second node, the first node from the consensus group.