System and method to create a highly available quorum for clustered solutions
A node for a distributed system includes computing resources and a node manager. The node manager identifies a failure of a second node of the distributed system; in response to identifying the failure: makes a first determination that the second node is a primary node of the distributed system; after making the first determination: identifies a new primary node by accessing a distributed system configuration stored in a primary share out-of-band management entity; and provides services of the distributed system to a client using the computing resources after identifying the new primary node.
1. A node for a distributed system comprising a plurality of nodes, wherein each node of the distributed system hosts a corresponding out-of-band management entity comprising separate computing resources from computing resources of the plurality of nodes of the distributed system, the computing resources of each of the out-of-band management entities store a copy of a distributed system configuration for the distributed system, comprising:
computing resources of the node used to provide the functionality of the node; and
a node manager programmed to:
identify a failure of a second node of the plurality of nodes of the distributed system;
in response to identifying the failure:
make a first determination that the second node is a primary node of the distributed system;
after making the first determination:
identify a new primary node of the plurality of nodes by accessing a copy of the distributed system configuration stored in a one of the plurality of out-of-band management entities; and
provide services of the distributed system to a client using the computing resources of the node after identifying the new primary node.
2. The node of claim 1 , wherein the node manager is further programmed to:
before identifying the new primary node:
make a second determination that the one of the plurality of the out-of-band management entities is reachable via an in-band channel routed through a network and an out out-of-band channel; and
in response to the second determination:
access the distributed system configuration via the in-band channel.
3. The node of claim 2 , wherein the one of the plurality of out-of-band management entities is not hosted by the node.
4. The node of claim 2 , wherein the one of the plurality of out-of-band management entities is hosted by the node.
5. The node of claim 1 , wherein the node manager is further programmed to:
before identifying the new primary node:
make a second determination that the one of the plurality of out-of-band management entities is unreachable via any in-band channels that is routed through a network and an out out-of-band channel; and
in response to the second determination:
access the distributed system configuration via an always-on in-band channel that directly connects the computing resources of the one of the plurality of the out-of-band management entities and the computing resources of the node.
6. The node of claim 5 , wherein the copy of the distributed system configuration is stored in a second out-of-band management entity.
7. The node of claim 6 , wherein the second out-of-band management entity is hosted by the node.
8. The node of claim 7 , wherein the always-on in-band channel is supported by an out-of-band management agent executing on the computing resources.
9. The node of claim 8 , wherein the always-on in-band channel utilizes a system bus that operably connects the computing resources and the second out-of-band management entity.
10. The node of claim 1 , wherein the one of the plurality of out-of-band management entities is programmed to:
identify changes to a configuration of the distributed system;
modify the distributed system configuration based on the identified changes to obtain an updated distributed system configuration; and
replicate the updated distributed system configuration in a second out-of-band management entity of the plurality of out of band management entities.
11. The node of claim 10 , wherein both of the one of the plurality out-of-band management entities and the second out-of-band management entity are associated with a floating internet protocol address.
12. The node of claim 10 , wherein the updated distributed system configuration is replicated to the second out-of-band management entity via an out-of-band channel that is routed through a network.
13. The node of claim 12 , wherein the one of the plurality of out-of-band management entities is programmed to:
provide the updated distributed system configuration to all nodes of the plurality of nodes of the distributed system via an in-band channel associated with the computing resources through a network; and
provide the updated distributed system configuration to only the node of the plurality of nodes via an always-on in-band channel associated with the one of the plurality of out-of-band management entities.
14. The node of claim 1 , wherein the node manager is further programmed to:
before providing the services of the distributed system:
make a second determination that the node is the new primary node; and
in response to the second determination:
modify a state of the node to active.
15. The node of claim 14 , wherein the services provided to the distributed system are based on the state of the node.
16. A method for managing a node of a distributed system comprising a plurality of nodes, wherein each node of the distributed system hosts a corresponding out-of-band management entity comprising separate computing resources from computing resources of the distributed system, wherein the computing resources of each of the out-of-band management entities store a copy of a distributed system configuration for the distributed system, comprising:
identifying a failure of a second node of the plurality of nodes of the distributed system;
in response to identifying the failure:
making a first determination that the second node is a primary node of the distributed system;
after making the first determination:
identifying a new primary node of the plurality of nodes by accessing a copy of the distributed system configuration stored in one of the plurality of out-of-band management entities; and
providing services of the distributed system to a client using computing resources of the node after identifying the new primary node.
17. The method of claim 16 , wherein the method further comprises:
before identifying the new primary node:
making a second determination that the one of the plurality of the out-of-band management entities is reachable via an in-band channel routed through a network and an out-of-band channel of the one of the plurality of the out-of-band management entities; and
in response to the second determination:
accessing the distributed system configuration via the in-band channel.
18. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing a node of a distributed system comprising a plurality of nodes, wherein each node of the distributed system hosts a corresponding out-of-band management entity comprising separate computing resources from computing resources of the distributed system, wherein the computing resources of each of the out-of-band management entities store a copy of a distributed system configuration for the distributed system, the method comprising:
identifying a failure of a second node of the plurality of nodes of the distributed system;
in response to identifying the failure:
making a first determination that the second node is a primary node of the distributed system;
after making the first determination:
identifying a new primary node of the plurality of nodes by accessing a copy of the distributed system configuration stored in one of the plurality of out-of-band management entities; and
providing services of the distributed system to a client using computing resources of the node after identifying the new primary node.
19. The non-transitory computer readable medium of claim 18 , wherein the method further comprises:
before identifying the new primary node:
making a second determination that the one of the plurality of the out-of-band management entities is reachable via an in-band channel routed through a network and an out-of-band channel of the one of the plurality of the out-of-band management entities; and
in response to the second determination:
accessing the distributed system configuration via the in-band channel.