Reaching a quorum with a number of master nodes
Managing containerized workloads, such as by Kubernetes, provides a robust and expandable platform. Kubernetes high-availability (HA) mode provides additional safeguards against failure that allows services to be maintained without interruption in the event of a partial system or network failure. However, Kubernetes requires an odd number of master nodes (e.g., three) in order to be able have a quorum and maintain certain operations (e.g., election of a leader). In the event a master node loses functionality, the remaining master nodes are unable to have a quorum. System and methods are provided to convert a worker-only node into a master node in order to reestablish a quorum. Once functionality is restored, the converted master is reverted back to a worker-only node and the restored master, and remaining masters, maintain the quorum.
1. A method, comprising:
determining, by a processor executing instructions from a non-transitory memory to exchange data in a network interconnecting components of a system, a loss of functionality of a first master node, wherein the first master node becomes unable to function as a master node;
wherein the components comprise master nodes and worker nodes;
wherein the master nodes comprise the first master node and a second master node; and
wherein each of the master nodes and each of the worker nodes are operable to execute an application from application instructions;
determining that the master nodes and the worker nodes execute the application in a first cluster and a second cluster; and
based on the determining of the loss of functionality of the first master node, converting one of the worker nodes to a new master node by copying state information to the one of the worker nodes.
2. The method of claim 1 , wherein the first master node is a member of the first cluster, wherein the one of the worker nodes is a member of the first cluster, wherein the second master node is a member of the second cluster, and wherein the state information is a member of the first cluster.
3. The method of claim 1 , wherein the determining the loss of functionality comprises:
periodically detecting an availability of a first building comprising the first cluster by the instructions executed by the processor on a component of a second building comprising the second cluster;
determining that the first building is not available on the network;
determining a number of master nodes within the second cluster; and
performing the converting the one of the worker nodes based on the determining that the first building is not available and the determining the number of master nodes within the second cluster.
4. The method of claim 3 , wherein the number of master nodes within the second cluster is one.
5. The method of claim 1 , wherein the converting the one of the worker nodes to the new master node comprises:
copying a configuration of the second master node comprising the state information at a time of the loss of functionality to the new master node.
6. The method of claim 5 , wherein the first master node is within the first cluster, wherein the second master node is within the second cluster, and wherein the configuration is saved in a database associated with the second cluster.
7. The method of claim 3 , further comprising determining that the first building is available, and converting the new master node to be the one of the worker nodes based on the determining that the first building is available.
8. A system, comprising:
a network interconnecting components of the system for exchange of data thereon;
at least one server with a processor and instructions for the processor in a non-transitory memory;
master nodes comprising a first master node and a second master node, wherein each of the master nodes are operable to execute an application from application instructions;
worker nodes;
a first cluster comprising the first master node;
a second cluster comprising the second master node and one of the worker nodes;
wherein the first master node loses functionality and the first master node becomes unable to function as a master node; and
wherein the one of the worker nodes is converted to a new master node by copying state information to the one of the worker nodes.
9. The system of claim 8 , wherein the second master node and the new master node comprise a quorum.
10. The system of claim 8 , wherein the conversion of the one of the worker nodes is based on the loss of functionality of the first master node.
11. The system of claim 8 , wherein the second master node comprises the state information that executes the application.
12. The system of claim 8 , wherein the state information enables the execution of the application.
13. The system of claim 8 , wherein the first cluster is located at a first building, wherein the second cluster is located at a second building, and wherein the first building and the second building are located in distinct geographical locations.
14. The system of claim 13 , wherein the first building comprises a third master node, and wherein the loss of functionality of the first master node comprises a loss of functionality of the first master node and the third master node.
15. The system of claim 8 , wherein the loss of functionality of the first master node comprises an inability to function as a master node.
16. The system of claim 8 , wherein the first cluster and the second cluster exchange the data to operate the system in a High Availability mode.
17. The system of claim 16 , wherein the High Availability mode functions to maintain an execution of the application to continuously provide a service from before the loss of functionality of the first master node to after the conversion of the new master node.
18. The system of claim 16 , wherein the system operates in the High Availability mode from before the loss of functionality of the first master node to after the conversion of the new master node.
19. A non-transitory computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to manage a worker node by:
determining, by a processor executing instructions from a non-transitory memory to exchange data in a network interconnecting components of a system, a loss of functionality of a first master node, wherein the first master node becomes unable to function as a master node;
wherein the components of the system comprise master nodes and worker nodes;
wherein the master nodes comprise the first master node and a second master node; and
wherein each of the master nodes are operable to execute an application from application instructions; and
based on the determining of the loss of functionality of the first master node, converting the worker node to a new master node, wherein the second master node and the new master node comprise a quorum.
20. The non-transitory computer-readable medium of claim 19 , wherein the loss of functionality comprises an inability to function as a master node.