Distributed control plane for handling worker node failures of a distributed storage architecture
Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. The distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. If a worker node fails and ownership of an object has changed from the failed worker node to another worker node, then subsequent commands are re-routed to a control plane controller paired with the other worker node now owning the object in place of the failed worker node.
1. A system, comprising:
a distributed storage architecture including a plurality of worker nodes managing distributed storage comprised of storage devices hosted by the plurality of worker nodes;
a container orchestration platform hosting applications running through containers;
a distributed control plane hosted within the container orchestration platform, wherein the distributed control plane comprises a plurality of pods hosting control plane controllers paired with the worker nodes, wherein the distributed control plane is configured to:
route a first command originating from an application running as a container within the container orchestration platform to a first control plane controller paired with a first worker node based upon the first command targeting an object owned by the first worker node; and
in response to detecting a failure of the first worker node, route a second command to a second control plane controller paired with a second worker node based upon the second command targeting the object whose ownership changed from the first worker node to the second worker node in response to the failure; and
the first control plane controller configured to:
translate the first command from being formatted according to a first model supported by the container orchestration platform into a first reformatted command formatted according to a second model supported by the distributed storage architecture; and
transmit the first reformatted command to the first worker node for execution through the distributed storage architecture.
2. The system of claim 1 , further comprising:
the second control plane controller configured to:
translate the second command from being formatted according to the first model into a second reformatted command formatted according to the second model; and
transmit the second reformatted command to the second worker node for execution through the distributed storage architecture.
3. The system of claim 1 , the distributed control plane is further configured to:
execute a polling thread to detect object ownership changes of objects amongst the worker nodes due to worker node failures.
4. The system of claim 1 , the distributed storage architecture is further configured to:
track health status information related to the plurality of worker nodes; and
provide the health status information to the distributed control plane for tracking ownership of objects by worker nodes, wherein ownership of a set of objects owned by a worker node is transferred to a different worker node based upon the health status information.
5. The system of claim 1 , the distributed storage architecture is further configured to:
track health status information related to the plurality of worker nodes; and
in response to the health status information indicating that the first worker node failed, transferring ownership of objects owned by the first worker node to the second worker node, wherein commands targeting the objects are routed to the second control plane controller paired with the second worker node based upon the ownership being transferred.
6. The system of claim 1 , the distributed storage architecture is further configured to:
change ownership of a first object from a worker node to a different worker node while retaining current storage locations of data of the first object within the distributed storage.
7. The system of claim 1 , the distributed control plane is further configured to:
in response to detecting the distributed storage architecture adding a new worker node to take over for a failed worker node, host a new control plane controller paired with the new worker node; and
route commands to the new control plane controller based upon the commands targeting objects owned by the new worker node.
8. The system of claim 1 , the distributed control plane is further configured to:
in response to detecting the distributed storage architecture removing a worker node, remove at least one of a control plane controller paired with the worker node or a pod hosting the control plane controller, wherein ownership of objects owned by the worker node are transferred to a different worker node.
9. The system of claim 1 , the distributed control plane is further configured to:
in response to detecting the failure of the first worker node, remove at least one of the first control plane controller paired with the first worker node or a pod hosting the first control plane controller, wherein ownership of objects owned by the first worker node are transferred to a different worker node.
10. A method, comprising:
receiving, by a distributed control plane hosted within a container orchestration platform hosting applications running through containers, a first command originating from an application running as a container within the container orchestration platform, wherein the command is formatted according to a first model supported by the container orchestration platform;
routing the first command from the application a first control plane controller paired with a first worker node of a distributed storage architecture based upon the first command targeting an object owned by the first worker node; and
in response to detecting a failure of the first worker node, routing a second command originating from the application to a second control plane controller paired with a second worker node of the distributed storage architecture based upon the second command targeting the object whose ownership changed from the first worker node to the second worker node based upon the failure.
11. The method of claim 10 , the method further comprising:
translating, by the first control plane controller, the first command from being formatted according to the first model into a first reformatted command formatted according to a second model supported by the distributed storage architecture; and
transmitting, by the first control plane controller, the first reformatted command to the first worker node for execution through the distributed storage architecture.
12. The method of claim 10 , the method further comprising:
translating, by the second control plane controller, the second command from being formatted according to the first model into a second reformatted command formatted according to a second model supported by the distributed storage architecture; and
transmitting, by the second control plane controller, the second reformatted command to the second worker node for execution through the distributed storage architecture.
13. The method of claim 10 , the method further comprising:
polling the distributed storage architecture to detect object ownership changes of objects amongst the worker nodes due to worker node failures, wherein ownership of objects of a failed worker node are transferred to a different worker node.
14. The method of claim 10 , the method further comprising:
providing, by the distributed storage architecture, health status information of the worker nodes to the distributed control plane, wherein the distributed control plane utilizes the health status information to determine which worker nodes are to be paired with which control plane controllers.
15. The method of claim 10 , the method further comprising:
tracking, by the first control plane controller, health status information of the first worker node, wherein ownership of objects owned by the first worker node are transferred to a different worker node based upon the health status information indicating a degraded health state of the first worker node.
16. The method of claim 10 , the method further comprising:
in response to health status information indicating that the first worker node failed, transferring ownership of objects owned by the first worker node to the second worker node, wherein commands targeting the objects are rerouted to the second control plane controller.
17. The method of claim 10 , the method further comprising:
in response to detecting the distributed storage architecture adding a new worker node to take over for a failed worker node, hosting a new control plane controller paired with the new worker node, wherein ownership of objects owned by the failed worker node are transferred to the new worker node.
18. The method of claim 10 , the method further comprising:
in response to detecting the distributed storage architecture removing a worker node, removing a control plane controller paired with the worker node, wherein ownership of objects owned by the worker node are transferred to a different worker node.
19. The method of claim 10 , the method further comprising:
in response to detecting the failure of the first worker node, removing the first control plane controller paired with the first worker node, wherein ownership of objects owned by the first worker node are transferred to a different worker node.
20. A non-transitory machine readable medium comprising instructions, which when executed by a machine, causes the machine to:
receive, by a distributed control plane hosted within a container orchestration platform hosting applications running through containers, a first command originating from an application running as a container within the container orchestration platform, wherein the command is formatted according to a first model supported by the container orchestration platform;
route the first command from the application to a first control plane controller paired with a first worker node of a distributed storage architecture based upon the first command targeting an object owned by the first worker node, wherein the first control plane controller reformats the first command according to a second model supported by the distributed storage architecture and transmits the first command to the first worker node; and
in response to detecting a failure of the first worker node, route a second command to a second control plane controller paired with a second worker node based upon the second command targeting the object whose ownership changed from the first worker node to the second worker node based upon the failure, wherein the second control plane controller reformats the second command according to the second model and transmits the second command to the second worker node.