Distributed software update by multiple nodes
During operation, a controller node, in multiple nodes in a cluster, may provide to the multiple nodes a set of operations associated with an update of controller software for a controller, where at least the controller node implements the controller. Moreover, at least a first node in the multiple nodes may perform the set of operations associated with the update of the controller software. Furthermore, at least a second node in the multiple nodes may monitor the performing of the set of operations by at least the first node. When the performing of the set of operations is completed by at least the first node, the first node may terminate the performing of the set of operations by the multiple nodes. Alternatively, when a failure occurs during the update of the controller software, at least the first node or the second node may automatically recover the multiple nodes.
1. A computer system, comprising:
multiple nodes in a cluster, wherein the computer system is configured to:
provide, from a controller node in the multiple nodes, a set of operations to the multiple nodes, wherein the set of operations are associated with an update of controller software for a controller of a network, and wherein at least the controller node implements the controller;
perform, by at least a first node in the multiple nodes, the set of operations associated with the update of the controller software;
monitor, by at least a second node in the multiple nodes, the performing of the set of operations by at least the first node;
when the performing of the set of operations is completed by at least the first node, terminate, by the first node, the performing of the set of operations by the multiple nodes; and
when a failure occurs during the update of the controller software, automatically recover, by at least the first node or the second node, the multiple nodes.
2. The computer system of claim 1 , wherein the multiple nodes are configured to independently perform at least an operation in the set of operations; and
wherein the multiple nodes are configured to sequentially perform at least the operation or concurrently perform at least the operation.
3. The computer system of claim 1 , wherein the first node is configured to operate as a leader node in the multiple nodes; and
wherein the leader node automatically performs the recovering when the failure occurs.
4. The computer system of claim 1 , wherein the second node is configured to operate as a leader node; and
wherein the leader node automatically performs the recovering when the failure occurs.
5. The computer system of claim 1 , wherein the computer system comprises multiple groups of nodes and a given group of nodes has an associated instance of a leader node configured to automatically perform the recovering of the given group when the failure occurs.
6. The computer system of claim 1 , wherein the monitoring comprises receiving, from at least the first node, a completion message when a given operation in the set of operations is completed; and
wherein, when the completion message is received, at least the second node is configured to skip performing the given operation in the set of operations.
7. The computer system of claim 1 , wherein at least the second node does not receive a completion message associated with a given operation in the set of operations from the first node within a predefined time interval, at least the second node is configured to:
terminate the performing of the set of operations by the multiple nodes; and
automatically recover the multiple nodes.
8. The computer system of claim 7 , wherein the completion message is not received within the predefined time interval when there is a network failure.
9. The computer system of claim 8 , wherein, when there is a network failure, at least the first node and the second node are configured to automatically recover the multiple nodes.
10. The computer system of claim 1 , wherein, when a given operation in the set of operations performed by the first node fails, the first node is configured to notify at least the second node;
wherein the second node is configured to skip performing operations following the given operation in the set of operations based at least in part on the notification; and
wherein the first node is configured to automatically recover the multiple nodes.
11. The computer system of claim 1 , wherein the recovering comprises: installing a backup image of the controller software on the multiple nodes; and restoring operation of the multiple nodes.
12. The computer system of claim 1 , wherein, when the second node determines that a connection with a third node in the multiple nodes is unavailable when the first node is performing the set of operations, the second node is configured to terminate the performing of the set of operations by the multiple nodes; and
wherein, after the second node terminates the performing of the set of operations by the multiple nodes, at least the first node is configured to automatically recover the multiple nodes.
13. The computer system of claim 1 , wherein the set of operations comprises: preparing the update; migrating a database; preparing the computer system for the update; upgrading the computer system; and performing post-update operations.
14. A non-transitory computer-readable storage medium for use in conjunction with a computer system that comprises multiple nodes, the computer-readable storage medium storing program instructions that, when executed by the computer system, cause the computer system to perform operations comprising:
provide, from a controller node in the multiple nodes, a set of operations to the multiple nodes, wherein the set of operations are associated with an update of controller software for a controller of a network, and wherein at least the controller node implements the controller;
perform, by at least a first node in the multiple nodes, the set of operations associated with the update of the controller software;
monitor, by at least a second node in the multiple nodes, the performing of the set of operations by at least the first node;
when the performing of the set of operations is completed by at least the first node, terminate, by the first node, the performing of the set of operations by the multiple nodes; and
when a failure occurs during the update of the controller software, automatically recover, by at least the first node or the second node, the multiple nodes.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the monitoring comprises receiving, from at least the first node, a completion message when a given operation in the set of operations is completed; and
wherein, when the completion message is received, the operations comprise at least the second node skipping performing the given operation in the set of operations.
16. The non-transitory computer-readable storage medium of claim 14 , wherein at least the second node does not receive a completion message associated with a given operation in the set of operations from the first node within a predefined time interval, the operations comprise at least the second node:
terminating the performing of the set of operations by the multiple nodes; and
automatically recovering the multiple nodes.
17. A method for performing an update of controller software, comprising:
by a computer system comprising multiple nodes:
provide, from a controller node in the multiple nodes, a set of operations to the multiple nodes, wherein the set of operations are associated with the update of the controller software for a controller of a network, and wherein at least the controller node implements the controller;
perform, by at least a first node in the multiple nodes, the set of operations associated with the update of the controller software;
monitor, by at least a second node in the multiple nodes, the performing of the set of operations by at least the first node;
when the performing of the set of operations is completed by at least the first node, terminate, by the first node, the performing of the set of operations by the multiple nodes; and
when a failure occurs during the update of the controller software, automatically recover, by at least the first node or the second node, the multiple nodes.
18. The method of claim 17 , wherein the monitoring comprises receiving, from at least the first node, a completion message when a given operation in the set of operations is completed; and
wherein, when the completion message is received, the method comprises at least the second node skipping performing the given operation in the set of operations.
19. The method of claim 17 , wherein the first node or the second node comprises a leader node in the multiple nodes; and
wherein, when the failure occurs, the leader node automatically performs the recovering.
20. The method of claim 17 , wherein at least the second node does not receive a completion message associated with a given operation in the set of operations from the first node within a predefined time interval, the method comprises at least the second node:
terminating the performing of the set of operations by the multiple nodes; and
automatically recovering the multiple nodes.