Parallel upgrade of nodes in a storage system
An apparatus comprises a processing device that is configured to identify first and second subsets of nodes in a plurality of nodes, the first subset of nodes having dependencies with a first node in the plurality of nodes and the second subset of nodes having dependencies with a second node in the plurality of nodes. The processing device is configured to select the first node for an upgrade and to determine that the second node is not included in the first subset. The processing device is configured to determine that the first node is not included in the second subset and to upgrade the first and second nodes at least partially in parallel based at least in part on the determination that the second node is not included in the first subset and the determination that the first node is not included in the second subset.
1. An apparatus comprising at least one processing device comprising a processor coupled to a memory, the at least one processing device configured:
to identify a first subset of nodes in a plurality of nodes of a storage system, the first subset of nodes comprising nodes having dependencies with a first node in the plurality of nodes;
to identify a second subset of nodes in the plurality of nodes, the second subset of nodes comprising nodes having dependencies with a second node in the plurality of nodes;
to select the first node for an upgrade;
to determine that the second node is not included in the first subset of nodes;
to determine that the first node is not included in the second subset of nodes; and
to upgrade the first and second nodes at least partially in parallel based at least in part on the determination that the second node is not included in the first subset of nodes and the determination that the first node is not included in the second subset of nodes;
wherein prior to upgrading the first node, the at least one processing device is further configured:
to determine that the first node comprises an active deployment container that is configured to manage upgrades for at least a subset of the plurality of nodes;
to determine that the first node was selected for the upgrade;
to identify a given node of the plurality of nodes that is not selected for the upgrade;
to activate a deployment container on the given node; and
to disable the deployment container on the first node;
wherein after the upgrade of the first node is complete, the at least one processing device is further configured:
to activate the deployment container on the first node; and
to disable the deployment container on the given node.
2. The apparatus of claim 1 wherein the at least one processing device is further configured:
to identify a third subset of nodes in the plurality of nodes, the third subset of nodes comprising nodes having dependencies with a third node in the plurality of nodes;
to identify a fourth subset of nodes in the plurality of nodes, the fourth subset of nodes comprising nodes having dependencies with a fourth node in the plurality of nodes;
to select the third node for an upgrade, the first subset of nodes comprising the third node;
to determine that the fourth node is not included in the third subset of nodes;
to determine that the third node is not included in the fourth subset of nodes; and
to upgrade the third and fourth nodes at least partially in parallel based at least in part on the determination that the fourth node is not included in the third subset of nodes and the determination that the third node is not included in the fourth subset of nodes, wherein the upgrading of the third and fourth nodes is performed asynchronously with the upgrading of the first and second nodes.
3. The apparatus of claim 1 wherein identifying the given node of the plurality of nodes that is not selected for the upgrade comprises one of determining that the given node has already been upgraded and determining that the given node is included in the first subset of nodes as a node that has a dependency with the first node.
4. The apparatus of claim 1 wherein the upgrading is performed based at least in part on an upgrade package comprising a plurality of components to be upgraded and wherein the at least one processing device is further configured:
to obtain a plurality of deployment manifests associated with the plurality of components, each deployment manifest comprising an indication of at least one health check associated with a corresponding component to be performed as part of the upgrade;
to aggregate the obtained deployment manifests into a health deployment manifest, the health deployment manifest comprising a combined list of health checks based at least in part on the indication included in each deployment manifest; and
to perform at least one health check based at least in part on the health deployment manifest.
5. The apparatus of claim 4 wherein the list of health checks comprises a priority level for each health check and wherein performing the at least one health check based at least in part on the health deployment manifest comprises performing at least one health check having a given priority level in the list of health checks.
6. The apparatus of claim 5 wherein, responsive to an occurrence of an error during the upgrade, the at least one processing device is further configured to perform one or more of:
at least one additional health check having a different priority level than the given priority level in the list of health checks; and
at least one additional health check that is not included in the list of health checks.
7. The apparatus of claim 1 wherein a given node in the plurality of nodes is configured to service a plurality of clusters and wherein the at least one processing device is further configured:
to associate each of the clusters with a separate state machine for the given node, each state in the state machine associated with a given cluster corresponding to a component to be upgraded on the given node for that cluster;
responsive to a detection of an error during an upgrade of a component of the given cluster on the given node, to revert the upgrade of the given cluster to a predetermined component of the given cluster based at least in part on the state machine associated with the given cluster; and
to restart the upgrade of the given cluster on the given node from the predetermined component of the given cluster.
8. A method comprising:
identifying a first subset of nodes in a plurality of nodes of a storage system, the first subset of nodes comprising nodes having dependencies with a first node in the plurality of nodes;
identifying a second subset of nodes in the plurality of nodes, the second subset of nodes comprising nodes having dependencies with a second node in the plurality of nodes;
selecting the first node for an upgrade;
determining that the second node is not included in the first subset of nodes;
determining that the first node is not included in the second subset of nodes; and
upgrading the first and second nodes at least partially in parallel based at least in part on the determination that the second node is not included in the first subset of nodes and the determination that the first node is not included in the second subset of nodes;
wherein prior to upgrading the first node, the method further comprises:
determining that the first node comprises an active deployment container that is configured to manage upgrades for at least a subset of the plurality of nodes;
determining that the first node was selected for the upgrade;
identifying a given node of the plurality of nodes that is not selected for the upgrade;
activating a deployment container on the given node; and
disabling the deployment container on the first node; and
wherein after the upgrade of the first node is complete, the method further comprises:
activating the deployment container on the first node; and
disabling the deployment container on the given node;
wherein the method is implemented by at least one processing device of the storage system, the at least one processing device comprising a processor coupled to memory.
9. The method of claim 8 wherein the method further comprises:
identifying a third subset of nodes in the plurality of nodes, the third subset of nodes comprising nodes having dependencies with a third node in the plurality of nodes;
identifying a fourth subset of nodes in the plurality of nodes, the fourth subset of nodes comprising nodes having dependencies with a fourth node in the plurality of nodes;
selecting the third node for an upgrade, the first subset of nodes comprising the third node;
determining that the fourth node is not included in the third subset of nodes;
determining that the third node is not included in the fourth subset of nodes; and
upgrading the third and fourth nodes at least partially in parallel based at least in part on the determination that the fourth node is not included in the third subset of nodes and the determination that the third node is not included in the fourth subset of nodes, wherein the upgrading of the third and fourth nodes is performed asynchronously with the upgrading of the first and second nodes.
10. The method of claim 8 wherein identifying the given node of the plurality of nodes that is not selected for the upgrade comprises one of determining that the given node has already been upgraded and determining that the given node is included in the first subset of nodes as a node that has a dependency with the first node.
11. The method of claim 8 wherein the upgrading is performed based at least in part on an upgrade package comprising a plurality of components to be upgraded and wherein the method further comprises:
obtaining a plurality of deployment manifests associated with the plurality of components, each deployment manifest comprising an indication of at least one health check associated with a corresponding component to be performed as part of the upgrade;
aggregating the obtained deployment manifests into a health deployment manifest, the health deployment manifest comprising a combined list of health checks based at least in part on the indication included in each deployment manifest; and
performing at least one health check based at least in part on the health deployment manifest.
12. The method of claim 11 wherein the list of health checks comprises a priority level for each health check and wherein performing the at least one health check based at least in part on the health deployment manifest comprises performing at least one health check having a given priority level in the list of health checks.
13. The method of claim 12 wherein, responsive to an occurrence of an error during the upgrade, the method further comprises performing one or more of:
at least one additional health check having a different priority level than the given priority level in the list of health checks; and
at least one additional health check that is not included in the list of health checks.
14. The method of claim 8 wherein a given node in the plurality of nodes is configured to service a plurality of clusters and wherein the method further comprises:
associating each of the clusters with a separate state machine for the given node, each state in the state machine associated with a given cluster corresponding to a component to be upgraded on the given node for that cluster;
responsive to a detection of an error during an upgrade of a component of the given cluster on the given node, reverting the upgrade of the given cluster to a predetermined component of the given cluster based at least in part on the state machine associated with the given cluster; and
restarting the upgrade of the given cluster on the given node from the predetermined component of the given cluster.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device of a storage system, the at least one processing device comprising a processor coupled to a memory, causes the at least one processing device:
to identify a first subset of nodes in a plurality of nodes of the storage system, the first subset of nodes comprising nodes having dependencies with a first node in the plurality of nodes;
to identify a second subset of nodes in the plurality of nodes, the second subset of nodes comprising nodes having dependencies with a second node in the plurality of nodes;
to select the first node for an upgrade;
to determine that the second node is not included in the first subset of nodes;
to determine that the first node is not included in the second subset of nodes; and
to upgrade the first and second nodes at least partially in parallel based at least in part on the determination that the second node is not included in the first subset of nodes and the determination that the first node is not included in the second subset of nodes;
wherein prior to upgrading the first node, the program code further causes the at least one processing device:
to determine that the first node comprises an active deployment container that is configured to manage upgrades for at least a subset of the plurality of nodes;
to determine that the first node was selected for the upgrade;
to identify a given node of the plurality of nodes that is not selected for the upgrade;
to activate a deployment container on the given node; and
to disable the deployment container on the first node;
wherein after the upgrade of the first node is complete, the program code further causes the at least one processing device:
to activate the deployment container on the first node; and
to disable the deployment container on the given node.
16. The computer program product of claim 15 wherein identifying the given node of the plurality of nodes that is not selected for the upgrade comprises one of determining that the given node has already been upgraded and determining that the given node is included in the first subset of nodes as a node that has a dependency with the first node.
17. The computer program product of claim 15 wherein the upgrading is performed based at least in part on an upgrade package comprising a plurality of components to be upgraded and wherein the program code further causes the at least one processing device:
to obtain a plurality of deployment manifests associated with the plurality of components, each deployment manifest comprising an indication of at least one health check associated with a corresponding component to be performed as part of the upgrade;
to aggregate the obtained deployment manifests into a health deployment manifest, the health deployment manifest comprising a combined list of health checks based at least in part on the indication included in each deployment manifest; and
to perform at least one health check based at least in part on the health deployment manifest.
18. The computer program product of claim 17 wherein the list of health checks comprises a priority level for each health check and wherein performing the at least one health check based at least in part on the health deployment manifest comprises performing at least one health check having a given priority level in the list of health checks.
19. The computer program product of claim 18 wherein, responsive to an occurrence of an error during the upgrade, the program code further causes the at least one processing device to perform one or more of:
at least one additional health check having a different priority level than the given priority level in the list of health checks; and
at least one additional health check that is not included in the list of health checks.
20. The computer program product of claim 15 wherein a given node in the plurality of nodes is configured to service a plurality of clusters and wherein the program code further causes the at least one processing device:
to associate each of the clusters with a separate state machine for the given node, each state in the state machine associated with a given cluster corresponding to a component to be upgraded on the given node for that cluster;
responsive to a detection of an error during an upgrade of a component of the given cluster on the given node, to revert the upgrade of the given cluster to a predetermined component of the given cluster based at least in part on the state machine associated with the given cluster; and
to restart the upgrade of the given cluster on the given node from the predetermined component of the given cluster.