Parallel multi-rack dataplane upgrades
Examples described herein include systems and methods for parallel upgrades of multiple hosts within an on-premises, multi-rack software-defined data center (“SDDC”). Upgrade instructions can be provided to a point-of-presence (“POP”) virtual machine (“VM”) of the SDDC, which can call an application programming interface (“API”) of a pod service executing on a control plane VM in the SDDC. For example, the pod service can generate a set of instructions for performing cluster upgrades in parallel. In one example, the pod service provides the instructions to a lifecycle management service executing on a VM of the SDDC, such as the control plane VM. In some examples, the lifecycle management service can add a spare host to a cluster of hosts in order to begin upgrading one or more hosts in that cluster. This can be performed in parallel, using the spare hosts from different racks simultaneously.
1. A method for upgrading a multi-rack, on-premises software defined data center (SDDC), comprising:
receiving upgrade instructions at a point-of-presence (POP) virtual machine (VM) executing on the SDDC;
generating, by a pod service, a set of instructions for performing cluster upgrades in parallel, wherein the set of instructions identifies a plurality of clusters of the SDDC located on separate racks; and
performing a parallel upgrade of at least two of the identified plurality of clusters located on separate racks, wherein performing the parallel upgrade comprises:
adding a spare host to each of the identified plurality of clusters, wherein each rack of the SDDC includes at least one spare host;
transferring data from at least one host on each of the identified plurality of clusters to the spare host added to that cluster;
taking the at least one host on each of the identified plurality of clusters offline;
upgrading at least one of software and firmware on the offline hosts;
bringing the offline hosts online; and
removing the spare hosts from the identified plurality of clusters.
2. The method of claim 1 , further comprising calling, by the POP VM, a first application programming interface (API) of the pod service executing on a control plane VM of the SDDC, the call including instructions to upgrade the plurality of clusters of the SDDC in parallel.
3. The method of claim 1 , wherein upgrading comprises upgrading both software and firmware, and wherein the upgrading is performed in parallel on the offline hosts.
4. The method of claim 1 , wherein upgrading firmware on the offline hosts is performed by hardware security modules executing on each of the identified plurality of clusters, respectively.
5. The method of claim 4 , wherein the hardware security modules upgrade respective firmware based on instructions received from a lifecycle management service executing on a control plane VM of the SDDC.
6. The method of claim 1 , wherein upgrading software on the offline hosts is performed by a lifecycle management service executing on a control plane VM of the SDDC.
7. The method of claim 1 , wherein each rack includes only one spare host.
8. A non-transitory, computer-readable medium containing instructions that, when executed by a hardware-based processor, performs stages for upgrading a multi-rack, on-premises software defined data center (SDDC), the stages comprising:
receiving upgrade instructions at a point-of-presence (POP) virtual machine (VM) executing on the SDDC;
generating, by a pod service, a set of instructions for performing cluster upgrades in parallel, wherein the set of instructions identifies a plurality of clusters of the SDDC located on separate racks; and
performing a parallel upgrade of at least two of the identified plurality of clusters located on separate racks, wherein performing the parallel upgrade comprises:
adding a spare host to each of the identified plurality of clusters, wherein each rack of the SDDC includes at least one spare host;
transferring data from at least one host on each of the identified plurality of clusters to the spare host added to that cluster;
taking the at least one host on each of the identified plurality of clusters offline;
upgrading at least one of software and firmware on the offline hosts;
bringing the offline hosts online; and
removing the spare hosts from the identified plurality of clusters.
9. The non-transitory, computer-readable medium of claim 8 , the stages further comprising calling, by the POP VM, a first application programming interface (API) of the pod service executing on a control plane VM of the SDDC, the call including instructions to upgrade the plurality of clusters of the SDDC in parallel.
10. The non-transitory, computer-readable medium of claim 8 , wherein upgrading comprises upgrading both software and firmware, and wherein the upgrading is performed in parallel on the offline hosts.
11. The non-transitory, computer-readable medium of claim 8 , wherein upgrading firmware on the offline hosts is performed by hardware security modules executing on each of the identified plurality of clusters, respectively.
12. The non-transitory, computer-readable medium of claim 11 , wherein the hardware security modules upgrade respective firmware based on instructions received from a lifecycle management service executing on a control plane VM of the SDDC.
13. The non-transitory, computer-readable medium of claim 8 , wherein upgrading software on the offline hosts is performed by a lifecycle management service executing on a control plane VM of the SDDC.
14. The non-transitory, computer-readable medium of claim 8 , wherein each rack includes only one spare host.
15. A system for upgrading a multi-rack, on-premises software defined data center (SDDC), comprising:
a memory storage including a non-transitory, computer-readable medium comprising instructions; and
a computing device including a hardware-based processor that executes the instructions to carry out stages comprising:
receiving upgrade instructions at a point-of-presence (POP) virtual machine (VM) executing on the SDDC;
generating, by a pod service, a set of instructions for performing cluster upgrades in parallel, wherein the set of instructions identifies a plurality of clusters of the SDDC located on separate racks; and
performing a parallel upgrade of at least two of the identified plurality of clusters located on separate racks, wherein performing the parallel upgrade comprises:
adding a spare host to each of the identified plurality of clusters, wherein each rack of the SDDC includes at least one spare host;
transferring data from at least one host on each of the identified plurality of clusters to the spare host added to that cluster;
taking the at least one host on each of the identified plurality of clusters offline;
upgrading at least one of software and firmware on the offline hosts;
bringing the offline hosts online; and
removing the spare hosts from the identified plurality of clusters.
16. The system of claim 15 , the stages further comprising calling, by the POP VM, a first application programming interface (API) of the pod service executing on a control plane VM of the SDDC, the call including instructions to upgrade the plurality of clusters of the SDDC in parallel.
17. The system of claim 15 , wherein upgrading comprises upgrading both software and firmware, and wherein the upgrading is performed in parallel on the offline hosts.
18. The system of claim 15 , wherein upgrading firmware on the offline hosts is performed by hardware security modules executing on each of the identified plurality of clusters, respectively.
19. The system of claim 18 , wherein the hardware security modules upgrade respective firmware based on instructions received from a lifecycle management service executing on a control plane VM of the SDDC.
20. The system of claim 15 , wherein upgrading software on the offline hosts is performed by a lifecycle management service executing on a control plane VM of the SDDC.