Degraded availability zone remediation for multi-availability zone clusters of host computers
System and computer-implemented method for managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment automatically detects a degraded state of a first AZ in the cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ and a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ. In response to the detection of the degraded state of the first AZ, a second cluster section of the multi-AZ cluster of host computers located in a second AZ is scaled out. In response to the detection of the recovered state of the first AZ, the second cluster section of the multi-AZ cluster of host computers located in the second AZ is scaled in.
1. A computer-implemented method for managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment, the method comprising:
automatically detecting a degraded state of a first AZ in the cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ;
scaling out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detecting a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ; and
scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
2. The computer-implemented method of claim 1 , wherein automatically detecting the degraded state of the first AZ includes receiving auto-remediation failure notifications of replacement host computers being unable to be provisioned in the first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notifications indicate that the first AZ is potentially a degraded AZ.
3. The computer-implemented method of claim 2 , wherein automatically detecting the degraded state of the first AZ includes determining that the first AZ is a degraded AZ when a virtualization manager for the multi-AZ cluster of host computers has lost connection to all the hosts in the first cluster section of the multi-AZ cluster of host computers located in the first AZ.
4. The computer-implemented method of claim 1 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes two or three host computers.
5. The computer-implemented method of claim 1 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals half of a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes more than three host computers.
6. The computer-implemented method of claim 1 , wherein automatically detecting the recovered state of the first AZ includes selecting a random multi-AZ cluster located partially in the first AZ to execute a recovery workflow on the random multi-AZ cluster, wherein the recovery workflow includes checking health states of all host computers in a section of the random multi-AZ cluster that is located in the first AZ to determine whether the first AZ has recovered.
7. The computer-implemented method of claim 6 , wherein the recovery workflow further includes determining that the first AZ has recovered when the health states of all the host computers in the section of the random multi-AZ cluster that is located in the first AZ are healthy.
8. The computer-implemented method of claim 7 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ is part of scaling in all multi-AZ clusters of host computers located partially in the first AZ based the successful scale-in operation of the another multi-AZ cluster located partially in the first AZ.
9. The computer-implemented method of claim 1 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes removing a number of host computers in the second cluster section of the multi-AZ cluster of host computers, wherein the number of host computers removed is equal to a number of host computers that were added to the second cluster section of the multi-AZ cluster of host computers located in the second AZ during the scaling out.
10. A non-transitory computer-readable storage medium containing program instructions for auto or managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment, wherein execution of the program instructions by one or more processors causes the one or more processors to perform steps comprising:
automatically detecting a degraded state of a first AZ in the cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ;
scaling out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detecting a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ; and
scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
11. The non-transitory computer-readable storage medium of claim 10 , wherein automatically detecting the degraded state of the first AZ includes receiving auto-remediation failure notifications of replacement host computers being unable to be provisioned in the first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notifications indicate that the first AZ is potentially a degraded AZ.
12. The non-transitory computer-readable storage medium of claim 11 , wherein automatically detecting the degraded state of the first AZ includes determining that the first AZ is a degraded AZ when a virtualization manager for the multi-AZ cluster of host computers has lost connection to all the hosts in the first cluster section of the multi-AZ cluster of host computers located in the first AZ.
13. The non-transitory computer-readable storage medium of claim 10 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes two or three host computers.
14. The non-transitory computer-readable storage medium of claim 10 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals half of a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes more than three host computers.
15. The non-transitory computer-readable storage medium of claim 10 , wherein automatically detecting the recovered state of the first AZ includes selecting a random multi-AZ cluster located partially in the first AZ to execute a recovery workflow on the random multi-AZ cluster, wherein the recovery workflow includes checking health states of all host computers in a section of the random multi-AZ cluster that is located in the first AZ to determine whether the first AZ has recovered.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the recovery workflow further includes determining that the first AZ has recovered when the health states of all the host computers in the section of the random multi-AZ cluster that is located in the first AZ are healthy.
17. The non-transitory computer-readable storage medium of claim 16 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ is part of scaling in all multi-AZ clusters of host computers located partially in the first AZ based the successful scale-in operation of the another multi-AZ cluster located partially in the first AZ.
18. The non-transitory computer-readable storage medium of claim 10 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes removing a number of host computers in the second cluster section of the multi-AZ cluster of host computers, wherein the number of host computers removed is equal to a number of host computers that were added to the second cluster section of the multi-AZ cluster of host computers located in the second AZ during the scaling out.
19. A system comprising:
memory; and
at least one processor configured to:
automatically detect a degraded state of a first availability zone (AZ) in a cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ;
scale out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detect a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ; and
scale in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
20. The system of claim 19 , wherein the at least one processor is configured to receive auto-remediation failure notifications of replacement host computers being unable to be provisioned in the first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notifications indicate that the first AZ is potentially a degraded AZ.