Degraded availability zone remediation for multi-availability zone clusters of host computers
System and computer-implemented method for managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment automatically detects a degraded state of a first AZ in the cloud computing environment based on host failure events for host computers in a first cluster section of a multi-AZ cluster of host computers located in the first AZ and a recovered state of the first AZ based a successful scale-in operation of another multi-AZ cluster located partially in the first AZ. In response to the detection of the degraded state of the first AZ, a second cluster section of the multi-AZ cluster of host computers located in a second AZ is scaled out. In response to the detection of the recovered state of the first AZ, the second cluster section of the multi-AZ cluster of host computers located in the second AZ is scaled in.
1 . A computer-implemented method for managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment, the method comprising:
automatically detecting a degraded state of a first AZ in the cloud computing environment having a first cluster section of a multi-AZ cluster of host computers located in the first AZ;
scaling out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detecting a recovered state of the first AZ, wherein automatically detecting the recovered state of the first AZ includes selecting a random multi-AZ cluster located partially in the first AZ to execute a recovery workflow on the random multi-AZ cluster, wherein the recovery workflow includes checking health states of all host computers in a section of the random multi-AZ cluster that is located in the first AZ to determine whether the first AZ has recovered; and
scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
2 . The computer-implemented method of claim 1 , wherein automatically detecting the degraded state of the first AZ includes receiving an auto-remediation failure notification of a replacement host computer being unable to be provisioned in a first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notification indicates that the first AZ is potentially a degraded AZ.
3 . The computer-implemented method of claim 2 , wherein automatically detecting the degraded state of the first AZ includes determining that the first AZ is a degraded AZ when a virtualization manager for the multi-AZ cluster of host computers has lost connection to all the hosts in the first cluster section of the multi-AZ cluster of host computers located in the first AZ.
4 . The computer-implemented method of claim 1 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes two or three host computers.
5 . The computer-implemented method of claim 1 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals half of a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes more than three host computers.
6 . The computer-implemented method of claim 1 wherein the recovery workflow further includes determining that the first AZ has recovered when the health states of all the host computers in the section of the random multi-AZ cluster that is located in the first AZ are healthy.
7 . The computer-implemented method of claim 6 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ is part of scaling in all multi-AZ clusters of host computers located partially in the first AZ based the successful scale-in operation of the another multi-AZ cluster located partially in the first AZ.
8 . The computer-implemented method of claim 1 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes removing a number of host computers in the second cluster section of the multi-AZ cluster of host computers, wherein the number of host computers removed is equal to a number of host computers that were added to the second cluster section of the multi-AZ cluster of host computers located in the second AZ during the scaling out.
9 . A non-transitory computer-readable storage medium containing program instructions for auto or managing multi-availability zone (AZ) clusters of host computers in a cloud computing environment, wherein execution of the program instructions by one or more processors causes the one or more processors to perform steps comprising:
automatically detecting a degraded state of a first AZ in the cloud computing environment having a first cluster section of a multi-AZ cluster of host computers located in the first AZ, wherein automatically detecting the recovered state of the first AZ includes selecting a random multi-AZ cluster located partially in the first AZ to execute a recovery workflow on the random multi-AZ cluster, wherein the recovery workflow includes checking health states of all host computers in a section of the random multi-AZ cluster that is located in the first AZ to determine whether the first AZ has recovered;
scaling out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detecting a recovered state of the first AZ; and
scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein automatically detecting the degraded state of the first AZ includes receiving an auto-remediation failure notification of a replacement host computer being unable to be provisioned in a first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notification indicates that the first AZ is potentially a degraded AZ.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein automatically detecting the degraded state of the first AZ includes determining that the first AZ is a degraded AZ when a virtualization manager for the multi-AZ cluster of host computers has lost connection to all the hosts in the first cluster section of the multi-AZ cluster of host computers located in the first AZ.
12 . The non-transitory computer-readable storage medium of claim 9 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes two or three host computers.
13 . The non-transitory computer-readable storage medium of claim 9 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals half of a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes more than three host computers.
14 . The non-transitory computer-readable storage medium of claim 9 , wherein the recovery workflow further includes determining that the first AZ has recovered when the health states of all the host computers in the section of the random multi-AZ cluster that is located in the first AZ are healthy.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ is part of scaling in all multi-AZ clusters of host computers located partially in the first AZ based the successful scale-in operation of the another multi-AZ cluster located partially in the first AZ.
16 . The non-transitory computer-readable storage medium of claim 9 , wherein scaling in the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes removing a number of host computers in the second cluster section of the multi-AZ cluster of host computers, wherein the number of host computers removed is equal to a number of host computers that were added to the second cluster section of the multi-AZ cluster of host computers located in the second AZ during the scaling out.
17 . A system comprising:
memory; and
at least one processor configured to:
automatically detect a degraded state of a first availability zone (AZ) in a cloud computing environment having a first cluster section of a multi-AZ cluster of host computers located in the first AZ;
scale out a second cluster section of the multi-AZ cluster of host computers located in a second AZ in response to the detecting of the degraded state of the first AZ;
automatically detect a recovered state of the first AZ, wherein automatically detecting the recovered state of the first AZ includes selecting a random multi-AZ cluster located partially in the first AZ to execute a recovery workflow on the random multi-AZ cluster, wherein the recovery workflow includes checking health states of all host computers in a section of the random multi-AZ cluster that is located in the first AZ to determine whether the first AZ has recovered; and
scale in the second cluster section of the multi-AZ cluster of host computers located in the second AZ in response to the detecting of the recovered state of the first AZ.
18 . The system of claim 17 , wherein automatically detecting the degraded state of the first AZ includes receiving an auto-remediation failure notification of a replacement host computer being unable to be provisioned in a first cluster section of the multi-AZ cluster of host computers located in the first AZ, wherein the auto-remediation failure notification indicates that the first AZ is potentially a degraded AZ.
19 . The system of claim 17 , wherein automatically detecting the degraded state of the first AZ includes determining that the first AZ is a degraded AZ when a virtualization manager for the multi-AZ cluster of host computers has lost connection to all the hosts in the first cluster section of the multi-AZ cluster of host computers located in the first AZ.
20 . The system of claim 17 , wherein scaling out the second cluster section of the multi-AZ cluster of host computers located in the second AZ includes adding a number of new host computers to the second cluster section of the multi-AZ cluster of host computers, wherein the number of new host computers equals a number of host computers in the first cluster section of the multi-AZ cluster of host computers located in the first AZ when each of the first and cluster sections of the multi-AZ cluster of host computers includes two or three host computers.