Selecting a witness service when implementing a recovery plan
Methods, systems, and computer program products for selection of a witness during virtualization system recovery after a disaster event. A recovery plan is configured to identify a witness that is then used to elect a leader to implement the recovery. Various system, and/or network, and/or component failures and/or various loss of function of components of the virtualization system can trigger initiation of the recovery plan. Based on the particular recovery plan that is invoked upon a determination of a network outage, or component failure or loss of function of a component of the virtualization system, a particular witness corresponding to a subset of entities of the particular recovery plan is selected. The witness is used to elect the leader, and the leader initiates actions of the recovery plan. The implementation of the recovery plan includes consideration of the health of components that would potentially be involved in the recovery actions.
1. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor cause the processor to perform acts comprising:
configuring a recovery plan to address a recovery in a virtualization system, wherein the recovery plan identifies a witness that corresponds to a subset of entities of the virtualization system, and wherein the witness is used to elect a leader to implement the recovery in the virtualization system;
identifying an event in the virtualization system that triggers the recovery, wherein the recovery is of at least one of the subset of entities in the recovery plan;
identifying the witness from the recovery plan, based at least in part on the subset of entities in the recovery plan; and
initiating actions to implement the recovery plan using the identified witness to elect the leader.
2. The non-transitory computer readable medium of claim 1 , wherein the event in the virtualization system is at least one of, unreachability of a node of the virtualization system, or a failure of a virtual machine of the virtualization system, or a loss of function of an entity of the virtualization system.
3. The non-transitory computer readable medium of claim 1 , wherein the leader used to implement the recovery in the virtualization system is a node that is referenced in the recovery plan, or an application that is referenced in the recovery plan, or a virtual machine that is referenced in the recovery plan.
4. The non-transitory computer readable medium of claim 1 , wherein the witness is one particular witness from among a plurality of witnesses that are distributed in different fault domains.
5. The non-transitory computer readable medium of claim 1 , wherein the witness is in a first fault domain that is different from a second fault domain of the leader.
6. The non-transitory computer readable medium of claim 1 , wherein the event in the virtualization system affects a first virtualization system component in a first cloud environment, and wherein implementation of the recovery comprises bring-up of a replacement entity in a second cloud environment.
7. The non-transitory computer readable medium of claim 6 , wherein the event in the virtualization system corresponds to at least one of, degradation of liveness of the first virtualization system component, or a loss of communication with the first virtualization system component, or a decrease of health of the first virtualization system component, or a loss of function of the first virtualization system component.
8. A method comprising:
configuring a recovery plan to address a recovery in a virtualization system, wherein the recovery plan identifies a witness that corresponds to a subset of entities of the virtualization system, and wherein the witness is used to elect a leader to implement the recovery in the virtualization system;
identifying an event in the virtualization system that triggers the recovery, wherein the recovery is of at least one of the subset of entities in the recovery plan;
identifying the witness from the recovery plan, based at least in part on the subset of entities in the recovery plan; and
initiating actions to implement the recovery plan using the identified witness to elect the leader.
9. The method of claim 8 , wherein the event in the virtualization system is at least one of, unreachability of a node of the virtualization system, or a failure of a virtual machine of the virtualization system, or a loss of function of an entity of the virtualization system.
10. The method of claim 8 , wherein the leader used to implement the recovery in the virtualization system is a node that is referenced in the recovery plan, or an application that is referenced in the recovery plan, or a virtual machine that is referenced in the recovery plan.
11. The method of claim 8 , wherein the witness is one particular witness from among a plurality of witnesses that are distributed in different fault domains.
12. The method of claim 8 , wherein the witness is in a first fault domain that is different from a second fault domain of the leader.
13. The method of claim 8 , wherein the event in the virtualization system affects a first virtualization system component in a first cloud environment, and wherein implementation of the recovery comprises bring-up of a replacement entity in a second cloud environment.
14. The method of claim 13 , wherein the event in the virtualization system corresponds to at least one of, degradation of liveness of the first virtualization system component, or a loss of communication with the first virtualization system component, or a decrease of health of the first virtualization system component, or a loss of function of the first virtualization system component.
15. A system comprising:
a storage medium having stored thereon a sequence of instructions; and
a processor that executes the sequence of instructions to cause the processor to perform acts comprising,
configuring a recovery plan to address a recovery in a virtualization system, wherein the recovery plan identifies a witness that corresponds to a subset of entities of the virtualization system, and wherein the witness is used to elect a leader to implement the recovery in the virtualization system;
identifying an event in the virtualization system that triggers the recovery, wherein the recovery is of at least one of the subset of entities in the recovery plan;
identifying the witness from the recovery plan, based at least in part on the subset of entities in the recovery plan; and
initiating actions to implement the recovery plan using the identified witness to elect the leader.
16. The system of claim 15 , wherein the event in the virtualization system is at least one of, unreachability of a node of the virtualization system, or a failure of a virtual machine of the virtualization system, or a loss of function of an entity of the virtualization system.
17. The system of claim 15 , wherein the leader used to implement the recovery in the virtualization system is a node that is referenced in the recovery plan, or an application that is referenced in the recovery plan, or a virtual machine that is referenced in the recovery plan.
18. The system of claim 15 , wherein the witness is one particular witness from among a plurality of witnesses that are distributed in different fault domains.
19. The system of claim 15 , wherein the witness is in a first fault domain that is different from second fault domain of the leader.
20. The system of claim 15 , wherein the event in the virtualization system affects a first virtualization system component in a first cloud environment, and wherein implementation of the recovery comprises bring-up of a replacement entity in a second cloud environment.
21. The system of claim 20 , wherein the event in the virtualization system corresponds to at least one of, degradation of liveness of the first virtualization system component, or a loss of communication with the first virtualization system component, or a decrease of health of the first virtualization system component, or a loss of function of the first virtualization system component.