Selecting a witness service when implementing a recovery plan
Methods, systems, and computer program products for selection of a witness during virtualization system recovery after a disaster event. A recovery plan is configured to identify a witness that is then used to elect a leader to implement the recovery. Various system, and/or network, and/or component failures and/or various loss of function of components of the virtualization system can trigger initiation of the recovery plan. Based on the particular recovery plan that is invoked upon a determination of a network outage, or component failure or loss of function of a component of the virtualization system, a particular witness corresponding to a subset of entities of the particular recovery plan is selected. The witness is used to elect the leader, and the leader initiates actions of the recovery plan. The implementation of the recovery plan includes consideration of the health of components that would potentially be involved in the recovery actions.
1. A non-transitory computer readable medium comprising a sequence of instructions which, when stored in memory and executed by a processor, cause the processor to perform a set of acts, the set of acts comprising:
determining a recovery plan that comprises information pertaining to multiple witnesses in a virtualization system;
determining a witness to implement a recovery of the virtualization system from the multiple witnesses based at least in part on the recovery plan; and
bringing up a replacement computing entity to replace another computing entity for the recovery of the virtualization system to implement the recovery at least by using at least the witness.
2. The non-transitory computer readable medium of claim 1 , bringing up the replacement computing entity to replace the another computing entity for the recovery comprising triggering the recovery of at least one subset of corresponding subsets of entities specified in the recovery plan in response to an event, wherein the recovery plan identifies the multiple witnesses that respectively correspond to the corresponding subsets of entities of the virtualization system, and the witness comprises a software service.
3. The non-transitory computer readable medium of claim 2 , wherein the witness is determined from the multiple witnesses based at least in part upon how the event affects the at least one subset of the corresponding subsets of entities, and an implementation of the recovery comprises bringing up the replacement computing entity in a cloud environment.
4. The non-transitory computer readable medium of claim 3 , wherein the recovery plan defines a relationship between at least two subsets of the corresponding subsets of entities, and the event in the virtualization system comprises at least one of degradation of unreachability of a computing node in the virtualization system, or a failure of a virtual machine in the virtualization system, or a loss of function of an entity in the virtualization system, or liveness of a first virtualization system component, or a loss of communication with the first virtualization system component, or a decrease of health of the first virtualization system component, or a loss of function of the first virtualization system component.
5. The non-transitory computer readable medium of claim 1 , wherein the witness is used to elect a leader that is used to implement the recovery in the virtualization system, and the leader comprises a node that is referenced in the recovery plan, or an application that is referenced in the recovery plan, or a virtual machine that is referenced in the recovery plan.
6. The non-transitory computer readable medium of claim 5 , wherein the witness is located in a first fault domain that is different from a second fault domain in which the leader is located, and the recovery plan identifies one or more specific acts to be carried out for the recovery and specifies one or more parameters pertaining to at least one of failover detection, or a witness location or usage, or a timing threshold, or an act to be carried out after a detected failure or outage, or an IP (Internet Protocol) address that accounts for a difference between heterogeneous environments.
7. The non-transitory computer readable medium of claim 1 , wherein the multiple witnesses are distributed in different fault domains, and the different fault domains comprise a first cloud computing domain and a second cloud computing domain.
8. The non-transitory computer readable medium of claim 1 , wherein two different recovery plans respectively correspond to two different witnesses, at least one recovery plan of the two different recovery plans comprises a specification of a plurality of subsets of an overall set of components of the virtualization system.
9. The non-transitory computer readable medium of claim 8 , wherein a different witness is determined, based at least in part upon the at least one recovery plan and an aspect of an event that triggers the recovery, for a different hierarchically-interrelated subset of a plurality of hierarchically-interrelated subsets of the recovery plan.
10. The non-transitory computer readable medium of claim 1 , wherein the witness is determined from the multiple witnesses based at least in part upon one or more parameters specified in the recovery plan, and the one or more parameters comprise at least one of respective locations or usage of the multiple witnesses, or a specific action to be carried out for bringing up a virtualization entity in the virtualization system for the recovery of the virtualization system, or a timing threshold.
11. A non-transitory computer readable medium comprising a sequence of instructions which, when stored in memory and executed by a processor, cause the processor to perform a set of acts, the set of acts comprising:
determining a recovery plan comprising a hierarchical relationship that includes multiple hierarchies to which a plurality of virtualization system components belong, wherein the recovery plan specifies, based at least in part upon the hierarchical relationship, a witness that facilitates recovery of a subset of the plurality of virtualization system components;
determining, based at least in part on the recovery plan, a lower hierarchy virtualization system component that is affected by an event;
determining a subset of virtualization system components for the recovery based at least in part upon the lower hierarchy virtualization system component; and
performing the recovery according to the recovery plan.
12. The non-transitory computer readable medium of claim 11 , wherein the set of acts further comprises:
determining a specific witness from the recovery plan based at least in part upon how the event affects the subset of virtualization system components, wherein a respective witness is determined for a corresponding subset of virtualization system components, and the specific witness or the respective witness comprises a software service.
13. The non-transitory computer readable medium of claim 12 , wherein the recovery plan comprises information pertaining to respective locations or respective usages of multiple witnesses from which the specific witness is determined for the subset of virtualization system components.
14. The non-transitory computer readable medium of claim 13 , wherein the witness is used to elect a leader computing node that further initiates one or more actions to implement the recovery plan for the recovery.
15. The non-transitory computer readable medium of claim 11 , wherein the recovery is performed to recover only the subset of virtualization system components of the plurality of virtualization system components.
16. The non-transitory computer readable medium of claim 11 , wherein determining the subset of virtualization system components comprises:
identifying a higher virtualization system component based at least in part upon information specified in the recovery plan, wherein the higher virtualization system component is located at a higher hierarchical level than a lower hierarchical level at which the lower hierarchy virtualization system component is located, and the recovery plan characterizes both the higher virtualization system component and the lower hierarchy virtualization system component.
17. The non-transitory computer readable medium of claim 11 , wherein the set of acts further comprises:
identifying, for the lower hierarchy virtualization system component, an entry from a plurality of entries in the recovery plan;
determining the subset of virtualization system components by using at least the entry; and
identifying a separate subset of virtualization system components based at least in part upon information or data codified in the recovery plan and the subset of virtualization system components, wherein the subset of virtualization system components corresponds to a first witness, the separate subset of virtualization system components corresponds to a second witness different from the first witness.
18. A method comprising:
determining a recovery plan that comprises multiple witnesses in a virtualization system;
determining a witness to implement a recovery of the virtualization system from the multiple witnesses based at least in part on the recovery plan; and
bringing up a replacement computing entity to replace another computing entity for the recovery of the virtualization system to implement the recovery at least by using at least the witness.
19. The method of claim 18 , wherein an event that triggers the recovery of at least a subset of entities in the virtualization system comprises at least one of unreachability of a computing node in the virtualization system, or a failure of a virtual machine in the virtualization system, or a loss of function of an entity in the virtualization system, and the event in the virtualization system affects a first virtualization system component in a first cloud environment, and an implementation of the recovery comprises bringing up the replacement computing entity in a cloud environment.
20. The method of claim 18 , wherein the witness that comprises a software service is used to elect a leader computing node that is used to implement the recovery in the virtualization system, and the leader computing node comprises a node that is referenced in the recovery plan, or an application that is referenced in the recovery plan, or a virtual machine that is referenced in the recovery plan, and the witness is located in a first fault domain that is different from a second fault domain in which the leader computing node is located.
21. A system comprising:
a storage medium having stored thereon a sequence of instructions; and
a processor that executes the sequence of instructions to cause the processor to perform a set of acts, the set of acts comprising:
determining a recovery plan comprising a hierarchical relationship that includes multiple hierarchies to which a plurality of virtualization system components belong, wherein the recovery plan specifies, based at least in part upon the hierarchical relationship, a witness that facilitates recovery of a subset of the plurality of virtualization system components;
determining, based at least in part on the recovery plan, a lower hierarchy virtualization system component that is affected by an event;
determining a subset of virtualization system components for a recovery based at least in part upon the lower hierarchy virtualization system component; and
performing the recovery according to the recovery plan.
22. The system of claim 21 , wherein the set of acts further comprises:
determining a specific witness from the recovery plan based at least in part upon how the event affects the subset of virtualization system components, wherein a respective witness is determined for a corresponding subset of virtualization system components, wherein the respective witness or the specific witness comprises a software service.
23. The system of claim 22 , wherein the recovery plan comprises information pertaining to respective locations or respective usages of multiple witnesses from which the specific witness is determined for the subset of virtualization system components, and the witness is used to elect a leader computing node that further initiates one or more actions to implement the recovery plan for the recovery.
24. The system of claim 21 , wherein determining the subset of virtualization system components comprises:
identifying a higher virtualization system component based at least in part upon information specified in the recovery plan, wherein the higher virtualization system component is located at a higher hierarchical level than a lower hierarchical level at which the lower hierarchy virtualization system component is located, and the recovery plan characterizes both the higher virtualization system component and the lower hierarchy virtualization system component.