System and method for handling multi-node failures in a disaster recovery cluster
A system and method for handling multi-node failures in a disaster recovery cluster is provided. In the event of an error condition, a switchover operation occurs from the failed nodes to one or more surviving nodes. Data stored in non-volatile random access memory is recovered by the surviving nodes to bring storage objects, e.g., disks, aggregates and/or volumes into a consistent state.
1. A method comprising:
determining that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;
identifying an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;
performing the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;
recovering data from a non-volatile memory to the storage object; and
bringing the storage object online.
2. The method of claim 1 , wherein the determining comprises:
determining that the candidate node is not available based upon a determination that the candidate node would service a number of storage objects that exceeds an object limit if the candidate node was used for the switchover operation.
3. The method of claim 1 , wherein the determining comprises:
determining that the candidate node is not available based upon a determination that the candidate node is suffering an error condition.
4. The method of claim 1 , wherein the plurality of nodes comprises a first node arranged in a high availability pairing with a second node.
5. The method of claim 4 , comprising:
detecting a failure on the first node; and
performing a failover operation from the first node to the second node.
6. The method of claim 1 , wherein the identifying comprises:
identifying the alternate node based upon a determination that the alternate node would not service a number of storage objects that exceeds an object limit if the alternate node was used for the switchover operation.
7. The method of claim 1 , wherein recovering data from the non-volatile memory comprises:
replaying a portion of data stored in the non-volatile memory that was mirrored from the failed node.
8. The method of claim 1 , wherein the identifying comprises:
determining that a potential alternate node is not available based upon a determination that the potential alternate node would service a number of storage objects that exceeds an object limit if the potential alternate node was used for the switchover operation.
9. The method of claim 8 , comprising:
modifying the object limit to create an increased object limit for the potential alternate node; and
determining that the potential alternate node is available as the alternate node based upon a subsequent determination that the potential alternate node would not service a number of storage objects that exceeds the increased object limit if the potential alternate node was used for the switchover operation.
10. A computing device comprising:
a memory containing computer readable medium having stored thereon instructions for performing a method; and
a processor coupled with the memory, the processor configured to execute the instructions to cause the processor to:
determine that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;
identify an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;
perform the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;
recover data from a non-volatile memory to the storage object; and
bring the storage object online.
11. The computing device of claim 10 , wherein the instructions cause the processor to:
determine that the candidate node is not available based upon a determination that the candidate node would service a number of storage objects that exceeds an object limit if the candidate node was used for the switchover operation.
12. The computing device of claim 10 , wherein the instructions cause the processor to:
determine that the candidate node is not available based upon a determination that the candidate node is suffering an error condition.
13. The computing device of claim 10 , wherein the instructions cause the processor to:
identify the alternate node based upon a determination that the alternate node would not service a number of storage objects that exceeds an object limit if the alternate node was used for the switchover operation.
14. The computing device of claim 10 , wherein the instructions cause the processor to:
replay a portion of data stored in the non-volatile memory that was mirrored from the failed node.
15. The computing device of claim 10 , wherein the instructions cause the processor to:
determine that a potential alternate node is not available for the switchover operation based upon a determination that the potential alternate node would service a number of storage objects that exceeds an object limit if the potential alternate node was used for the switchover operation.
16. The computing device of claim 15 , wherein the instructions cause the processor to:
modify the object limit to create an increased object limit for the potential alternate node.
17. The computing device of claim 16 , wherein the instructions cause the processor to:
determine that the potential alternate node is available for the switchover operation based upon a subsequent determination that the potential alternate node would not service a number of storage objects that exceeds the increased object limit if the potential alternate node was used for the switchover operation.
18. The computing device of claim 10 , wherein the storage object is stored within a storage device that is modified, based upon the switchover operation, to conform to the alternate node.
19. A computer readable medium comprising program instructions, that when executed by a processor, cause the processor to:
determine that a candidate node is not available for a switchover operation to takeover for a failed node of a plurality of nodes;
identify an alternate node for the switchover operation based upon a determination that the alternate node has resources capable of handling a load of the failed node and that an object limit option for the alternate node is disabled;
perform the switchover operation to directly transfer ownership of a storage object of the failed node to the alternate node without transferring ownership of the storage object to the candidate node;
recover data from a non-volatile memory to the storage object; and
bring the storage object online.
20. The computer readable medium of claim 19 , wherein the program instructions cause the processor to:
replay a portion of data stored in the non-volatile memory that was mirrored from the failed node.