Heartbeat monitoring of virtual machines for initiating failover operations in a data storage management system, including operations by a master monitor node
An illustrative “VM heartbeat monitoring network” of heartbeat monitor nodes monitors target VMs in a data storage management system. Accordingly, target VMs are distributed and re-distributed among illustrative worker monitor nodes according to preferences in an illustrative VM distribution logic. Worker heartbeat monitor nodes use an illustrative ping monitoring logic to transmit special-purpose heartbeat packets to respective target VMs and to track ping responses. If a target VM is ultimately confirmed failed by its worker monitor node, an illustrative master monitor node triggers an enhanced storage manager to initiate failover for the failed VM. The enhanced storage manager communicates with the heartbeat monitor nodes and also manages VM failovers and other storage management operations in the system. Special features for cloud-to-cloud failover scenarios enable a VM in a first region of a public cloud to fail over to a second region.
1. A method comprising:
configuring a first data agent as a master monitor node in a data storage management system, wherein:
the first data agent is in communication with a storage manager that manages storage management operations in the data storage management system, and
the first data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a first virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
wherein the master monitor node comprises an instance of a distributed file system comprising a plurality of files, including a local copy of a first data file for managing heartbeat monitoring of virtual machines, wherein changes to the first data file are propagated by the distributed file system to other monitor nodes in the data storage management system;
configuring a second data agent as a first worker monitor node in the data storage management system, wherein the first worker monitor node performs heartbeat monitoring of a plurality of virtual machines assigned thereto by the master monitor node, wherein:
the second data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a second virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
the first worker monitor node comprises an instance of the distributed file system, including a corresponding local copy of the first data file that indicates the plurality of virtual machines to be monitored by the first worker monitor node;
by the master monitor node, based on a change made by the distributed file system to a local copy of at least one of the plurality of files in the distributed file system, detecting indicia that the first worker monitor node has failed;
querying one of: (a) the nonvirtualized computing device, (b) the hypervisor that hosts the second virtual machine, and (c) a controller of a virtual machine data center comprising the second virtual machine about an operational status of the second data agent; and
if, responsive to the querying, the operational status of the second data agent is reported to be failed, (I) updating, by the master monitor node, a list of failed worker monitor nodes in the distributed file system indicating that the first worker monitor node has failed, and (II) re-assigning, by the master monitor node, the plurality of virtual machines assigned to the failed first worker monitor node to a third data agent configured as a second worker monitor node in the data storage management system, wherein the updated list of failed worker monitor nodes and the re-assigning are reflected in changes made by the distributed file system to at least the first data file, which is propagated by the distributed file system to other monitor nodes in the storage management system, including to an instance of the distributed file system at the second worker monitor node.
2. The method of claim 1 wherein after the re-assigning, the second worker monitor node performs heartbeat monitoring of the plurality of virtual machines assigned thereto by the master monitor node.
3. The method of claim 1 wherein:
the third data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a third virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory.
4. The method of claim 1 wherein:
the third data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a third virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
detecting, by the second worker monitor node, based on an intentional change in the distributed file system, that the plurality of virtual machines have been re-assigned thereto.
5. The method of claim 1 further comprising:
if, responsive to the querying, the operational status of the first worker monitor node is reported to be failed, before the re-assigning of the plurality of virtual machines, (III) determining whether any of the plurality of virtual machines are also confirmed failed in the data storage management system, and (IV) notifying the storage manager to call failover for any of the plurality of virtual machines confirmed failed.
6. The method of claim 1 further comprising:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine.
7. The method of claim 1 further comprising:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine, wherein prior to the failover operation, the storage manager managed a replication operation of the first virtual machine to the second virtual machine.
8. The method of claim 1 further comprising:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine, wherein prior to the failover operation, the storage manager managed a live synchronization operation of the first virtual machine to the second virtual machine.
9. The method of claim 1 , wherein:
the first data file comprises a hierarchical data structure, and
intentional changes to the hierarchical data structure are propagated to the other monitor nodes in the storage management system to signal status changes in one or more of the virtual machines of the plurality of virtual machines.
10. The method of claim 1 , wherein:
the status of the first worker monitor node is implemented as an ephemeral node, and
the intentional deletion in the distributed file system comprises a deletion of the ephemeral node.
11. A non-transitory, computer-readable media having computer-executable instructions stored thereon that, when executed by one or more processors, configures a system to perform a plurality of operations comprising:
configuring a first data agent as a master monitor node in a data storage management system, wherein:
the first data agent is in communication with a storage manager that manages storage management operations in the data storage management system, and
the first data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a first virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
the master monitor node comprises an instance of a distributed file system comprising a plurality of files, including a local copy of a first data file for managing heartbeat monitoring of virtual machines, wherein changes to the first data file are propagated by the distributed file system to other monitor nodes in the data storage management system;
configuring a second data agent as a first worker monitor node in the data storage management system, wherein the first worker monitor node performs heartbeat monitoring of a plurality of virtual machines assigned thereto by the master monitor node, wherein:
the second data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a second virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
the first worker monitor node comprises an instance of the distributed file system, including a corresponding local copy of the first data file that indicates the plurality of virtual machines to be monitored by the first worker monitor node;
by the master monitor node, based on a change made by the distributed file system to a local copy of at least one of the plurality of files in the distributed file system, detecting indicia that the first worker monitor node has failed;
querying one of: (a) the nonvirtualized computing device, (b) the hypervisor that hosts the second virtual machine, and (c) a controller of a virtual machine data center comprising the second virtual machine about an operational status of the second data agent; and
if, responsive to the querying, the operational status of the second data agent is reported to be failed, (I) updating, by the master monitor node, a list of failed worker monitor nodes in the distributed file system indicating that the first worker monitor node has failed, and (II) re-assigning, by the master monitor node, the plurality of virtual machines assigned to the failed first worker monitor node to a third data agent configured as a second worker monitor node in the data storage management system, wherein the updated list of failed worker monitor nodes and the re-assigning are reflected in changes made by the distributed file system to at least the first data file, which is propagated by the distributed file system to other monitor nodes in the storage management system, including to an instance of the distributed file system at the second worker monitor node.
12. The non-transitory, computer-readable media of claim 11 wherein, after the re-assigning, the second worker monitor node performs heartbeat monitoring of the plurality of virtual machines assigned thereto by the master monitor node.
13. The non-transitory, computer-readable media of claim 11 , wherein:
the third data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a third virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory.
14. The non-transitory, computer-readable media of claim 11 , wherein:
the third data agent executes on one of (i) a nonvirtualized computing device comprising one or more processors and computer memory, and (ii) a third virtual machine hosted by a hypervisor executing on a computing device comprising one or more processors and computer memory, and
detecting, by the second worker monitor node, based on an intentional change in the distributed file system, that the plurality of virtual machines have been re-assigned thereto.
15. The non-transitory, computer-readable media of claim 11 , wherein the plurality of operations further comprises:
if, responsive to the querying, the operational status of the first worker monitor node is reported to be failed, before the re-assigning of the plurality of virtual machines, (III) determining whether any of the plurality of virtual machines are also confirmed failed in the data storage management system, and (IV) notifying the storage manager to call failover for any of the plurality of virtual machines confirmed failed.
16. The non-transitory, computer-readable media of claim 11 , wherein the plurality of operations further comprises:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine.
17. The non-transitory, computer-readable media of claim 11 , wherein the plurality of operations further comprises:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine, wherein prior to the failover operation, the storage manager managed a replication operation of the first virtual machine to the second virtual machine.
18. The non-transitory, computer-readable media of claim 11 , wherein the plurality of operations further comprises:
managing, by the storage manager, a respective failover operation for any of the plurality of virtual machines confirmed failed, wherein the respective failover operation activates a corresponding second virtual machine to operate in place of a failed first virtual machine, wherein prior to the failover operation, the storage manager managed a live synchronization operation of the first virtual machine to the second virtual machine.
19. The non-transitory, computer-readable media of claim 11 , wherein:
the first data file comprises a hierarchical data structure, and
intentional changes to the hierarchical data structure are propagated to the other monitor nodes in the storage management system to signal status changes in one or more of the virtual machines of the plurality of virtual machines.
20. The non-transitory, computer-readable media of claim 11 , wherein:
the status of the first worker monitor node is implemented as an ephemeral node, and
the intentional deletion in the distributed file system comprises a deletion of the ephemeral node.