Snapshot-based disaster recovery orchestration of virtual machine failover and failback operations
Snapshot-based disaster recovery (DR) orchestration systems and methods for virtual machine (VM) failover and failback do not require that VMs or their corresponding datastores be actively operating at the DR site before a DR orchestration job is initiated, i.e., before failover. An illustrative data storage management system deploys proprietary components at source data center(s) and at DR site(s). The proprietary components (e.g., storage manager, data agents, media agents, backup nodes, etc.) interoperate with each other and with the source and DR components to ensure that VMs will successfully failover and/or failback. DR orchestration jobs are suitable for testing VM failover scenarios (“clone testing”), for conducting planned VM failovers, and for unplanned VM failovers. DR orchestration jobs also handle failback and integration of DR-generated data into the failback site, including restoring VMs that never failed over to fully re-populate the source/failback site.
1. A data storage management system for orchestrating virtual machine failover, the system comprising:
a first computing device comprising one or more hardware processors;
wherein the first computing device is configured to:
cause a primary data storage (i) to take a first snapshot of a first data storage volume hosting a first datastore for a first virtual machine, and (ii) to store the first snapshot at the primary data storage, wherein the first virtual machine executes on a first virtual machine host computing device comprising one or more hardware processors and a hypervisor;
cause the primary data storage to replicate the first snapshot to a second snapshot stored in a second data storage volume at a failover data storage that is distinct from the primary data storage,
wherein the primary data storage and the failover data storage have a mirror-relationship that enables replication of snapshots therebetween;
initiate a disaster recovery orchestration job for the first virtual machine to fail over to a second virtual machine that is currently powered off, based on administrative settings that define the second virtual machine as a failover destination of the first virtual machine;
cause a failover virtualization manager to create, for the second virtual machine, a second datastore in the second data storage volume, wherein the second datastore is based on the second snapshot;
cause the failover virtualization manager to cause a second virtual machine host computing device to power up the second virtual machine and to provide the second virtual machine with access to the second datastore, wherein the second virtual machine operates with data in the second datastore replicated from the first snapshot; and
wherein the second datastore is created, and the second virtual machine is powered up, on-demand by the disaster recovery orchestration job.
2. The system of claim 1 , wherein the disaster recovery orchestration job is for an unplanned failover of the first virtual machine to the second virtual machine.
3. The system of claim 1 , wherein the disaster recovery orchestration job is initiated based on detecting a failure at one or more of: (a) the first virtual machine host computing device, a (b)) the primary data storage, and a (c) a first virtualization manager associated with the first virtual machine host computing device.
4. The system of claim 1 , wherein as part of the disaster recovery orchestration job, the first computing device is further configured to activate, on-demand, a data agent associated with the failover virtualization manager and a media agent associated with the failover data storage.
5. The system of claim 1 , wherein the first virtual machine executes in one of: a first virtualized data center and a first cloud computing environment; and
wherein after the disaster recovery orchestration job, the second virtual machine executes in one of: another distinct virtualized data center configured for disaster recovery and another cloud computing environment configured for disaster recovery.
6. The system of claim 1 , wherein the first computing device is further configured to:
initiate a second disaster recovery orchestration job that causes the second virtual machine to fail back to the first virtual machine, wherein the second disaster recovery orchestration job causes a first virtualization manager to re-activate the first virtual machine and establishes in the primary data storage the first datastore of the re-activated first virtual machine based on a snapshot replicated from the failover data storage.
7. The system of claim 6 , wherein the first computing device is further configured to, during the second disaster recovery orchestration job:
determine that a third virtual machine is not included in a failover group that comprises the first virtual machine;
determine that the third virtual machine is powered off and did not fail over in the disaster recovery orchestration job;
identify a backup copy of the third virtual machine;
initiate a restore job that restores the backup copy of the third virtual machine to a third datastore at the primary data storage; and
cause the first virtualization manager to re-activate the third virtual machine with access to the third datastore.
8. The system of claim 1 , wherein the first virtual machine and the second virtual machine are part of a failover group defined in the administrative settings.
9. The system of claim 1 , wherein the first computing device is further configured to cause the mirror-relationship to break when initiating the disaster recovery orchestration job.
10. The system of claim 1 , wherein the disaster recovery orchestration job causes the mirror-relationship to break.
11. The system of claim 1 , wherein the first computing device executes a storage manager that controls storage operations in the data storage management system.
12. A non-transitory computer-readable medium comprising instructions that, when executed by a first computing device comprising one or more hardware processors, cause the first computing device to:
initiate a disaster recovery orchestration job for a first virtual machine, wherein the disaster recovery orchestration job comprises:
one or more of: detect that the first virtual machine has failed, and cause the first virtual machine to be powered off, wherein the first virtual machine has a first datastore in a primary data storage
cause the primary data storage (i) to take a first snapshot of a first data storage volume hosting the first datastore, and (ii) to store the first snapshot at the primary data storage;
cause the primary data storage to replicate the first snapshot to a second snapshot at a failover data storage that is distinct from the primary data storage,
wherein the second snapshot is stored in a second data storage volume at the failover data storage, and
wherein the primary data storage and the failover data storage have a mirror-relationship that enables replication of snapshots therebetween;
one or more of: cause the mirror-relationship to break, and confirm that the mirror-relationship is broken;
cause the failover data storage to bring the second data storage volume online;
cause a failover virtualization manager to create, for a second virtual machine that is powered off, a second datastore in the second data storage volume, wherein the second datastore is based on the second snapshot;
cause the failover virtualization manager to cause a virtual machine host computing device to power up the second virtual machine and to provide the second virtual machine with access to the second datastore,
wherein the second virtual machine operates with data in the second datastore replicated from the first snapshot; and
wherein the second datastore is created, and the second virtual machine is powered up, on-demand by the disaster recovery orchestration job.
13. The non-transitory computer-readable medium of claim 12 , wherein the first virtual machine has failed due to one or more of: ( a ) a failure at a first virtual machine host computing device that hosts the first virtual machine, ( b ) a failure at the primary data storage, and ( c ) a failure at a first virtualization manager associated with the first virtual machine host computing device.
14. The non-transitory computer-readable medium of claim 12 , wherein the instructions further cause the first computing device to activate, on-demand, a data agent associated with the failover virtualization manager and a media agent associated with the failover data storage.
15. The non-transitory computer-readable medium of claim 12 , wherein the first virtual machine executes in one of: a first virtualized data center and a first cloud computing environment; and
wherein after the disaster recovery orchestration job, the second virtual machine executes in one of: another distinct virtualized data center configured for disaster recovery and another cloud computing environment configured for disaster recovery.
16. The non-transitory computer-readable medium of claim 12 , wherein the instructions further cause the first computing device to initiate a second disaster recovery orchestration job that causes the second virtual machine to fail back to the first virtual machine, wherein the second disaster recovery orchestration job causes a first virtualization manager to re-activate the first virtual machine and establishes in the primary data storage the first datastore of the re-activated first virtual machine based on a snapshot replicated from the failover data storage.
17. The non-transitory computer-readable medium of claim 16 , wherein the instructions further cause the first computing device to:
determine that a third virtual machine is not included in a failover group that comprises the first virtual machine;
determine that the third virtual machine is powered off and did not fail over in the disaster recovery orchestration job;
identify a backup copy of the third virtual machine;
initiate a restore job that restores the backup copy of the third virtual machine to a third datastore at the primary data storage; and
cause the first virtualization manager to re-activate the third virtual machine with access to the third datastore.
18. The non-transitory computer-readable medium of claim 12 , wherein the first virtual machine and the second virtual machine are part of a failover group defined in administrative settings.
19. The non-transitory computer-readable medium of claim 12 , wherein the instructions further cause the first computing device to control storage operations in a data storage management system.
20. The non-transitory computer-readable medium of claim 12 , wherein the instructions further cause the first computing device to instruct a data agent to cause a first virtualization manager to power off the first virtual machine.