Automated disaster recovery test verification
A Disaster Recovery (DR) verification process or system verifies readiness to execute a Disaster Recovery (DR) test. Such DR verification is automatically provided on a regular basis, with minimum impact on both a production and recovery environment. However, DR verification is only enabled at a time when no disaster recovery or DR testing of other sites is already under way. Configuration data needed for DR verification is constantly replicated from production to recovery, by dedicated data movers. DR verification allocates and configures recovery resources in an encapsulated environment, confirms successful their instantiation (such as boot up and network connections), and reports the outcome.
1. A method for verifying readiness for a recovery environment to execute a Disaster Recovery (DR) test of a production environment, the method executing on one or more data processors that access program code to:
provide the production environment comprising:
one or more protected virtual machines (VMs),
a production data mover component, for replicating configuration and content data for the one or more protected VMs to the recovery environment, the production data mover component operating continuously during normal operation mode of the production environment,
the data mover component dedicated exclusively to this recovery environment replication task and thus operating independently of other data mover or backup tasks in the production environment;
provide the recovery environment comprising:
one or more recovery VMs;
a recovery data mover component, for receiving and storing the configuration and content data of the one or more protected VMs;
an operations database maintaining operations data for the recovery environment, the operations data including operations other than DR verify;
and the method further comprising:
executing a scheduler process, for:
determining a schedule for running a DR verify process, based on availability of the one or more recovery VMs and other resources in the recovery environment, and not being based on any current operational mode of the production environment or availability of production resources therein;
executing a DR verification process, for:
at a time set according to the schedule:
providing access to selected ones of the one or more recovery VMs to the stored configuration and content data;
verifying successful boot of the selected one or more recovery VMs;
provisioning network resources for use by the selected one or more recovery VMs;
determining a recovery time needed to boot the selected ones of the recovery VMs and provision the other recovery resources; and
reporting the recovery time.
2. The method of claim 1 wherein the DR verification process further comprises a predefined subset of operations specified by the DR test.
3. The method of claim 1 additionally comprising:
comparing the recovery time against a Recovery Time Objective (RTO) for the production environment.
4. The method of claim 1 comprising two or more production environments, and the data mover components further comprise:
a virtual recovery appliance associated with each set of protected VMs for a given production environment;
a virtual recovery appliance associated with one or more of the recovery VMs;
a virtual data mover manager, for configurating and controlling the virtual recovery appliances.
5. The method of claim 4 wherein the schedule additionally includes two or more times for different ones of the respective production environments, and additionally comprising:
determining recovery times for each respective production environment; and
comparing the recovery times against a Recovery Time Objective (RTO) for each respective production environment.
6. The method of claim 1 wherein at the time of execution of the DR verification process, further direct access to data in the production environment is not provided during successful boot of the selected one or more recovery VMs.
7. The method of claim 1 wherein a hypervisor performs the step of verifying whether the VMs booted correctly.
8. The method of claim 1 wherein one or more of the data mover components operate with different types of hypervisors.
9. The method of claim 1 wherein determining the schedule further comprises:
determining whether disaster recovery or DR testing is currently being performed for other production environments, and if so, then not proceeding with the DR verification.
10. The method of claim 1 wherein the selected one or more recovery VMs are arranged in an encapsulated environment dedicated to the DR verification.
11. A system for verifying readiness for a recovery environment to execute a Disaster Recovery (DR) test of a production environment, the system comprising one or more processors configured to execute program code stored in one or more memories, the program code for further providing:
the production environment comprising:
one or more protected virtual machines (VMs),
a production data mover component, for replicating configuration and content data for the one or more protected VMs to the recovery environment, the production data mover component operating continuously during normal operation mode of the production environment,
the data mover component dedicated exclusively to this recovery environment replication task and thus operating independently of other data mover or backup tasks in the production environment;
the recovery environment comprising:
one or more recovery VMs;
a recovery data mover component, for receiving and storing the configuration and content data of the one or more protected VMs;
an operations database maintaining operations data for the recovery environment, the operations data including operations other than DR verify;
a scheduler process, for:
determining a schedule for running a DR verify process, based on availability of the one or more recovery VMs and other resources in the recovery environment, and not being based on any current operational mode of the production environment or availability of production resources therein;
a DR verification process, for:
at a time set according to the schedule:
providing access to selected ones of the one or more recovery VMs to the stored configuration and content data;
verifying successful boot of the selected one or more recovery VMs;
provisioning network resources for use by the selected recovery VMs;
determining a recovery time needed to boot the selected one or more recovery VMs and provision the other recovery resources; and
reporting the recovery time.
12. The system of claim 11 wherein the DR verification process further comprises a predefined subset of operations specified by the DR test.
13. The system of claim 11 additionally comprising:
comparing the recovery time against a Recovery Time Objective (RTO) for the production environment.
14. The system of claim 11 comprising two or more production environments, and the data mover components further comprise:
a virtual recovery appliance associated with each set of protected VMs for a given production environment;
a virtual recovery appliance associated with one or more of the recovery VMs;
a virtual data mover manager, for configurating and controlling the virtual recovery appliances.
15. The system of claim 14 wherein the schedule additionally includes two or more times for different ones of the respective production environments, and additionally comprising:
determining recovery times for each respective production environment; and
comparing the recovery times against a Recovery Time Objective (RTO) for each respective production environment.
16. The system of claim 11 wherein at the time of execution of the DR verification process, further direct access to data in the production environment is not provided during successful boot of the selected one or more recovery VMs.
17. The system of claim 11 wherein a hypervisor performs the step of verifying whether the VMs booted correctly.
18. The system of claim 11 wherein one or more of the data mover components operate with different types of hypervisors.
19. The system of claim 11 wherein determining the schedule further comprises:
determining whether disaster recovery or DR testing is currently being performed for other production environments, and if so, then not proceeding with the DR verification.
20. The system of claim 11 wherein the selected one or more recovery VMs are arranged in an encapsulated environment dedicated to the DR verification.