Resilient backups for large Hyper-V cluster shared volume environments
Embodiments are directed to a large-scale data backup system and method. A backup process accesses a defined policy defining how the virtual machines are to be grouped into a plurality of groups, and groups the plurality of virtual machines into the plurality of groups according to the defined policy with each group containing a number of virtual machines that can be backed up in a single backup operation without causing a system error. It then takes a snapshot backup of each group of the plurality of groups in a respective data rollover operation, and saves the snapshot backup of each group to a storage media. The policy may be a user-defined policy or a dynamic system-defined policy, and may be based on characteristics such as total number of virtual machines, system characteristics, and backup system constraints.
1. A computer-implemented method of backing up a large number of virtual machines in a large-scale data storage system including a backup server computer having a microprocessor and executing an operating system and accessing the virtual machines over a network, comprising:
configuring a virtual shadow copy service (VSS) as an operating system resident snapshot creation component to initiate and manage snapshot backups of the virtual machines;
accessing a defined policy defining how the virtual machines are to be grouped into a plurality of groups, wherein the virtual machines are organized as cluster shared volume (CSV) disks;
determining, as part of the defined policy, a maximum number of virtual machines that can be backed up by the VSS at any one time, wherein the maximum number is determined by at least one of prior knowledge of VSS failure conditions due to excessive numbers of backup targets, operating system guidelines, and test data indicating how many backup targets trigger the failure conditions;
dividing, by the microprocessor, a total number of virtual machines by the maximum number to derive a group size such that the group size is less than or equal to the maximum number;
grouping, in a component of the backup server computer, the plurality of virtual machines into the plurality of groups according to the defined policy, each group containing a number of virtual machines that can be backed up in a single snapshot backup operation using the VSS to ensure that a backup operation performed by the backup server computer does not fail due to the excessive number of backup targets and thereby improve efficiency of the data storage system;
taking, through the backup, a VSS snapshot backup of each group of the plurality of groups in a respective data rollover operation; and
saving the snapshot backup of each group to a storage media.
2. The method of claim 1 wherein the large-scale data storage system comprises a private cloud network infrastructure.
3. The method of claim 1 wherein the snapshot comprises a point-in-time, read-only backup of one or more files or volumes on one or more virtual machines, and wherein the backup operation comprises one of a full system backup or an incremental backup.
4. The method of claim 1 wherein the defined policy comprises one of a user-defined policy, and a dynamic system-defined policy.
5. The method of claim 4 wherein the CSV disks comprise a cluster disk containing file system volumes that are made accessible for read and write operations by all nodes within a cluster, and wherein CSVs are resolved to respective cluster owner nodes.
6. The method of claim 5 wherein the snapshot backup comprises one of: a full point-in-time copy, a pointer-based snapshot, and a copy-on-first-write snapshot backup.
7. The method of claim 6 wherein the user-defined policy comprises at least one of: a priority of sets of virtual machines within a total number of virtual machines in the large-scale data storage system, and a proportional distribution scheme dividing the total number of virtual machines into approximately equal size groups.
8. The method of claim 6 wherein the dynamic policies comprise at least one of: a total number of virtual machines in the large-scale data storage system, the identity of each CSV, a size of each CSV, and an identity of the respective cluster owner nodes.
9. The method of claim 6 further comprising, prior to accessing the defined policy, determining characteristics of the large-scale backup system, the characteristics comprising: the total number of virtual machines, type and amount of data on each virtual machine, organization of the virtual machines on each CSV, the identity of each cluster owner node, and the size of the CSV volumes.
10. The method of claim 9 further comprising checking if a user-defined policy exists and executing the user-defined policy prior to a dynamic policy if so, otherwise executing the dynamic policy, wherein the dynamic policy utilizes the characteristics of the large-scale backup system.
11. The method of claim 1 wherein the snapshot backup for each group is saved to the storage media in successive rollover operations in which a previous snapshot job spawns a successive snapshot job, each snapshot job backing up a respective group.
12. A system comprising a processor-based executable module configured to back up a large number of virtual machines organized as cluster shared volume (CSV) disks in a large-scale data storage system comprising a backup server having a microprocessor and a executing an operating system, comprising:
a virtual shadow copy service (VSS) configured as an operating system resident snapshot creation component to initiate and manage snapshot backups of the virtual machines;
a policy manager component of the backup server computer, storing a policy defining how the virtual machines are to be grouped into a plurality of groups;
a component determining, as part of the defined policy, a maximum number of virtual machines that can be backed up by the VSS at any one time, wherein the maximum number is determined by at least one of prior knowledge of VSS failure conditions due to excessive numbers of backup targets, operating system guidelines, and test data indicating how many backup targets trigger the failure conditions;
a microprocessor component dividing a total number of virtual machines by the maximum number to derive a group size such that the group size is less than or equal to the maximum number, wherein the group size is derived to ensure that a backup operation performed by the backup server computer does not fail due to the excessive number of backup targets and thereby improve efficiency of the data storage system;
a backup component of the backup server computer taking a VSS snapshot backup of each group of the plurality of groups in a respective data rollover operation; and
a storage media interface component saving the snapshot backup of each group to a storage media.
13. The system of claim 12 wherein the snapshot backup for each group is saved to the storage media by the backup component in successive rollover operations in which a previous snapshot job spawns a successive snapshot job, each snapshot job backing up a respective group.
14. The system of claim 12 wherein the CSV disks comprise a cluster disk containing file system volumes that are made accessible for read and write operations by all nodes within a cluster, and wherein CSVs are resolved to respective cluster owner nodes.
15. The system of claim 14 wherein the defined policy comprises one of a user-defined policy, and a dynamic system-defined policy, and wherein the user-defined policy comprises at least one of: a priority of sets of virtual machines within a total number of virtual machines in the large-scale data storage system, and a proportional distribution scheme dividing the total number of virtual machines into approximately equal size groups, and further wherein the dynamic policies comprise at least one of: a total number of virtual machines in the large-scale data storage system, an identity of each CSV, a size of each CSV, and an identity of the respective cluster owner nodes.
16. The system of claim 15 wherein the policy manager component determines characteristics of the large-scale backup system, the characteristics comprising: the total number of virtual machines, type and amount of data on each virtual machine, organization of the virtual machines on each CSV, the identity of each cluster owner node, and the size of the CSV volumes.
17. The system of claim 16 wherein the policy manager component is configured to first check if a user-defined policy exists and execute the user-defined policy prior to a dynamic policy if so, otherwise it is configured to execute the dynamic policy, wherein the dynamic policy utilizes the characteristics of the large-scale backup system.
18. The system of claim 12 wherein the large-scale data storage system comprises a deduplication data backup system utilizing a hypervisor to expose the virtual machines to one or more networks.
19. The system of claim 18 wherein the snapshot comprises a point-in-time, read-only backup of one or more files or volumes on one or more virtual machines, and wherein the backup operation comprises one of a full system backup or an incremental backup.
20. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for backing up a large number of virtual machines in a large-scale data storage system including a backup server computer having a microprocessor and executing an operating system and accessing the virtual machines over a network, by:
configuring a virtual shadow copy service (VSS) as an operating system resident snapshot creation component to initiate and manage snapshot backups of the virtual machines;
accessing a defined policy defining how the virtual machines are to be grouped into a plurality of groups, wherein the virtual machines are organized as cluster shared volume (CSV) disks;
determining, as part of the defined policy, a maximum number of virtual machines that can be backed up by the VSS at any one time, wherein the maximum number is determined by at least one of prior knowledge of VSS failure conditions due to excessive numbers of backup targets, operating system guidelines, and test data indicating how many backup targets trigger the failure conditions;
dividing, by the microprocessor, a total number of virtual machines by the maximum number to derive a group size such that the group size is less than or equal to the maximum number;
grouping, in a component of the backup server computer, the plurality of virtual machines into the plurality of groups according to the defined policy, each group containing a number of virtual machines that can be backed up in a single snapshot backup operation using the VSS to ensure that a backup operation performed by the backup server computer does not fail due to the excessive number of backup targets and thereby improve efficiency of the data storage system;
taking, through the backup, a VSS snapshot backup of each group of the plurality of groups in a respective data rollover operation; and
saving the snapshot backup of each group to a storage media.