Data protection component scaling in a cloud-based data storage system
Systems described herein may dynamically add one or more proxy data protection agents to a cloud data storage system to process a data protection job. Upon completion of the job or at some other appropriate interval, the system can power down and decommission the proxy data protection agents and/or the virtual machines on which the data protection proxies reside according to a cleanup schedule (e.g., at hourly or minute intervals). In order to improve the allocation of computing resources, the system takes into account currently existing proxies or virtual machines when processing a backup request to determine the need for new proxies to service the backup request. In this manner the system can save costs and computing resources through efficient virtual machine deployment and retirement.
1. A system comprising:
a group of computing resources comprising one or more physical host computers and a plurality of first virtual machines implemented on the one or more physical host computers, each of the plurality of first virtual machines having one or more software applications executing thereon; and
a management subsystem implemented on one or more computing devices, the one or more computing devices in networked communication with the group of computing resources, the management subsystem configured to:
analyze job information associated with a first data protection job request, the job information identifying at least a first set of the plurality of first virtual machines for which to perform data protection operations;
based on the job information, calculate a first number of data protection agents to install within the group of computing resources; and
cause transmission of one or more commands to the group of computing resources, wherein the one or more commands include commands to:
install the first number of data protection agents on one or more second virtual machines within the group of computing resources;
assign the first number of data protection agents to corresponding virtual machines of the first set of the plurality of first virtual machines; and
perform the first data protection job request, wherein during the performance of the first data protection job request, the first number of data protection agents direct performance of data protection operations associated with the corresponding virtual machines of the first set of the plurality of first virtual machines to which the first number of data protection agents are assigned.
2. The system of claim 1 , wherein the group of computing resources is a group of cloud resources, and wherein the management subsystem causes the transmission of the one or more commands by interacting with an application programming interface.
3. The system of claim 1 , wherein the management subsystem comprises a resource scaling manager implemented on a first computing device of the one or more computing devices and further comprises a storage manager implemented on a second computing device of the one or more computing devices, and wherein the storage manager transmits the first data protection job request to the resource scaling manager.
4. The system of claim 1 , wherein a first data protection agent of the first number of data protection agents is installed on a first virtual machine of the one or more second virtual machines, and wherein the management subsystem is further configured to cause the first virtual machine to power off or shut down in response to one or both of an activity level of the first data protection agent and an activity level of the first virtual machine.
5. The system of claim 4 , wherein the management subsystem is further configured to cause decommissioning of the first virtual machine.
6. The system of claim 4 , wherein the management subsystem is further configured to cause the first virtual machine to return from power off or shut down to an active state.
7. The system of claim 1 , wherein, at a point in time the management subsystem analyzes the job information, a second number of data protection agents is already instantiated within the group of computing resources, and wherein the management subsystem uses the second number in calculating the first number of data protection agents.
8. The system of claim 1 , wherein the management subsystem uses a total number of virtual machines in the plurality of first virtual machines in calculating the first number of data protection agents.
9. The system of claim 1 , wherein the management subsystem uses a data size associated with the first data protection job request and a data protection agent throughput value in calculating the first number of data protection agents.
10. The system of claim 1 wherein each of the first number of data protection agents comprises at least one data agent and at least one media agent.
11. A method comprising:
with a management subsystem implemented in a computing device that is in networked communication with a group of computing resources:
analyzing job information associated with a first data protection job request, the job information identifying at least a first set of a plurality of first virtual machines for which to perform data protection operations, the plurality of first virtual machines implemented on one or more first computing devices of the group of computing resources;
based on the job information, calculating a first number of data protection agents to install on the first computing devices; and
causing transmission of one or more commands to the group of computing resources, wherein the one or more commands include commands to:
install the first number of data protection agents on one or more second virtual machines within the group of computing resources;
assign the first number of data protection agents to corresponding virtual machines of the first set of the plurality of first virtual machines; and
perform the first data protection job request, wherein during the performance of the first data protection job request, the first number of data protection agents direct performance of data protection operations associated with the corresponding virtual machines of the first set of the plurality of first virtual machines to which the first number of data protection agents are assigned.
12. The method of claim 11 , wherein the group of computing resources is a group of cloud resources, and wherein said causing the transmission of the one or more commands is achieved by interacting with an application programming interface.
13. The method of claim 11 , wherein a first data protection agent of the first number of data protection agents is installed on a first virtual machine of the one or more second virtual machines, and the method further comprising causing the first virtual machine to power off or shut down in response to one or both of an activity level of the first data protection agent and an activity level of the first virtual machine.
14. The method of claim 13 , further comprising causing decommissioning of the first virtual machine.
15. The method of claim 13 , further comprising causing the first virtual machine to return to an active state.
16. The method of claim 11 , wherein, at a point in time the management subsystem analyzes the job information, a second number of data protection agents is already instantiated within the group of computing resources, and wherein said calculating comprises using the second number to calculate the first number of data protection agents.
17. The method of claim 11 , wherein said calculating comprises using a total number of virtual machines in the plurality of first virtual machines to calculate the first number of data protection agents.
18. The method of claim 11 , wherein said calculating comprises using a data size associated with the first data protection job request and a data protection agent throughput value to calculate the first number of data protection agents.
19. A non-transitory computer-readable medium storing instructions, which when executed by a computing device, cause the computing device to perform a method comprising:
analyzing job information associated with a first data protection job request, the job information identifying at least a first set of a plurality of first virtual machines for which to perform data protection operations, the plurality of first virtual machines implemented on one or more first computing devices of a group of computing resources;
based on the job information, calculating a first number of data protection agents to install on the first computing devices; and
causing transmission of one or more commands to the group of computing resources, wherein the one or more commands include commands to:
install the first number of data protection agents on one or more second virtual machines within the group of computing resources;
assign the first number of data protection agents to corresponding virtual machines of the first set of the plurality of first virtual machines; and
perform the first data protection job request, wherein during the performance of the first data protection job request, the first number of data protection agents direct performance of data protection operations associated with the corresponding virtual machines of the first set of the plurality of first virtual machines to which the first number of data protection agents are assigned.