Data protection component scaling in a cloud-based data storage system
Systems described herein may dynamically add one or more proxy agents to a cloud data storage system to process an information management job, such as restore job. Upon completion of the job or at some other appropriate interval, the system can power down and decommission the proxy agents and/or the virtual machines on which the proxies reside according to a cleanup schedule (e.g., at hourly or minute intervals). The system takes into account currently existing proxies or virtual machines when processing a restore request to determine the need for new proxies to service the restore request. In this manner the system can save costs and computing resources through efficient virtual machine deployment and retirement.
1 . A computer-implemented method of managing resources for performing information management operations, comprising:
receiving an information management operation job associated with one or more virtual machines,
wherein the information management operation job comprises at least one of a backup or restore operation;
calculating, based on a Central Processing Unit (CPU) count, a CPU-based number of virtual server agent (VSA) proxy agents required to perform the information management operation job;
determining a difference between the calculated number of required VSA proxy agents and a currently existing number of VSA proxy agents;
determining a total number of available data streams across all currently available VSA proxy agents;
calculating a stream-based number of additional VSA proxy agents required by determining a difference between a total virtual machine count associated with the information management operation job and the total number of available data streams, and dividing the difference by a maximum number of data streams predefined for a corresponding type of information management operation job; and
allocating, for performance of the information management operation job, a higher number of VSA proxy agents as determined by comparing the CPU-based number of VSA proxy agents and the stream-based number of additional VSA proxy agents.
2 . The computer-implemented method of claim 1 , further comprising initiating instantiation of additional VSA proxy agents based upon the allocation, wherein instantiation is performed automatically within a computing environment.
3 . The computer-implemented method of claim 1 , wherein the step of calculating the number of VSA proxy agents based on CPU count comprises multiplying a total CPU count available for processing the information management operation job by a predetermined number of data streams per CPU, wherein the predetermined number of data streams per CPU varies according to a corresponding type of operation being performed.
4 . The computer-implemented method of claim 1 , wherein the maximum number of data streams predefined for the information management operation job varies according to a hypervisor associated with the one or more virtual machines.
5 . The computer-implemented method of claim 1 , further comprising dynamically adjusting the number of allocated VSA proxy agents based on real-time monitoring of CPU utilization and data stream usage during performance of the information management operation job.
6 . The computer-implemented method of claim 1 , wherein determining the number of additional VSA proxy agents comprises rounding up to a next integer to ensure sufficient resources are provisioned for the information management operation job.
7 . The computer-implemented method of claim 1 , further comprising storing historical performance data associated with previously completed information management operation jobs, and utilizing the historical performance data to refine future calculations of required VSA proxy agents.
8 . The computer-implemented method of claim 1 , further comprising limiting the number of additional VSA proxy agents to not exceed at least one of:
a predefined policy threshold, or
a maximum number of streams allowed for a restore operation.
9 . A computer-implemented system for managing resources for performing information management operations, the computer-implemented system configured to:
receive an information management operation job associated with one or more virtual machines,
wherein the information management operation job comprises at least one of a backup or restore operation;
calculate, based on a Central Processing Unit (CPU) count, a CPU-based number of virtual server agent (VSA) proxy agents required to perform the information management operation job;
determine a difference between the calculated number of required VSA proxy agents and a currently existing number of VSA proxy agents;
determine a total number of available data streams across all currently available VSA proxy agents;
calculate a stream-based number of additional VSA proxy agents required by determining a difference between a total virtual machine count associated with the information management operation job and the total number of available data streams, and dividing the difference by a maximum number of data streams predefined for a corresponding type of information management operation job; and
allocate, for performance of the information management operation job,
a higher number of VSA proxy agents as determined by comparing the CPU-based number of VSA proxy agents and the stream-based number of additional VSA proxy agents.
10 . The computer-implemented system of claim 9 , the computer-implemented system further configured to initiate instantiation of additional VSA proxy agents based upon the allocation, wherein instantiation is performed automatically within a computing environment.
11 . The computer-implemented system of claim 9 , wherein the calculation of the number of VSA proxy agents based on CPU count comprises multiplying a total CPU count available for processing the information management operation job by a predetermined number of data streams per CPU,
wherein the predetermined number of data streams per CPU varies according to a corresponding type of operation being performed.
12 . The computer-implemented system of claim 9 , wherein the maximum number of data streams predefined for the information management operation job varies according to a hypervisor associated with the one or more virtual machines.
13 . The computer-implemented system of claim 9 , the computer-implemented system further configured to dynamically adjust the number of allocated VSA proxy agents based on real-time monitoring of CPU utilization and data stream usage during performance of the information management operation job.
14 . The computer-implemented system of claim 9 , wherein determining the number of additional VSA proxy agents comprises rounding up to a next integer to ensure sufficient resources are provisioned for the information management operation job.
15 . The computer-implemented system of claim 9 , the computer-implemented system further configured to store historical performance data associated with previously completed information management operation jobs, and utilizing the historical performance data to refine future calculations of required VSA proxy agents.
16 . The computer-implemented system of claim 9 , the computer-implemented system further configured to limit the number of additional VSA proxy agents to not exceed at least one of:
a predefined policy threshold, or
a maximum number of streams allowed for a restore operation.