Dynamic throughput ingestion of backup sources
A method and apparatus for dynamically adjusting an ingestion rate for backup operations on a source system. The method generally includes monitoring a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload. Based on the monitored resource utilization, the backup system determines a data ingestion rate for backup operations on the source system. The backup system ingests data from the source system to a backup repository at the determined data ingestion rate.
1. A method, comprising:
monitoring a resource utilization related to one or more performance metrics of a source system in performing at least a primary workload, wherein a processor is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;
determining a data ingestion rate for backup operations on the source system based on the monitored resource utilization;
adjusting, by a backup engine of a backup system, the data ingestion rate from the source system, wherein the data ingestion rate from the source system is adjusted based on the one or more performance metrics of the source system in performing at least the primary workload, wherein a rate of change at which the data ingestion rate decreases as the primary workload at the source system increases is based on a priority of the primary workload with respect to data ingestion for backup operations;
ingesting data from the source system at the adjusted data ingestion rate; and
writing the ingested data to a distributed file system at a backup repository.
2. The method of claim 1 , wherein the data ingestion rate from the source system is adjusted based in part on one or more activity thresholds, wherein the one or more activity thresholds includes a high activity threshold, wherein the high activity threshold indicates a primary workload resource utilization at which the data ingestion rate is to be reduced.
3. The method of claim 2 , further comprising reducing the data ingestion rate from the source system in response to the primary workload resource utilization being greater than the high activity threshold.
4. The method of claim 1 , wherein the data ingestion rate from the source system is adjusted based in part on one or more activity thresholds, wherein the one or more activity thresholds includes a low activity threshold, wherein the low activity threshold indicates a primary workload resource utilization at which the data ingestion rate is to be increased.
5. The method of claim 4 , further comprising increasing the data ingestion rate from the source system in response to the primary workload resource utilization being less than the low activity threshold.
6. The method of claim 1 , further comprising determining whether the backup operations are completed.
7. The method of claim 6 , further comprising adjusting the data ingestion rate from the source system based on changes to the one or more performance metrics of the source system in performing at least the primary workload.
8. The method of claim 1 , wherein the one or more performance metrics of the source system in performing at least the primary workload include at least one or more of CPU usage, memory usage, or network throughput.
9. The method of claim 1 , wherein the source system is configured to host one or more virtual machines that share resources of the source system.
10. The method of claim 1 , wherein the source system is configured to provide one or more services.
11. The method of claim 1 , wherein the source system is configured to be an internet server.
12. The method of claim 1 , wherein the source system is configured to host one or more databases.
13. The method of claim 1 , wherein the source system is configured to provide a test environment for software updates.
14. The method of claim 1 , wherein the source system is configured to be a production system that is deploying a current version of software.
15. A non-transitory computer readable storage medium and comprising computer instructions for:
monitoring a resource utilization related to one or more performance metrics of a source system in performing at least a primary workload, wherein a processor is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;
determining a data ingestion rate for backup operations on the source system based on the monitored resource utilization;
adjusting, by a backup engine of a backup system, the data ingestion rate from the source system, wherein the data ingestion rate from the source system is adjusted based on the one or more performance metrics of the source system in performing at least the primary workload, wherein a rate of change at which the data ingestion rate decreases as the primary workload at the source system increases is based on a priority of the primary workload with respect to data ingestion for backup operations;
ingesting data from the source system at the adjusted data ingestion rate; and
writing the ingested data to a distributed file system at a storage repository.
16. The non-transitory computer readable storage medium of claim 15 , wherein the data ingestion rate from the source system is adjusted based in part on one or more activity thresholds, wherein the one or more activity thresholds includes a high activity threshold, wherein the high activity threshold indicates a primary workload resource utilization at which the data ingestion rate is to be reduced.
17. The non-transitory computer readable storage medium of claim 16 , further comprising instructions for reducing the data ingestion rate from the source system in response to the primary workload resource utilization being greater than the high activity threshold.
18. The non-transitory computer readable storage medium of claim 15 , wherein the data ingestion rate from the source system is adjusted based in part on one or more activity thresholds, wherein the one or more activity thresholds includes a low activity threshold, wherein the low activity threshold indicates a primary workload resource utilization at which the data ingestion rate is to be increased.
19. The non-transitory computer readable storage medium of claim 18 , further comprising instructions for increasing the data ingestion rate from the source system in response to the primary workload resource utilization being less than the low activity threshold.
20. A system, comprising:
a processor configured to:
monitor a resource utilization related to one or more performance metrics of a source system in performing at least a primary workload, wherein the processor is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;
determine a data ingestion rate for backup operations on the source system based on the monitored resource utilization;
adjust, by a backup engine of a backup system, the data ingestion rate from the source system, wherein the data ingestion rate from the source system is adjusted based on the one or more performance metrics of the source system in performing at least the primary workload and one or more activity thresholds, wherein a rate of change at which the data ingestion rate decreases as the primary workload at the source system increases is based on a priority of the primary workload with respect to data ingestion for backup operations;
ingest data from the source system at the adjusted data ingestion rate; and
write the ingested data to a distributed file system at a storage repository; and
a memory coupled to the processor and configured to provide the processor with instructions.