Intelligent scheduling for garbage collection
An intelligent method of scheduling garbage collection (GC) in a storage system. A GC scheduler obtains capacity utilization and ingest rate of the storage system and calculate therefrom a predicted capacity utilization. When the predicted capacity utilization reaches a threshold, the GC scheduler invokes GC, but otherwise skips GC until such time as predicted capacity utilization reaches the threshold. The ingest rage may be calculated by performing linear fit on past data ingest. The GC scheduler may calculate predicted capacity utilization periodically according to preset period. The GC scheduler may calculate the predicted capacity utilization to a future date beyond the next period. The future date may be at least as far as the next period plus total ingest time.
1. A computerized method for executing garbage collection in a backup system, the method comprising:
ingesting data for backup storage;
at each preset time period:
obtaining an ingestion rate of the ingesting data based on a slope of a most recent linear segment of an ingest graph;
obtaining a current system capacity indicating a current amount of storage space occupied by stored data;
using the ingestion rate and the current system capacity to estimate a prospective system capacity at a future time date equal to or longer than a sum of a preset time period and a garbage collection (GC) run time; and,
in response to the prospective system capacity at the future time date exceeds a preset threshold invoking a garbage collection (GC), in response to the prospective system capacity at the future time date doesn't exceed the preset threshold skipping the garbage collection until next preset time period.
2. The method of claim 1 , further comprising obtaining GC run time indicating time it takes to complete a GC run.
3. The method of claim 1 , wherein estimating the prospective system capacity comprises summing up the current system capacity and a product of the ingestion rate and number of days to the future time date.
4. The method of claim 1 , wherein obtaining the ingestion rate comprises calculating the ingestion rate for a past time span ending at current time.
5. The method of claim 1 , wherein obtaining the ingestion rate comprises fitting a line to a plot of an amount of data ingested over a past time span ending at current time and setting a slope of the line as the ingestion rate.
6. The method of claim 5 , wherein fitting a line comprises performing linear regression on an amount of data ingested over a past time span ending at current time.
7. The method of claim 6 , wherein performing linear regression comprises performing multiple linear regressions over multiple past time span ending at current time, wherein each of the multiple past time span has a different length.
8. The method of claim 7 , wherein the ingestion rate is determined using the linear regression resulting in highest r-squared value or the lowest sum of squared error.
9. A system comprising:
a plurality of storage disks;
a backup processor ingesting and storing data in containers within the storage disks;
a garbage collection module operable to clear dead memory space within the storage disks;
an ingestion rate calculator generating an ingestion rate of the backup processor based on a slope of a most recent linear segment of an ingest graph;
a garbage collection scheduler receiving the ingestion rate and invoking the garbage collection module by calculating a capacity prediction based upon the ingestion rate, wherein the garbage collection scheduler includes a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the garbage collection scheduler to
calculate the capacity prediction at a future time date equal to or longer than a sum of a preset time period and a garbage collection (GC) run time upon expiration of each preset period of time, and
invoke the garbage collection module in response to the capacity prediction at the future time date reaches a threshold, skip invoking the garbage collection module in response to the capacity prediction at the future time date is below the threshold.
10. The system of claim 9 , wherein the garbage collection scheduler is further to calculate the capacity prediction for a future date beyond next preset period of time.
11. The system of claim 9 , wherein the ingestion rate calculator includes a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the ingestion rate calculator to perform a linear fit on a total amount of data ingested by the backup processor over past time and determine the ingestion rate from the linear fit.
12. The system of claim 11 , wherein the ingestion rate calculator is further to perform the linear fit by linear regression over multiple segments of past times.
13. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code operating in a computing storage system and including instructions to:
obtain a current capacity utilization of the storage system;
obtain an ingestion rate of the storage system based on a slope of a most recent linear segment of an ingest graph;
at a start of each preprogrammed time period:
use the current capacity utilization and the ingestion rate to calculate a predicted capacity utilization at a future time date equal to or longer than a sum of a preset time period and a garbage collection (GC) run time at end of a run period and, in response to the predicted capacity utilization at the future time date reaches a threshold invoke a garbage collection, in response to the predicted capacity utilization at the future time date doesn't reach the threshold skip the garbage collection.
14. The computer program product of claim 13 , including instructions to define the run period at least as long as sum of the preprogrammed time period and max garbage collection time, wherein max garbage collection time defines the longest time for performing garbage collection on the storage system.
15. The computer program product of claim 13 , wherein the program code includes further instructions to obtain the ingestion rate by performing linear fit on a total amount of data ingested by the storage system over past time and determine the ingestion rate from the linear fit.
16. The computer program product of claim 15 , wherein the program code includes further instructions to perform the linear fit by linear regression over multiple segments of past times and selecting a linear fit having the least error.
17. The computer program product of claim 16 , wherein the program code includes further instructions to obtain the ingestion rate by calculating ingestion rate for a past time span ending at current time.
18. The computer program product of claim 13 , wherein the program code includes further instructions to calculate the predicted capacity utilization using a sum of the current capacity utilization and a product of the ingestion rate and the run period.