Optimizing garbage collection based on survivor lifetime prediction
A predictive method for scheduling of the operations is described. The predictive method utilizes data generated from computing an expected lifetime of the individual files or objects within the container. The expected lifetime of individual files or objects can be generated based on machine learning techniques. Operations such as garbage collection are scheduled at an epoch where computational efficiencies are realized for performing the operation.
1. A method of determining whether to perform a garbage collection operation comprising:
identifying a container, wherein the container comprises one or more discrete objects;
determining a deletion predictor related to the container at a first epoch;
computing a cost of garbage collection for the container at the first epoch;
estimating a benefit of performing garbage collection for the container at the first epoch, the benefit comprising an amount of freed storage, a reduction in overhead of maintaining metadata and tracking deleted files, and a reduction in fragmentation of the container;
determining a net benefit of performing garbage collection for the container at the first epoch based on the computed cost and the estimated benefit.
2. The method of claim 1 , wherein determining the deletion predictor comprises performing a machine learning classification algorithm to generate a binary deletion predictor.
3. The method of claim 1 , wherein determining the deletion predictor comprises performing a machine learning regression algorithm to generate a probability.
4. The method of claim 3 , wherein the machine learning regression algorithm is a neural network.
5. The method of claim 1 , comprising generating a user notification regarding the garbage collection.
6. The method of claim 5 , wherein the user notification includes a request for a user to approve a garbage collection.
7. The method of claim 5 , wherein the user notification includes information related to a schedule of garbage collection.
8. The method of claim 5 , wherein the user notification includes information indicating the expected benefit of the garbage collection.
9. The method of claim 1 , comprising scheduling a garbage collection at the first epoch when the net benefit is positive.
10. The method of claim 9 , comprising determining the deletion predictor related to the container for a pre-determined number of epochs following the first epoch.
11. The method of claim 10 , comprising generating a net benefit for the container at each epoch of the pre-determined number of epochs following the first epoch.
12. The method of claim 11 , comprising:
identifying the epoch at which the net benefit is the greatest; and
scheduling a garbage collection at the epoch at which the net benefit is the greatest.
13. The method of claim 11 , comprising scheduling a garbage collection at the earliest epoch from the pre-determined number of next epochs where the net benefit is above a pre-determined threshold.
14. The method of claim 11 , comprising generating an additional net benefit for an additional epoch when within a pre-determined range of the earliest epoch following the first epoch.
15. The method of claim 11 , comprising re-evaluating a metric related to the identified container upon the occurrence of a trigger event.
16. The method of claim 15 , wherein the trigger event is a user command to perform a garbage collection on the identified container.
17. The method of claim 11 , comprising re-evaluating a metric related to the identified container at the start of any epoch after the first epoch.
18. A system configured to schedule garbage collection, the system comprising:
one or more non-volatile storage medium wherein each non-volatile storage medium contains one or more containers of data, each sector comprising one or more discrete objects;
a non-transitory computer readable storage medium containing instructions, the instructions when executed configured to perform the steps of:
identifying a container;
determining a deletion predictor related to the container at a first epoch;
computing a cost of garbage collection for the container at the first epoch;
estimating a benefit of performing garbage collection for the container at the first epoch based on at least the determined deletion predictor, the benefit comprising an amount of freed storage, a reduction in overhead of maintaining metadata and tracking deleted files, and a reduction in fragmentation of the container;
determining a net benefit of performing garbage collection for the container at the first epoch based on the computed cost and the estimated benefit.
19. The system of claim 18 comprising the instructions configured to generate a notification for a user.
20. A non-transitory computer readable storage medium containing instructions, the instructions when executed configured to perform the steps of:
identifying a container;
determining a deletion predictor related to the container at a first epoch;
computing a cost of garbage collection for the container at the first epoch;
estimating a benefit of performing garbage collection for the container at the first epoch based on at least the determined deletion predictor related to the container, the benefit comprising an amount of freed storage, a reduction in overhead of maintaining metadata and tracking deleted files, and a reduction in fragmentation of the container; and
determining a net benefit of performing garbage collection for the container at the first epoch based on the computed cost and the estimated benefit.