Garbage collection for data storage
Methods, systems, apparatus, including computer programs encoded on computer storage media, for reclaiming storage space in a storage environment. In one aspect, the method includes actions of aggregating data that is indicative of access to one or more data objects, determining a future storage cost associated with each of a plurality of data objects, determining an access window for each of the plurality of data objects, identifying a data object based on (i) the future storage cost that satisfies a predetermined threshold and (ii) a data object access window, providing a notification to a user device that requests feedback from a user indicating whether the data object can be deleted, and in response to receiving data that indicates that the data object can be deleted, generating an instruction to cause deletion of the data object upon the expiration of the access window.
1. A computer-implemented method comprising:
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of storage objects in a storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold; and
for each identified first data object:
providing, by the one or more server computers and to a computer system hosting a computing agent, first data that prompts the computing agent to determine whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, second data, from the computer system hosting the computing agent, that is indicative of whether the data object can be deleted after expiration of the access window; and
in response to receiving, by the one or more server computers, the second data that indicates that the data object can be deleted after expiration of the access window from the computer system hosting the computing agent, generating an instruction to cause deletion of the data object upon the expiration of the access window.
2. The computer-implemented method of claim 1 , wherein the computing agent comprises one or more software modules or one or more hardware modules.
3. The computer-implemented method of claim 1 , the method further comprising:
determining, by the computer system that hosts the computing agent, a confidence score for each data object of the first data objects, wherein the confidence score indicates a likelihood that the data object will be accessed after the access window for the data object expires.
4. The computer-implemented method of claim 3 , the method further comprising:
determining, by the computing agent that is hosted by the computer system, whether the data object can be deleted after expiration of the access window based on the confidence score.
5. The computer-implemented method of claim 1 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment comprises:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.
6. The computer-implemented method of claim 1 , the method further comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.
7. The computer-implemented method of claim 1 ,
the method further comprising:
obtaining, by the one or more server computers, audit log data records from one or more audit logs maintained by the storage environment, wherein each audit log data record includes data that (i) identifies a data object that was accessed and (ii) a timestamp indicating when the access to the data object occurred, and
wherein determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment comprises:
analyzing, by the one or more server computers, one or more of the obtained audit log data records to determine a period of time where the data object is likely to be accessed.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of storage objects in a storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold; and
for each identified first data object:
providing, by the one or more server computers and to a computer system hosting a computing agent, first data that prompts the computing agent to determine whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, second data, from the computer system hosting the computing agent, that is indicative of whether the data object can be deleted after expiration of the access window; and
in response to receiving, by the one or more server computers, the second data that indicates that the data object can be deleted after expiration of the access window from the computer system hosting the computing agent, generating an instruction to cause deletion of the data object upon the expiration of the access window.
9. The system of claim 8 , wherein the computing agent comprises one or more software modules or one or more hardware modules.
10. The system of claim 8 , the operations further comprising:
determining, by the computer system that hosts the computing agent, a confidence score for each data object of the first data objects, wherein the confidence score indicates a likelihood that the data object will be accessed after the access window for the data object expires.
11. The system of claim 10 , the operations further comprising:
determining, by the computing agent that is hosted by the computer system, whether the data object can be deleted after expiration of the access window based on the confidence score.
12. The system of claim 8 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment comprises:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.
13. The system of claim 8 , the operations further comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.
14. The system of claim 8 ,
the operations further comprising:
obtaining, by the one or more server computers, audit log data records from one or more audit logs maintained by the storage environment, wherein each audit log data record includes data that (i) identifies a data object that was accessed and (ii) a timestamp indicating when the access to the data object occurred, and
wherein determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment comprises:
analyzing, by the one or more server computers, one or more of the obtained audit log data records to determine a period of time where the data object is likely to be accessed.
15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of storage objects in a storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold; and
for each identified first data object:
providing, by the one or more server computers and to a computer system hosting a computing agent, first data that prompts the computing agent to determine whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, second data, from the computer system hosting the computing agent, that is indicative of whether the data object can be deleted after expiration of the access window; and
in response to receiving, by the one or more server computers, the second data that indicates that the data object can be deleted after expiration of the access window from the computer system hosting the computing agent, generating an instruction to cause deletion of the data object upon the expiration of the access window.
16. The computer-readable medium of claim 15 , wherein the computing agent comprises one or more software modules or one or more hardware modules.
17. The computer-readable medium of claim 15 , the operations further comprising:
determining, by the computer system that hosts the computing agent, a confidence score for each data object of the first data objects, wherein the confidence score indicates a likelihood that the data object will be accessed after the access window for the data object expires.
18. The computer-readable medium of claim 17 , the operations further comprising:
determining, by the computing agent that is hosted by the computer system, whether the data object can be deleted after expiration of the access window based on the confidence score.
19. The computer-readable medium of claim 15 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment comprises:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.
20. The computer-readable medium of claim 15 , the operations further comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.