Garbage collection for data storage
Methods, systems, apparatus, including computer programs encoded on computer storage media, for reclaiming storage space in a storage environment. In one aspect, the method includes actions of aggregating data that is indicative of access to one or more data objects, determining a future storage cost associated with each of a plurality of data objects, determining an access window for each of the plurality of data objects, identifying a data object based on (i) the future storage cost that satisfies a predetermined threshold and (ii) a data object access window, providing a notification to a user device that requests feedback from a user indicating whether the data object can be deleted, and in response to receiving data that indicates that the data object can be deleted, generating an instruction to cause deletion of the data object upon the expiration of the access window.
1. A computer-implemented method comprising:
aggregating, by one or more server computers, data that is descriptive of access to data objects stored in a storage environment;
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of data objects in the storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold;
for each identified first data object:
providing, by the one or more server computers, a notification to user device that requests feedback from a user indicating whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, data indicative of whether the data object can be deleted after expiration of the access window from the user device; and
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction to cause deletion of the data object upon the expiration of the access window.
2. The computer-implemented method of claim 1 , the method further comprising:
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction that instructs the storage environment to label the data object for deletion upon the expiration of the access window.
3. The computer-implemented method of claim 1 , the method further comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.
4. The computer-implemented method of claim 1 , wherein aggregating, by the one or more server computers, data that is descriptive of access to the data objects stored in the storage environment includes:
obtaining, by the one or more server computers, audit log data records from one or more audit logs maintained by the storage environment, wherein each audit log data record includes data that (i) identifies a data object that was accessed and (ii) a timestamp indicating when the access to the data object occurred; and
storing, by the one or more server computers, the obtained audit log data records in a database.
5. The computer-implemented method of claim 4 , wherein determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment includes:
analyzing, by the one or more server computers, the audit log data in the database to determine a period of time where the data object is likely to be accessed.
6. The computer-implemented method of claim 1 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment includes:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.
7. The computer-implemented method of claim 1 , further comprising:
sorting, by the one or more server computers, the one or more first data objects based on (i) the future storage cost that can be reclaimed for each respective data object, and (ii) a confidence score indicating whether or not each respective data object will be accessed outside of the access window.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by one or more computers, to cause the one or more computers to perform the operations comprising:
aggregating, by one or more server computers, data that is descriptive of access to data objects stored in a storage environment;
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of data objects in the storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold;
for each identified first data object:
providing, by the one or more server computers, a notification to user device that requests feedback from a user indicating whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, data indicative of whether the data object can be deleted after expiration of the access window from the user device; and
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction to cause deletion of the data object upon the expiration of the access window.
9. The system of claim 8 , the operations comprising:
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction that instructs the storage environment to label the data object for deletion upon the expiration of the access window.
10. The system of claim 8 , the operations comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.
11. The system of claim 8 , wherein aggregating, by the one or more server computers, data that is descriptive of access to the data objects stored in the storage environment includes:
obtaining, by the one or more server computers, audit log data records from one or more audit logs maintained by the storage environment, wherein each audit log data record includes data that (i) identifies a data object that was accessed and (ii) a timestamp indicating when the access to the data object occurred; and
storing, by the one or more server computers, the obtained audit log data records in a database.
12. The system of claim 11 , wherein determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment includes:
analyzing, by the one or more server computers, the audit log data in the database to determine a period of time where the data object is likely to be accessed.
13. The system of claim 8 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment includes:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.
14. The system of claim 8 , the operations further comprising:
sorting, by the one or more server computers, the one or more first data objects based on (i) the future storage cost that can be reclaimed for each respective data object, and (ii) a confidence score indicating whether or not each respective data object will be accessed outside of the access window.
15. A non-transitory computer-readable medium encoded with instructions that, when executed by or more computers, cause the one or more computers to perform the operations comprising:
aggregating, by one or more server computers, data that is descriptive of access to data objects stored in a storage environment;
determining, by the one or more server computers, a future storage cost associated with each data object of a plurality of data objects in the storage environment;
determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment, each access window having a respective expiration date in the future;
identifying, by the one or more server computers, one or more first data objects, each of the first data objects being a data object for which the determined future storage cost satisfies a predetermined future storage cost threshold; for each identified first data object:
providing, by the one or more server computers, a notification to user device that requests feedback from a user indicating whether the data object can be deleted after expiration of the access window;
receiving, by the one or more server computers, data indicative of whether the data object can be deleted after expiration of the access window from the user device;
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction to cause deletion of the data object for deletion upon the expiration of the access window.
16. The computer-readable medium of claim 15 , the operations comprising:
in response to receiving data that indicates that the data object can be deleted after expiration of the access window from the user device, generating an instruction that instructs the storage environment to label the data object for deletion upon the expiration of the access window.
17. The computer-readable medium of claim 15 , the operations comprising:
in response to receiving data that indicates that the data object cannot be deleted after expiration of the access window from the user device, adjusting heuristics used to determine the access window.
18. The computer-readable medium of claim 15 , wherein aggregating, by the one or more server computers, data that is descriptive of access to the data objects stored in the storage environment includes:
obtaining, by the one or more server computers, audit log data records from one or more audit logs maintained by the storage environment, wherein each audit log data record includes data that (i) identifies a data object that was accessed and (ii) a timestamp indicating when the access to the data object occurred; and
storing, by the one or more server computers, the obtained audit log data records in a database.
19. The computer-readable medium of claim 18 , wherein determining, by the one or more server computers, an access window for each data object of the plurality of data objects in the storage environment includes:
analyzing, by the one or more server computers, the audit log data in the database to determine a period of time where the data object is likely to be accessed.
20. The computer-readable medium of claim 15 , wherein determining, by the one or more server computers, a future storage cost associated with each data object of the plurality of data objects in the storage environment includes:
using one or more heuristics to estimate the potential future storage costs of the data object based on the predicted future depreciation of storage prices.