Garbage collection based on cloud storage capacity
A storage capacity of a cloud storage system is determined. A recipe contains instructions for reclaiming storage of the cloud storage system. The storage capacity is insufficient to perform the entire recipe. In response, one or more data objects are deleted. A portion of the recipe is executed after the deletion. In some cases, local storage can be used for garbage collection when cloud storage is insufficient to perform even the portion of the recipe. Other embodiments are described and claimed.
1. A computer-implemented method, comprising:
determining a) a storage capacity of a cloud storage system, and b) a recipe containing instructions for reclaiming storage of the cloud storage system that is used by data objects;
in response to the storage capacity satisfying a threshold,
deleting, one or more of the data objects that are marked for deletion in the recipe; and
performing, after the deletion of the one or more of the data objects that are marked for deletion, a portion of one or more copy forward instructions of the recipe, the portion determined based on the storage capacity of the cloud storage system after the deletion of the data objects that are marked for deletion; and
in response to a) none of the data objects being marked for deletion in the recipe, and b) the storage capacity being insufficient for a single one of the one or more copy forward instructions, uploading new data objects containing live segments of the one or more data objects to the cloud storage system to replace the one or more of the data objects of the cloud storage system.
2. The method of claim 1 , further comprising performing one or more additional portions of the one or more copy forward instructions of the recipe, in successive operations.
3. The method of claim 2 , wherein each of the one or more additional portions of the one or more copy forward instructions require an increasing amount of storage capacity for the cloud storage system to perform.
4. The method of claim 2 , wherein, when it is determined that the storage capacity is sufficient, a remaining portion of the recipe is performed by in a final operation.
5. The method of claim 1 wherein generating the new data objects includes
downloading one or more of the data objects in a local storage, resulting in one or more local data objects with the live segments; and
performing the portion of the one or more copy forward instructions on the one or more local data objects to copy forward the live segments from the one or more local data objects to new data objects.
6. The method of claim 5 , further comprising, performing additional one or more portions of the one or more copy forward instructions of the recipe, after replacing the one or more of the data objects of the cloud storage system that were downloaded with the new data objects.
7. The method of claim 1 , wherein the method is performed by a garbage collector that is integrated in a data processing system that is separate from the cloud storage system.
8. The method of claim 1 , wherein the threshold is based on a difference between the storage capacity of the cloud storage system and a number of new data objects that would be created as a result of the one or more copy forward instructions of the recipe.
9. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of a networked computing device, cause the computing device to perform operations comprising:
determining a) a storage capacity of a cloud storage system, and b) a recipe containing instructions for reclaiming storage of the cloud storage system that is used by data objects;
in response to the storage capacity satisfying a threshold,
deleting, one or more of the data objects that are marked for deletion in the recipe; and
performing, after the deletion of the one or more of the data objects that are marked for deletion, a portion of one or more copy forward instructions of the recipe, the portion determined based on the storage capacity of the cloud storage system after the deletion of the data objects that are marked for deletion; and
in response to a) none of the data objects being marked for deletion in the recipe, and b) the storage capacity being insufficient for a single one of the one or more copy forward instructions, uploading new data objects containing live segments of the one or more data objects to the cloud storage system to replace the one or more of the data objects of the cloud storage system.
10. The non-transitory computer-readable medium of claim 9 , further comprising performing one or more additional portions of the one or more copy forward instructions of the recipe, in successive operations.
11. The non-transitory computer-readable medium of claim 10 , wherein each of the one or more additional portions of the one or more copy forward instructions require an increasing amount of storage capacity for the cloud storage system to perform.
12. The non-transitory computer-readable medium of claim 10 , wherein, when it is determined that the storage capacity is sufficient, a remaining portion of the recipe is performed by in a final operation.
13. The non-transitory computer-readable medium of claim 9 wherein generating the new data objects includes
downloading one or more of the data objects in a local storage, resulting in one or more local data objects with the live segments; and
performing the portion of the one or more copy forward instructions on the one or more local data objects to copy forward the live segments from the one or more local data objects to new data objects.
14. The non-transitory computer-readable medium of claim 13 , further comprising, performing additional one or more portions of the one or more copy forward instructions of the recipe, after replacing the one or more of the data objects of the cloud storage system that were downloaded with the new data objects.
15. A data processing system, comprising:
computer-readable memory; and
a processor that executes instructions stored on the computer-readable memory, causing the data processing system to perform operations including:
determining a) a storage capacity of a cloud storage system, and b) a recipe containing instructions for reclaiming storage of the cloud storage system that is used by data objects;
in response to the storage capacity satisfying a threshold,
deleting, one or more of the data objects that are marked for deletion in the recipe; and
performing, after the deletion of the one or more of the data objects that are marked for deletion, a portion of one or more copy forward instructions of the recipe, the portion determined based on the storage capacity of the cloud storage system after the deletion of the data objects that are marked for deletion; and
in response to a) none of the data objects being marked for deletion in the recipe, and b) the storage capacity being insufficient for a single one of the one or more copy forward instructions, uploading new data objects containing live segments of the one or more data objects to the cloud storage system to replace the one or more of the data objects of the cloud storage system.
16. The data processing system of claim 15 , further comprising performing one or more additional portions of the one or more copy forward instructions of the recipe, in successive operations.
17. The data processing system of claim 16 , wherein each of the one or more additional portions of the one or more copy forward instructions require an increasing amount of storage capacity for the cloud storage system to perform.
18. The data processing system of claim 16 , wherein, when it is determined that the storage capacity is sufficient, a remaining portion of the recipe is performed by in a final operation.
19. The data processing system of claim 15 , wherein generating the new data objects includes
downloading one or more of the data objects in a local storage, resulting in one or more local data objects with the live segments; and
performing the portion of the one or more copy forward instructions on the one or more local data objects to copy forward the live segments from the one or more local data objects to new data objects.
20. The data processing system of claim 19 , further comprising, performing additional one or more portions of the one or more copy forward instructions of the recipe, after replacing the one or more of the data objects of the cloud storage system that were downloaded with the new data objects.