Tiering data to a cold storage tier of cloud object storage
Techniques for tiering data to a cold storage tier of a cloud object storage platform are provided. In one set of embodiments, a computer system can identify one or more old snapshots of a data set that reside in a first storage tier of the cloud object storage platform, where the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days. The computer system can further, for each snapshot in the one or more old snapshots: identify one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots; write the one or more data blocks to a second (i.e., cold) storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and cause the one or more data blocks to be deleted from the first storage tier.
1. A method comprising:
identifying, by a computer system, one or more old snapshots of a data set that reside in a first storage tier of a cloud object storage platform, wherein the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days, and wherein the period of N days corresponds to a minimum storage duration imposed on the second storage tier by a provider of the cloud object storage platform, the minimum storage duration being a minimum time period for which data should be maintained on the second storage tier prior to deletion in order to avoid an early deletion fee; and
for each snapshot in the one or more old snapshots:
identifying, by the computer system, one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots;
writing, by the computer system, the one or more data blocks to a second storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and
causing, by the computer system, the one or more data blocks to be deleted from the first storage tier.
2. The method of claim 1 wherein the first storage tier is a standard storage tier of the cloud object storage platform and wherein the second storage tier is a cold storage tier of the cloud object storage platform.
3. The method of claim 1 wherein the first storage tier is a first cold storage tier of the cloud object storage platform and wherein the second storage tier is a second cold storage tier of the cloud object storage platform.
4. The method of claim 1 wherein the data set is associated with a non-linear snapshot retention policy and wherein the one or more old snapshots are snapshots created at least N days ago.
5. The method of claim 1 wherein causing the one or more data blocks to be deleted from the first storage tier comprises:
updating one or more metadata structures to indicate that the one or more data blocks are no longer a part of, or referenced by, an existing snapshot in the first storage tier.
6. The method of claim 5 wherein upon updating the one or more metadata structures, the one or more data blocks are garbage collected from the first storage tier by a garbage collector component running in a cloud compute platform associated with the cloud object storage platform.
7. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method comprising:
identifying one or more old snapshots of a data set that reside in a first storage tier of a cloud object storage platform, wherein the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days, and wherein the period of N days corresponds to a minimum storage duration imposed on the second storage tier by a provider of the cloud object storage platform, the minimum storage duration being a minimum time period for which data should be maintained on the second storage tier prior to deletion in order to avoid an early deletion fee; and
for each snapshot in the one or more old snapshots:
identifying one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots;
writing the one or more data blocks to a second storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and
causing the one or more data blocks to be deleted from the first storage tier.
8. The non-transitory computer readable storage medium of claim 7 wherein the first storage tier is a standard storage tier of the cloud object storage platform and wherein the second storage tier is a cold storage tier of the cloud object storage platform.
9. The non-transitory computer readable storage medium of claim 7 wherein the first storage tier is a first cold storage tier of the cloud object storage platform and wherein the second storage tier is a second cold storage tier of the cloud object storage platform.
10. The non-transitory computer readable storage medium of claim 7 wherein the data set is associated with a non-linear snapshot retention policy and wherein the one or more old snapshots are snapshots created at least N days ago.
11. The non-transitory computer readable storage medium of claim 7 wherein causing the one or more data blocks to be deleted from the first storage tier comprises:
updating one or more metadata structures to indicate that the one or more data blocks are no longer a part of, or referenced by, an existing snapshot in the first storage tier.
12. The non-transitory computer readable storage medium of claim 11 wherein upon updating the one or more metadata structures, the one or more data blocks are garbage collected from the first storage tier by a garbage collector component running in a cloud compute platform associated with the cloud object storage platform.
13. A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
identify one or more old snapshots of a data set that reside in a first storage tier of a cloud object storage platform, wherein the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days, and wherein the period of N days corresponds to a minimum storage duration imposed on the second storage tier by a provider of the cloud object storage platform, the minimum storage duration being a minimum time period for which data should be maintained on the second storage tier prior to deletion in order to avoid an early deletion fee; and
for each snapshot in the one or more old snapshots:
identify one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots;
write the one or more data blocks to a second storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and
cause the one or more data blocks to be deleted from the first storage tier.
14. The computer system of claim 13 wherein the first storage tier is a standard storage tier of the cloud object storage platform and wherein the second storage tier is a cold storage tier of the cloud object storage platform.
15. The computer system of claim 13 wherein the first storage tier is a first cold storage tier of the cloud object storage platform and wherein the second storage tier is a second cold storage tier of the cloud object storage platform.
16. The computer system of claim 13 wherein the data set is associated with a non-linear snapshot retention policy and wherein the one or more old snapshots are snapshots created at least N days ago.
17. The computer system of claim 13 wherein causing the one or more data blocks to be deleted from the first storage tier comprises:
updating one or more metadata structures to indicate that the one or more data blocks are no longer a part of, or referenced by, an existing snapshot in the first storage tier.
18. The computer system of claim 17 wherein upon updating the one or more metadata structures, the one or more data blocks are garbage collected from the first storage tier by a garbage collector component running in a cloud compute platform associated with the cloud object storage platform.