Fine-grained shared multi-tenant de-duplication system
In one example, a method includes receiving, at a cloud storage site, chunks that each take the form of a hash of a combination that includes two or more salts and a file object, and one of the salts is a retention salt shared by the chunks, monitoring a time period associated with the retention salt, when the time period has expired, removing the chunks that include the retention salt, and depositing the removed chunks in a deleted items cloud store.
1. A method, comprising:
receiving, at a cloud storage site, chunks that each take the form of a hash of a combination that comprises a plurality of salts and a file object, and one of the salts is a retention salt shared by the chunks, and another of the salts is a user-specific salt;
monitoring a time period associated with the retention salt;
when the time period has expired, removing the chunks that include the retention salt;
depositing the removed chunks in a deleted items cloud store;
keeping a running total of chunk sizes with which the user-specific salt is associated, and the running total accounts for both ingestion of data chunks and the removed chunks; and
based on the running total, determining an amount of backend storage capacity consumed by objects associated with the user.
2. The method as recited in claim 1 , wherein receiving the chunks comprises receiving an object that includes the chunks, and removing and depositing the chunks comprises removing the object and depositing the object in the deleted items cloud store.
3. The method as recited in claim 1 , wherein all the chunks are received from a common container of a local premises site that is remote from the cloud storage site.
4. The method as recited in claim 1 , wherein removing and depositing the chunks is performed without the use of a copy-forward garbage collection process.
5. The method as recited in claim 1 , wherein the chunks are received from a deduplication system.
6. The method as recited in claim 1 , wherein the chunks are received at the cloud storage site as part of a tiering process in which the chunks are stored at the cloud storage site according to a particular priority of the chunks relative to other chunks that are, or will be, stored at the cloud storage site.
7. The method as recited in claim 1 , wherein the chunks are stored in a content salted deduplication domain at the cloud storage site.
8. The method as recited in claim 1 , wherein when received at the cloud storage site, the chunks are contained in a container and are then stored in an object store of the cloud storage site.
9. The method as recited in claim 1 , wherein the operations further comprise scrubbing chunks stored at the cloud storage site.
10. A non-transitory storage medium having stored therein instructions executable by one or more hardware processors to perform operations comprising:
receiving, at a cloud storage site, chunks that each take the form of a hash of a combination that comprises a plurality of salts and a file object, and one of the salts is a retention salt shared by the chunks, and another of the salts is a user-specific salt;
monitoring a time period associated with the retention salt;
when the time period has expired, removing the chunks that include the retention salt;
depositing the removed chunks in a deleted items cloud store;
identifying live objects associated with the user-specific salt by traversing an object namespace associated with the user; and
determining an amount of backend storage capacity consumed by live objects by totaling a size of live data chunks in the object namespace.
11. The non-transitory storage medium as recited in claim 10 , wherein receiving the chunks comprises receiving an object that includes the chunks, and removing and depositing the chunks comprises removing the object and depositing the object in the deleted items cloud store.
12. The non-transitory storage medium as recited in claim 10 , wherein all the chunks are received from a common container of a local premises site that is remote from the cloud storage site.
13. The non-transitory storage medium as recited in claim 10 , wherein one of the salts is a user-specific salt.
14. The non-transitory storage medium as recited in claim 10 , wherein removing and depositing the chunks is performed without the use of a copy-forward garbage collection process.
15. The non-transitory storage medium as recited in claim 10 , wherein the chunks are received from a deduplication system.
16. The non-transitory storage medium as recited in claim 10 , wherein the chunks are received at the cloud storage site as part of a tiering process in which the chunks are stored at the cloud storage site according to a particular priority of the chunks relative to other chunks that are, or will be, stored at the cloud storage site.
17. The non-transitory storage medium as recited in claim 10 , wherein the chunks are stored in a content salted deduplication domain at the cloud storage site.
18. The non-transitory storage medium as recited in claim 10 , wherein when received at the cloud storage site, the chunks are contained in a container and are then stored in an object store of the cloud storage site.
19. The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise scrubbing chunks stored at the cloud storage site.
20. A non-transitory storage medium having stored therein instructions executable by one or more hardware processors to perform operations comprising:
receiving, at a cloud storage site, chunks that each take the form of a hash of a combination that comprises a plurality of salts and a file object, and one of the salts is a retention salt shared by the chunks, and another of the salts is a user-specific salt;
monitoring a time period associated with the retention salt;
when the time period has expired, removing the chunks that include the retention salt;
depositing the removed chunks in a deleted items cloud store;
traversing physical data chunks associated with the user; and
determining an amount of backend storage capacity consumed by live objects associated with the user by totaling a size of live physical data chunks identified as a result of the traversing.