Managing deduplication of data in storage systems
A method is used in managing deduplication of data in storage systems. A digest is determined for a data object received for deduplication. A candidate data object is identified for deduplicating the data object. A digest associated with the candidate data object matches the digest associated with the data object. The digest in a digest segment is maintained based on identification of the candidate data object. The digest segment includes a set of digests associated with a set of data blocks identified for deduplication in an ordered arrangement.
1. A method for use in managing deduplication of data in storage systems, the method comprising:
determining a digest for a data object received for deduplication;
determining whether a candidate data object exists for deduplicating the data object by searching for a digest, associated with the candidate data object, that matches the digest associated with the data object;
upon determining that no candidate data object exist for deduplicating the data object, adding the data object to a data segment in the same order in which the data object has been received for performing deduplication to maintain locality of the data object;
upon determining that no candidate data object exist for deduplicating the data object, adding the digest to a digest segment, wherein the digest segment includes a set of digests associated with a set of data blocks identified for deduplication as a group, wherein the set of digests are stored in the digest segment in an ordered arrangement; and
upon determining that the candidate data object exists, performing deduplication of the data object.
2. The method of claim 1 , wherein a set of data objects resides on a storage extent comprising a set of LUNs, wherein each LUN of the set of LUNs is a logical representation of a subset of a physical disk storage.
3. The method of claim 1 , wherein a data object includes a data block, wherein a data block is a fixed size chunk of physical disk storage.
4. The method of claim 1 , wherein the digest is based on contents of the data object that uniquely identifies the data object.
5. The method of claim 1 , wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data object that satisfies a sampling criterion, wherein each index entry of the set of index entries includes a digest of the data object and information regarding a digest segment associated with the data object.
6. The method of claim 1 , wherein the data object is deduplicated upon receiving a request to write data to the data object.
7. The method of claim 1 , wherein the digest segment is stored on a fast access memory.
8. The method of claim 1 , wherein determining whether the candidate data object exists for deduplicating the data object further comprising:
evaluating digests stored in a rapid lookup digest segment to determine whether another digest matching the digest associated with the data block is stored in the rapid lookup digest segment, wherein the rapid lookup digest segment is a current digest segment organized in a memory of a storage system;
based on the evaluation, evaluating digests stored in an index table; and
based on the evaluation, retrieving a digest segment from the index table, wherein the digest segment is associated with a digest stored in the index table that matches the digest associated with the data block.
9. The method of claim 8 , further comprising:
applying a deduplicating technique to the data object; and
updating the index table with information from the rapid lookup digest segment.
10. The method of claim 9 , wherein applying the deduplicating technique further comprising:
updating an address mapping of the candidate data object to point to the data object, wherein the candidate data object and the data object contain identical content of data, wherein an address mapping indicates a location of a data object on a disk storage; and
deleting the content of the candidate data object from the disk storage.
11. A system for use in managing deduplication of data in storage systems, the system comprising a processor configured to:
determine a digest for a data object received for deduplication;
determine whether a candidate data object exists for deduplicating the data object by searching for a digest, associated with the candidate data object, that matches the digest associated with the data object;
upon determining that no candidate data object exist for deduplicating the data object, add the data object to a data segment in the same order in which the data object has been received for performing deduplication to maintain locality of the data object;
upon determining that no candidate data object exist for deduplicating the data object, add the digest to a digest segment based on identification of the candidate data object, wherein the digest segment includes a set of digests associated with a set of data blocks, identified for deduplication as a group, in an ordered arrangement; and
upon determining that the candidate data object exists, perform deduplication of the data object.
12. The system of claim 11 , wherein a set of data objects resides on a storage extent comprising a set of LUNs, wherein each LUN of the set of LUNs is a logical representation of a subset of a physical disk storage.
13. The system of claim 11 , wherein a data object includes a data block, wherein a data block is a fixed size chunk of physical disk storage.
14. The system of claim 11 , wherein the digest is based on contents of the data object that uniquely identifies the data object.
15. The system of claim 11 , wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data object that satisfies a sampling criterion, wherein each index entry of the set of index entries includes a digest of the data object and information regarding a digest segment associated with the data object.
16. The system of claim 11 , wherein the data object is deduplicated upon receiving a request to write data to the data object.
17. The system of claim 11 , wherein the digest segment is stored on a fast access memory.
18. The system of claim 11 , wherein determining whether the candidate data object exists for deduplicating the data object further comprising:
evaluate digests stored in a rapid lookup digest segment to determine whether another digest matching the digest associated with the data block is stored in the rapid lookup digest segment, wherein the rapid lookup digest segment is a current digest segment organized in a memory of a storage system;
based on the evaluation, evaluate digests stored in an index table; and
based on the evaluation, retrieve a digest segment from the index table, wherein the digest segment is associated with a digest stored in the index table that matches the digest associated with the data block.
19. The system of claim 18 , further comprising:
apply a deduplicating technique to the data object; and
update the index table with information from the rapid lookup digest segment.
20. The system of claim 19 , wherein applying the deduplicating technique further comprising:
update an address mapping of the candidate data object to point to the data object, wherein the candidate data object and the data object contain identical content of data, wherein an address mapping indicates a location of a data object on a disk storage; and
delete the content of the candidate data object from the disk storage.