Managing deduplication of data in storage systems

1. A method for use in managing deduplication of data in storage systems, the method comprising:

searching a rapid lookup digest segment for a digest of a data block, the rapid lookup digest segment including a set of digests associated with a set of data blocks previously received for deduplication;

adding the digest of the data block to the rapid lookup digest segment;

identifying, in an index table, a digest segment with at least one digest in common with the set of digests of the rapid lookup digest segment, prior to addition of the digest of the data block; and

overwriting the digest segment in the index table with the rapid lookup digest segment, the rapid lookup digest segment being associated with a data object for deduplication.

2. The method of claim 1 , wherein a set of data objects resides on a storage extent comprising a set of LUNs, wherein each LUN of the set of LUNs is a logical representation of a subset of a physical disk storage.

3. The method of claim 1 , wherein the data object includes the data block, wherein the data block is a fixed size chunk of physical disk storage.

4. The method of claim 1 , wherein the digest is based on contents of the data object that uniquely identifies the data object.

5. The method of claim 1 , wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data object that satisfies a sampling criterion, wherein each index entry of the set of index entries includes a digest of the data object and information regarding a digest segment associated with the data object.

6. The method of claim 1 , wherein the data object is deduplicated upon receiving a request to write data to the data object.

7. The method of claim 1 , wherein the rapid lookup digest segment is stored on a fast access memory.

8. The method of claim 1 , wherein the set of digests in the rapid lookup digest segment are organized in an ordered arrangement.

9. The method of claim 1 , further comprising:

evaluating digests stored in an index table upon not finding a matching digest in the rapid lookup digest segment; and

based on the evaluation, retrieving a digest segment from the index table, wherein the digest segment is associated with a digest stored in the index table that matches the digest associated with the data block.

10. The method of claim 1 , wherein applying the deduplicating technique further comprising:

updating an address mapping of the candidate data object to point to the data object, wherein the candidate data object and the data object contain identical content of data, wherein an address mapping indicates a location of a data object on a disk storage; and

deleting the content of the candidate data object from the disk storage.

11. A system for use in managing deduplication of data in storage systems, the system comprising a processor configured to:

search a rapid lookup digest segment for a digest of a data block, the rapid lookup digest segment including a set of digests associated with a set of data blocks previously received for deduplication;

add the digest of the data block to the rapid lookup digest segment;

identify, in an index table, a digest segment with at least one digest in common with the set of digests of the rapid lookup digest segment, prior to addition of the digest of the data block; and

overwrite the digest segment in the index table with the rapid lookup digest segment, the rapid lookup digest segment being associated with a data object for deduplication.

12. The system of claim 11 , wherein a set of data objects resides on a storage extent comprising a set of LUNs, wherein each LUN of the set of LUNs is a logical representation of a subset of a physical disk storage.

13. The system of claim 11 , wherein the data object includes the data block, wherein the data block is a fixed size chunk of physical disk storage.

14. The system of claim 11 , wherein the digest is based on contents of the data object that uniquely identifies the data object.

15. The system of claim 11 , wherein an index table comprises a set of index entries, wherein each index entry of the set of index entries is associated with a digest of a data object that satisfies a sampling criterion, wherein each index entry of the set of index entries includes a digest of the data object and information regarding a digest segment associated with the data object.

16. The system of claim 11 , wherein the data object is deduplicated upon receiving a request to write data to the data object.

17. The system of claim 11 , wherein the rapid lookup digest segment is stored on a fast access memory.

18. The system of claim 11 , wherein the set of digests in the rapid lookup digest segment are organized in an ordered arrangement.

19. The system of claim 11 , further comprising:

evaluate digests stored in an index table upon not finding a matching digest in the rapid lookup digest segment; and

based on the evaluation, retrieving a digest segment from the index table, wherein the digest segment is associated with a digest stored in the index table that matches the digest associated with the data block.

20. The system of claim 11 , wherein applying the deduplicating technique further comprising:

update an address mapping of the candidate data object to point to the data object, wherein the candidate data object and the data object contain identical content of data, wherein an address mapping indicates a location of a data object on a disk storage; and

delete the content of the candidate data object from the disk storage.