Elective deduplication

1. A method for electing deduplication in a storage system, the method comprising:

calculating a similarity hash signature for a data unit of a storage system;

searching a digest table of the storage system for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit;

using the search to determine whether to add a similarity hash signature or a strong hash signature of the data unit to the digest table; and

deduplicating the data unit from a storage device of the storage system based on the determination of whether the similarity hash signature or the strong hash signature is added to the digest table.

2. The method of claim 1 , wherein determining whether to add the similarity hash signature or the strong hash signature of the data unit to the digest table comprises:

adding the similarity hash signature of the data unit to the digest table if no similarity hash signature in the digest table falls within the predetermined distance of the similarity hash signature for the data unit.

3. The method of claim 1 , wherein determining whether to add the similarity hash signature or the strong hash signature of the data unit to the digest table comprises:

adding the strong hash signature of the data unit to the digest table if the digest table includes a similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.

4. The method of claim 1 , wherein determining whether to add the similarity hash signature or the strong hash signature of the data unit to the digest table comprises:

calculating a strong hash signature for the data unit; and

searching the digest table for a strong hash signature that matches the strong hash signature for the data unit.

5. The method of claim 4 , wherein determining whether to add the similarity hash signature or the strong hash signature of the data unit to the digest table comprises:

if the digest table includes a match of the strong hash signature, deduplicating the data unit based on the strong hash signature.

6. The method of claim 4 , wherein determining whether to add the similarity hash signature or the strong hash signature of the data unit to the digest table comprises:

if the digest table does not include a match of the strong hash signature, deduplicating the data unit based on the similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.

7. A system for electing deduplication in a storage system, the system including a processor configured to:

calculate a similarity hash signature for a data unit of a storage system;

search a digest table of the storage system for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit;

use the search to determine whether to add a similarity hash signature or a strong hash signature of the data unit to the digest table; and

deduplicate the data unit from a storage device of the storage system based on the determination of whether the similarity hash signature or the strong hash signature is added to the digest table.

8. The system of claim 7 , wherein the processor is further configured to:

add the similarity hash signature of the data unit to the digest table if no similarity hash signature in the digest table falls within the predetermined distance of the similarity hash signature for the data unit.

9. The system of claim 7 , wherein the processor is further configured to:

add the strong hash signature of the data unit to the digest table if the digest table includes a similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.

10. The system of claim 7 , wherein the processor is further configured to:

calculate a strong hash signature for the data unit; and

search the digest table for a strong hash signature that matches the strong hash signature for the data unit.

11. The system of claim 10 , wherein the processor is further configured to:

if the digest table includes a match of the strong hash signature, deduplicating the data unit based on the strong hash signature.

12. The system of claim 10 , wherein the processor is further configured to:

if the digest table does not include a match of the strong hash signature, deduplicating the data unit based on the similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.

13. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of coordinating inline deduplication among nodes in a storage system comprising:

calculating a similarity hash signature for a data unit of a storage system;

searching a digest table of the storage system for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit;

using the search to determine whether to add a similarity hash signature or a strong hash signature of the data unit to the digest table; and

deduplicating the data unit from a storage device of the storage system based on the determination of whether the similarity hash signature or the strong hash signature is added to the digest table.

14. The non-transitory computer readable medium of claim 13 , further comprising code that, when executed, performs:

adding the similarity hash signature of the data unit to the digest table if no similarity hash signature in the digest table falls within the predetermined distance of the similarity hash signature for the data unit.

15. The non-transitory computer readable medium of claim 13 , further comprising code that, when executed, performs:

adding the strong hash signature of the data unit to the digest table if the digest table includes a similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.

16. The non-transitory computer readable medium of claim 13 , further comprising code that, when executed, performs:

calculating a strong hash signature for the data unit; and

searching the digest table for a strong hash signature that matches the strong hash signature for the data unit.

17. The non-transitory computer readable medium of claim 16 , further comprising code that, when executed, performs:

if the digest table includes a match of the strong hash signature, deduplicating the data unit based on the strong hash signature.

18. The non-transitory computer readable medium of claim 16 , further comprising code that, when executed, performs:

if the digest table does not include a match of the strong hash signature, deduplicating the data unit based on the similarity hash signature within the predetermined distance of the similarity hash signature for the data unit.