Techniques for data deduplication

1. A method of processing data comprising:

receiving, using a processor, a first data block stored at a first logical address in a first data set, wherein a first hash value is derived from the first data block;

determining, using a processor and in accordance with one or more selection criteria, whether the first hash value is included in a subset of hash values;

responsive to determining the first hash value is included in the subset, performing first processing using a processor, wherein said first processing includes updating a table in accordance with the first hash value and the first data set, wherein the table includes information about hash values in the subset, and wherein the table includes a first entry for the first hash value and includes first information, wherein the first information identifies one or more data sets that reference the first data block and, for each of the one or more data sets of the first information, specifies a reference count denoting a number of times the first data block is referenced by said each data set;

determining, using a processor and in accordance with the information in the table, to perform deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block, wherein said determining to perform deduplication processing for the first data block further includes:

determining, using the first information of the table associated with the first entry for the first hash value, a first data deduplication percentage for the first data set including the first data block, wherein the first data deduplication percentage denotes a percentage of data blocks of the first data set which have corresponding hash values in the subset and which have been deduplicated; and

determining that the first data deduplication percentage is greater than a threshold;

responsive to determining the first data deduplication percentage is greater than the threshold, performing said deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block included in a cached portion of a data deduplication store, wherein the cached portion of the data deduplication store is stored in a cache of a data storage system;

determining, using a processor and in accordance with said deduplication processing for the first data block, whether the first data block is a duplicate of the another stored data block included in the cached portion of the data deduplication store;

responsive to determining the first data block is a duplicate of the another stored data block included in the cached portion, storing, using a processor, the first data block as a deduplicated data block that references the another stored data block, and otherwise, storing the first data block as a new unique data block; and

responsive to determining the first data deduplication percentage is greater than the threshold, storing, using a processor, a corresponding entry for the first data block and the first hash value in the cache including the cached portion of the data deduplication store, and otherwise not storing the corresponding entry in the cache including the cached portion of the data deduplication data store.

2. The method of claim 1 , further comprising:

determining, in accordance with the information in the table, whether to store the corresponding entry in one of a plurality of a cached portions of the deduplication data store, wherein each of the plurality of cached portions is stored in a different one of a plurality of caching levels of the cache of the data storage system, wherein the cached portion is included in the plurality of cached portions, wherein the cached portion is stored in a first portion of the cache, and wherein the first portion of the cache corresponds to a first of the plurality of caching levels.

3. The method of claim 1 , wherein each of the first data set and the one or more data sets is a logically defined storage entity.

4. The method of claim 3 , wherein the logically defined storage entity includes any of: one or more logical devices, a portion of a logical device, a plurality of portions of a plurality of logical devices, one or more files, one or more file systems, and one or more directories.

5. The method of claim 1 , wherein the first processing further updates the table in accordance with the first logical address at which the first data block is stored in the first data set.

6. The method of claim 5 , wherein the table includes a second entry for a second hash value and includes second information, wherein the second hash value is derived from a second data block, and wherein the second information identifies one or more data sets that reference the second data block and, for each of the one or more data sets of the second information, specifies a reference count denoting a number of times the second data block is referenced by said each data set of the second information.

7. The method of claim 6 , wherein the second data block is stored at a second logical address in a second data set and is also stored at a third logical address of a third data set, wherein the second information further includes the second logical address and denotes that the second logical address of the second data set references the second data block from which the second hash value is derived, and wherein the second information further includes the third logical address and denotes that third logical address of the third data set references the second data block from which the second hash value is derived.

8. The method of claim 7 , wherein the first data block is stored at a fourth logical address of a fourth data set, wherein the first information further includes the fourth logical address and denotes that the fourth logical address of the fourth data set references the first data block from which the first hash value is derived.

9. The method of claim 8 , further comprising:

determining a first distance between the first logical address and the fourth logical address of the first entry;

determining a second distance between the second logical address and the third logical address of the second entry;

determining whether the first distance equals the second distance; and

responsive to determining the first distance equals the second distance, performing deduplication processing to locate duplicate data blocks in the first data set and the second data set.

10. The method of claim 1 , wherein the method is performed inline as part of an I/O or data path.

11. The method of claim 1 , wherein the method is performed offline not as part of an I/O or data path.

12. A system comprising:

one or more processors; and

one or more memories comprising code stored thereon that, when executed, performs a method of processing data comprising:

receiving, using a processor, a first data block stored at a first logical address in a first data set, wherein a first hash value is derived from the first data block;

determining, using a processor and in accordance with one or more selection criteria, whether the first hash value is included in a subset of hash values;

responsive to determining the first hash value is included in the subset, performing first processing using a processor, wherein said first processing includes updating a table in accordance with the first hash value and the first data set, wherein the table includes information about hash values in the subset, and wherein the table includes a first entry for the first hash value and includes first information, wherein the first information identifies one or more data sets that reference the first data block and, for each of the one or more data sets of the first information, specifies a reference count denoting a number of times the first data block is referenced by said each data set;

determining, using a processor and in accordance with the information in the table, to perform deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block, wherein said determining to perform deduplication processing for the first data block further includes:

determining, using the first information of the table associated with the first entry for the first hash value, a first data deduplication percentage for the first data set including the first data block, wherein the first data deduplication percentage denotes a percentage of data blocks of the first data set which have corresponding hash values in the subset and which have been deduplicated; and

determining that the first data deduplication percentage is greater than a threshold; and

responsive to determining the first data deduplication percentage is greater than the threshold, performing said deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block included in a cached portion of a data deduplication store, wherein the cached portion of the data deduplication store is stored in a cache of a data storage system;

determining, using a processor and in accordance with said deduplication processing for the first data block, whether the first data block is a duplicate of the another stored data block included in the cached portion of the data deduplication store;

responsive to determining the first data block is a duplicate of the another stored data block included in the cached portion, storing, using a processor, the first data block as a deduplicated data block that references the another stored data block, and otherwise, storing the first data block as a new unique data block; and

responsive to determining the first data deduplication percentage is greater than the threshold, storing, using a processor, a corresponding entry for the first data block and the first hash value in the cache including the cached portion of the data deduplication store, and otherwise not storing the corresponding entry in the cache including the cached portion of the data deduplication data store.

13. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of processing data comprising:

receiving, using a processor, a first data block stored at a first logical address in a first data set, wherein a first hash value is derived from the first data block;

determining, using a processor and in accordance with one or more selection criteria, whether the first hash value is included in a subset of hash values;

responsive to determining the first hash value is included in the subset, performing first processing using a processor, wherein said first processing includes updating a table in accordance with the first hash value and the first data set, wherein the table includes information about hash values in the subset, and wherein the table includes a first entry for the first hash value and includes first information, wherein the first information identifies one or more data sets that reference the first data block and, for each of the one or more data sets of the first information, specifies a reference count denoting a number of times the first data block is referenced by said each data set;

determining, using a processor and in accordance with the information in the table, to perform deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block, wherein said determining to perform deduplication processing for the first data block further includes:

determining, using the first information of the table associated with the first entry for the first hash value, a first data deduplication percentage for the first data set including the first data block, wherein the first data deduplication percentage denotes a percentage of data blocks of the first data set which have corresponding hash values in the subset and which have been deduplicated; and

determining that the first data deduplication percentage is greater than a threshold;

responsive to determining the first data deduplication percentage is greater than the threshold, performing said deduplication processing for the first data block to determine whether the first data block is a duplicate of another stored data block included in a cached portion of a data deduplication store, wherein the cached portion of the data deduplication store is stored in a cache of a data storage system;

determining, using a processor and in accordance with said deduplication processing for the first data block, whether the first data block is a duplicate of the another stored data block included in the cached portion of the data deduplication store;

responsive to determining the first data block is a duplicate of the another stored data block included in the cached portion, storing, using a processor, the first data block as a deduplicated data block that references the another stored data block, and otherwise, storing the first data block as a new unique data block; and

responsive to determining the first data deduplication percentage is greater than the threshold, storing, using a processor, a corresponding entry for the first data block and the first hash value in the cache including the cached portion of the data deduplication store, and otherwise not storing the corresponding entry in the cache including the cached portion of the data deduplication data store.