Data de-duplication for information storage systems

1. A method for identifying candidates for de-duplication in a data storage system, comprising:

maintaining at least one de-duplication data structure, the de-duplication data structure including a plurality of entries of B+ trees, each B+ tree corresponding to a unit of data and each root node of each B+ tree comprising a unique fingerprint of a signature of the unit of data;

coordinating with a kernel module to identify a unit of data for de-duplication;

calculating a signature of the identified unit of data;

searching the de-duplication data structure to determine whether the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure;

upon determining that the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure, queuing the identified unit of data for de-duplication in a de-duplication queue;

upon determining that the signature of the identified unit of data does not match the signature of at least one of the entries in the de-duplication data structure, adding a new entry to the de-duplication data structure corresponding to the identified unit of data;

reviewing the entries of the de-duplication data structure and removing obsolete entries from the de-duplication data structure; and

coordinating with the kernel module to perform de-duplication operations on the de-duplication queue.

2. The method of claim 1 , wherein the units of data are 64 KB chunks of data.

3. The method of claim 1 , wherein coordinating with a kernel module comprises identifying a unit of data for de-duplication concurrently with I/O operations.

4. The method of claim 1 , wherein coordinating with a kernel module comprises identifying a unit of data for de-duplication based on one of: Information Lifecycle Management (ILM) information, First Write Bitmap (FWB) information and information regarding whether the unit of data changed during a previous de-duplication period.

5. The method of claim 1 , wherein each entry in the de-duplication data structure further comprise a timestamp indicating a time of calculation of the signature of the unit of data, and removing obsolete entries further comprises removing at least one of: an entry corresponding to a unit of data that changed during a previous de-duplication period, an entry having a timestamp older than a predetermined time and an entry corresponding to a unit of data associated with multiple entries in the de-duplication data structure.

6. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon that, when executed by a storage system computer, cause the storage system computer to:

maintain at least one de-duplication data structure, the de-duplication data structure including a plurality of entries of B+ trees, each B+ tree corresponding to a unit of data and each root node of each B+ tree comprising a unique fingerprint of a signature of the unit of data and a timestamp;

coordinate with a kernel module to identify a unit of data for de-duplication;

calculate a signature of the identified unit of data;

search the de-duplication data structure to determine whether the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure;

upon determining that the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure, queue the identified unit of data for de-duplication in a de-duplication queue;

upon determining that the signature of the identified unit of data does not match the signature of at least one of the entries in the de-duplication data structure, add a new entry to the de-duplication data structure corresponding to the identified unit of data;

review the entries of the de-duplication data structure and remove obsolete entries from the de-duplication data structure; and

coordinate with the kernel module to perform de-duplication operations on the de-duplication queue.

7. The non-transitory computer-readable storage medium of claim 6 , having further computer-executable instructions stored thereon that, when executed by the storage system computer, cause the storage system computer to coordinate with a kernel module by identifying a unit of data for de-duplication concurrently with I/O operations.

8. The non-transitory computer-readable storage medium of claim 6 , having further computer-executable instructions stored thereon that, when executed by the storage system computer, cause the storage system computer to coordinate with a kernel module by identifying a unit of data for de-duplication based on one of: Information Lifecycle Management (ILM) information, First Write Bitmap (FWB) information and information regarding whether the unit of data changed during a previous de-duplication period.

9. The non-transitory computer-readable storage medium of claim 6 , having further computer-executable instructions stored thereon that, when executed by the storage system computer, cause the storage system computer to predict a period of low load on the storage system computer using Information Lifecycle Management (ILM) information and coordinate with the kernel module to perform de-duplication operations on the de-duplication queue during the period of low load.

10. The non-transitory computer-readable storage medium of claim 6 , having further computer-executable instructions stored thereon that, when executed by the storage system computer, cause the storage system computer to coordinate with the kernel module to perform de-duplication operations on the de-duplication queue based on a predetermined schedule.

11. A data storage system, comprising:

a processing unit;

a memory in communication with the processing unit; and

a de-duplication lister module configured to:

coordinate with a kernel module to identify a unit of data for de-duplication; and

calculate a signature of the identified unit of data;

a de-duplication queue module configured to manage a de-duplication queue including units of data marked for de-duplication;

a de-duplication tracker module configured to:

maintain at least one de-duplication data structure, the de-duplication data structure including a plurality of entries of B+ trees, each B+ tree corresponding to a unit of data and each root node of each B+ tree comprising a unique fingerprint of a signature of the unit of data and a timestamp;

search the de-duplication data structure to determine whether the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure;

upon determining that the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure, mark the identified unit of data for de-duplication and communicate the marked unit of data to the de-duplication queue module;

upon determining that the signature of the identified unit of data does not match the signature of at least one of the entries in the de-duplication data structure, add a new entry to the de-duplication data structure corresponding to the identified unit of data;

a de-duplication maintainer module configured to review the entries of the de-duplication data structure and remove obsolete entries from the de-duplication data structure; and

a de-duplication engine module configured to coordinate with the kernel module to perform de-duplication operations on the de-duplication queue.

12. The data storage system of claim 11 , wherein the de-duplication lister module is further configured to coordinate with a kernel module by identifying a unit of data for de-duplication concurrently with I/O operations.

13. The data storage system of claim 11 , wherein the de-duplication lister module is further configured to coordinate with a kernel module by identifying a unit of data for de-duplication based on one of: Information Lifecycle Management (ILM) information, First Write Bitmap (FWB) information and information regarding whether the unit of data changed during a previous de-duplication period.

14. The data storage system of claim 11 , wherein the de-duplication maintainer module is further configured to review the entries of the de-duplication data structure by traversing the B+ tree.

15. The data storage system of claim 14 , wherein the obsolete entries in the de-duplication data structure are at least one of: an invalid entry, a duplicative entry or an aged entry in the B+ tree.