Systems and methods for de-duplication in storage systems
In accordance with embodiments of the present disclosure, a storage system may include a storage array comprising one or more storage resources, a processor communicatively coupled to the storage array, and a de-duplication module comprising instructions embodied on a computer-readable medium communicatively coupled to the processor. The de-duplication module may be configured to, when read and executed by the processor: generate a fingerprint for an item of data stored on the storage array; identify a partition for the fingerprint; associate the partition with a hardware instance selected from a plurality of hardware instances, wherein each particular hardware instance comprises one or more information handling resources; and query the selected hardware instance to determine if the fingerprint exists on the hardware instance.
1. A storage system, comprising:
a storage array comprising one or more physical storage resources;
a processor communicatively coupled to the storage array;
a de-duplication module comprising instructions embodied on a non-transitory computer-readable medium communicatively coupled to the processor, the de-duplication module configured to, when read and executed by the processor:
generate a fingerprint for an item of data stored on the storage array;
identify a partition for the fingerprint;
associate the partition with a selected hardware instance selected from a plurality of hardware instances, wherein each particular hardware instance comprises one or more information handling resources; and
query the selected hardware instance to determine whether the fingerprint exists on the hardware instance; wherein the de-duplication module configured to, in order to identify each partition, calculate a function to identify a partition for each fingerprint: wherein the function is operable to map all possible fingerprints to a corresponding partition; and the function operable to map each fingerprint to a partition such that the number of fingerprints associated with each partition would be approximately equal.
2. The storage system according to claim 1 , further comprising the plurality of hardware instances, each hardware instance configured to respond to the query with an indication of whether the fingerprint exists on the hardware instance.
3. The storage system according to claim 2 , at least one of the de-duplication module and at least one of the hardware instances further configured to add the fingerprint to the hardware instance if the fingerprint does not exist on the hardware instance.
4. The storage system according to claim 1 , the de-duplication module further configured to de-duplicate data associated with the fingerprint if the fingerprint exists on the hardware instance.
5. The storage system according to claim 1 , wherein one of the plurality of hardware instances comprises the processor.
6. The storage system according to claim 1 , the de-duplication module configured to generate fingerprints, identify partitions, and associate partitions with hardware instances in concert with one or more other de-duplication modules executing on one or more other processors.
7. The storage system according to claim 1 , the de-duplication module configured to, in order to associate each partition with a hardware instance, calculate a second function to assign each partition to a hardware instance; wherein the second function is operable to map all possible partitions to a corresponding hardware instance.
8. The storage system according to claim 7 , the second function operable to map each partition to a hardware instance such that the number of partitions associated with each hardware instance would be approximately equal.
9. The storage system according to claim 1 , the de-duplication module configured to, in response to a change in the number of hardware instances, re-assign at least one partition to a different hardware instance selected from the plurality of hardware instances.
10. A method, comprising:
generating a fingerprint for an item of data stored on the storage array;
identifying a partition for the fingerprint;
associating the partition with a selected hardware instance selected from a plurality of hardware instances, wherein each particular hardware instance comprises one or more information handling resources; and
querying the selected hardware instance to determine whether the fingerprint exists on the hardware instance; wherein in order to identify each partition, calculating a function to identify a partition for each fingerprint; wherein the function is operable to map all possible fingerprints to a corresponding partition, wherein the function operable to map each fingerprint to a partition such that the number of fingerprints associated with each partition would be approximately equal.
11. The method according to claim 10 , further comprising, responding, by each hardware instance, to the query with an indication of whether the fingerprint exists on the hardware instance.
12. The method according to claim 11 , further comprising adding the fingerprint to the hardware instance if the fingerprint does not exist on the hardware instance.
13. The method according to claim 10 , further comprising de-duplicating data associated with the fingerprint if the fingerprint exists on the hardware instance.
14. The method according to claim 10 , in order to associate each partition with a hardware instance, calculation a second function to assign each partition to a hardware instance;
wherein the second function is operable to map all possible partitions to a corresponding hardware instance.
15. The method according to claim 14 , the second function operable to map each partition to a hardware instance such that the number of partitions associated with each hardware instance would be approximately equal.
16. The method according to claim 10 , further comprising, in response to a change in the number of hardware instances, re-assigning at least one partition to a different hardware instance selected from the plurality of hardware instances.