Methods and systems for de-duplicating blocks of data
Disclosed herein are systems and method for de-duplicating blocks of data. In one aspect, an exemplary method comprises for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor, creating a set of hash components of the previously de-duplicated blocks, and for each newly received block of data for de-duplication, calculating a translation tolerant hash vector including a predetermined number of hash components, determining a similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set, and when the received block is determined as being similar to the previously processed blocks based on the comparison, storing the block without duplication in the de-duplication engine, including pages of the block referenced by hashes and a block descriptor.
1. A method for de-duplicating blocks of data, the method comprising:
for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor;
creating a set of hash components of the previously de-duplicated blocks; and
for each newly received block of data for de-duplication,
calculating a translation tolerant hash vector including a predetermined number of hash components,
determining a similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set, and
when the received block is determined as being similar to the previously de-duplicated blocks based on the comparison, storing the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
2. The method of claim 1 , wherein the received block is determined as being similar to the previously processed blocks when the number of hash components of the received block contained in the set of hash components exceeds a predetermined threshold.
3. The method of claim 1 , wherein the predetermined number of hash components are evenly distributed by stripping high order bits from hashes of pages, and selecting the predetermined number of hash components based on remaining bits of the hashes.
4. The method of claim 1 , wherein the de-duplication engine is scalable.
5. The method of claim 1 , wherein each block consists multiple pages.
6. The method of claim 1 , wherein the similarity of the received block with previously processed blocks is determined using a scalable Bloom filter.
7. The method of claim 1 , further comprising: periodically removing data and objects of the de-duplication engine that are no longer needed.
8. The method of claim 1 , further comprising: updating the set of hash components when the received block is added to a storage of the de-duplicated engine.
9. A system for de-duplicating blocks of data, comprising:
at least one processor of a server, the processor configured to:
for each previously de-duplicated block of data of a de-duplication engine, store de-duplicated pages references by hashes and a block descriptor;
create a set of hash components of the previously de-duplicated blocks; and
for each newly received block of data for de-duplication,
calculate a translation tolerant hash vector including a predetermined number of hash components,
determine a similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set, and
when the received block is determined as being similar to the previously de-duplicated blocks based on the comparison, store the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
10. The system of claim 9 , wherein the received block is determined as being similar to the previously processed blocks when the number of hash components of the received block contained in the set of hash components exceeds a predetermined threshold.
11. The system of claim 9 , wherein the predetermined number of hash components are evenly distributed by stripping high order bits from hashes of pages, and selecting the predetermined number of hash components based on remaining bits of the hashes.
12. The system of claim 9 , wherein the de-duplication engine is scalable.
13. The system of claim 9 , wherein each block consists multiple pages.
14. The system of claim 9 , wherein the similarity of the received block with previously processed blocks is determined using a scalable Bloom filter.
15. The system of claim 9 , the configuration further comprising a configuration for:
periodically removing data and objects of the de-duplication engine that are no longer needed.
16. The system of claim 9 , the configuration further comprising a configuration for:
updating the set of hash components when the received block is added to a storage of the de-duplicated engine.
17. A non-transitory computer readable medium storing thereon computer executable instructions for de-duplicating blocks of data, including instructions for:
for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor;
creating a set of hash components of the previously de-duplicated blocks; and
for each newly received block of data for de-duplication,
calculating a translation tolerant hash vector including a predetermined number of hash components,
determining a similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set, and
when the received block is determined as being similar to the previously de-duplicated blocks based on the comparison, storing the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
18. The non-transitory computer readable medium of claim 17 , wherein the received block is determined as being similar to the previously processed blocks when the number of hash components of the received block contained in the set of hash components exceeds a predetermined threshold.
19. The non-transitory computer readable medium of claim 17 , wherein the predetermined number of hash components are evenly distributed by stripping high order bits from hashes of pages, and selecting the predetermined number of hash components based on remaining bits of the hashes.
20. The non-transitory computer readable medium of claim 17 , wherein the de-duplication engine is scalable.
21. The non-transitory computer readable medium of claim 17 , wherein each block consists multiple pages.
22. The non-transitory computer readable medium of claim 17 , wherein the similarity of the received block with previously processed blocks is determined using a scalable Bloom filter.
23. The non-transitory computer readable medium of claim 17 , the instructions further comprising instructions for: periodically removing data and objects of the de-duplication engine that are no longer needed.
24. The non-transitory computer readable medium of claim 17 , the instructions further comprising instructions for: updating the set of hash components when the received block is added to a storage of the de-duplicated engine.