Methods and systems for de-duplicating blocks of data
Disclosed herein are systems and method for de-duplicating blocks of data. In one aspect, an exemplary method comprises for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor. The method comprises receiving, at the de-duplication engine, a new block of data for de-duplication assessment and determining a similarity of the received block to the previously de-duplicated blocks. When the received block is determined as being similar to the previously de-duplicated blocks, the method comprises storing the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
1. A method for de-duplicating blocks of data, the method comprising:
for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor;
receiving, at the de-duplication engine, a new block of data for de-duplication assessment;
determining a similarity of the received block to the previously de-duplicated blocks;
when the received block is determined as being similar to the previously de-duplicated blocks, storing the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
2. The method of claim 1 , wherein determining the similarity comprises:
creating a set of hash components of the previously de-duplicated blocks; and
calculating a translation tolerant hash vector including a predetermined number of hash components of the received block; and
determining the similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set.
3. The method of claim 2 , wherein the received block is determined as being similar to the previously de-duplicated blocks when the number of hash components of the received block contained in the set of hash components exceeds a predetermined threshold.
4. The method of claim 2 , wherein the predetermined number of hash components are evenly distributed by stripping high order bits from hashes of pages, and selecting the predetermined number of hash components based on remaining bits of the hashes.
5. The method of claim 2 , further comprising updating the set of hash components when the received block is added to a storage of the de-duplicated engine.
6. The method of claim 1 , wherein the de-duplication engine is scalable.
7. The method of claim 1 , wherein each block comprises multiple pages.
8. The method of claim 1 , wherein the similarity of the received block with previously de-duplicated blocks is determined using a scalable Bloom filter.
9. The method of claim 1 , further comprising periodically removing data and objects of the de-duplication engine that are no longer needed.
10. A system for de-duplicating blocks of data, comprising:
at least one processor configured to:
for each previously de-duplicated block of data of a de-duplication engine, store de-duplicated pages references by hashes and a block descriptor;
receive, at the de-duplication engine, a new block of data for de-duplication assessment;
determine a similarity of the received block to the previously de-duplicated blocks;
when the received block is determined as being similar to the previously de-duplicated blocks, store the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
11. The system of claim 10 , wherein the at least one processor is configured to determine the similarity by:
creating a set of hash components of the previously de-duplicated blocks; and
calculating a translation tolerant hash vector including a predetermined number of hash components of the received block; and
determining the similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set.
12. The system of claim 11 , wherein the received block is determined as being similar to the previously de-duplicated blocks when the number of hash components of the received block contained in the set of hash components exceeds a predetermined threshold.
13. The system of claim 11 , wherein the predetermined number of hash components are evenly distributed by stripping high order bits from hashes of pages, and selecting the predetermined number of hash components based on remaining bits of the hashes.
14. The system of claim 11 , further comprising updating the set of hash components when the received block is added to a storage of the de-duplicated engine.
15. The system of claim 10 , wherein the de-duplication engine is scalable.
16. The system of claim 10 , wherein each block comprises multiple pages.
17. The system of claim 10 , wherein the similarity of the received block with previously de-duplicated blocks is determined using a scalable Bloom filter.
18. The system of claim 10 , wherein the at least one processor is configured to periodically remove data and objects of the de-duplication engine that are no longer needed.
19. A non-transitory computer readable medium storing thereon computer executable instructions for de-duplicating blocks of data, including instructions for:
for each previously de-duplicated block of data of a de-duplication engine, storing de-duplicated pages references by hashes and a block descriptor;
receiving, at the de-duplication engine, a new block of data for de-duplication assessment;
determining a similarity of the received block to the previously de-duplicated blocks;
when the received block is determined as being similar to the previously de-duplicated blocks, storing the received block without duplication in the de-duplication engine, including pages of the block referenced by the hashes and the block descriptor.
20. The non-transitory computer readable medium of claim 19 , wherein an instruction for determining the similarity comprises instructions for:
creating a set of hash components of the previously de-duplicated blocks; and
calculating a translation tolerant hash vector including a predetermined number of hash components of the received block; and
determining the similarity of the received block to the previously de-duplicated blocks based on a comparison of the hash components of the received block with the hash components in the set.