Methods and systems for vectored data de-duplication
View Patent ↗The present invention is directed toward methods and systems for data de-duplication. More particularly, in various embodiments, the present invention provides systems and methods for data de-duplication that may utilize a vectoring method for data de-duplication wherein a stream of data is divided into “data sets” or blocks. For each block, a code, such as a hash or cyclic redundancy code may be calculated and stored. The first block of the set may be written normally and its address and hash can be stored and noted. Subsequent block hashes may be compared with previously written block hashes.
1. A computerized method for vectored data de-duplication, comprising:
comparing a de-duplication code for a first block of data in a first vessel to a de-duplication code for a previously processed block of data stored in a repository, where the first vessel stores a non-empty set of capsules, and where a capsule stores a non-empty set of blocks; and
upon determining that the de-duplication code for the first block of data matches the de-duplication code for the previously processed block of data:
storing, in the repository, in a location where the first block of data would have been placed, a vector that includes data for locating the previously processed block of data;
where the first vessel can be recreated from the repository without reference to other de-duplication data structures, and where the repository includes self-describing data.
2. The method of claim 1 , where the first vessel is a data storage apparatus.
3. The method of claim 2 , where the data storage apparatus is a tape, a disk, a flash drive, a solid state drive, or a storage area network.
4. The method of claim 3 , where the repository is a second, different vessel.
5. The method of claim 4 , where the second vessel is a second, different data storage apparatus.
6. The method of claim 5 , where the second data storage apparatus is a tape, a disk, a flash drive, a solid state drive, or a storage area network.
7. The method of claim 1 , where the first vessel is stored on two or more data storage devices.
8. The method of claim 5 , where the second vessel is stored on two or more data storage devices.
9. The method of claim 7 , where the second vessel is stored on two or more data storage devices.
10. The method of claim 1 , where the vector includes a vessel number, a block number, an offset, or a count.
11. The method of claim 1 , where the vector includes a start address and an offset.
12. The method of claim 1 , where the vector includes data for locating a duplicate of the previously processed block of data in the repository.
13. The method of claim 1 , comprising:
upon determining that the de-duplication code for the first block of data matches the de-duplication code for the previously processed block of data, verifying that the first block of data matches the previously processed block of data.
14. The method of claim 13 , where verifying that the first block of data matches the previously processed block of data includes performing a bit-by-bit comparison of the first block of data and the previously processed block of data, a byte-by-byte comparison of the first block of data and the previously processed block of data, or a word-by-word comparison of the first block of data and the previously processed block of data.
15. An apparatus, comprising:
a processor;
a memory;
a set of computer hardware components that perform vector based de-duplication for a vessel of computer-readable data blocks, where the vessel stores a non-empty set of capsules, and where a capsule stores a non-empty set of blocks; and
an interface that connects the processor, the memory, and the set of computer hardware components,
where the set of computer hardware components includes:
a first hardware component that identifies whether a candidate block of data in the vessel is a duplicate of a stored block of data in the vessel by comparing a de-duplication code for the candidate block of data and a de-duplication code for the stored block of data; and
a second hardware component that selectively replaces the candidate block of data with a vector;
where the vessel can be recreated from the repository without reference to other de-duplication data structures, and where the repository includes self-describing data.
16. The apparatus of claim 15 , comprising:
a third hardware component that selectively leaves up to a threshold number of redundant copies of the candidate block of data in the vessel.
17. The apparatus of claim 15 , where the vessel is stored on two or more data storage devices.
18. The apparatus of claim 17 , where the two or more data storage devices include a tape, a disk, a flash drive, a solid state drive, or a storage area network.
19. The apparatus of claim 15 , where the vector includes a vessel number, a block number, an offset, or a count.
20. The apparatus of claim 15 , where the vector includes data for locating a duplicate of the stored block of data in the vessel.