Methods and system for vectored data de-duplication
View Patent ↗The present invention is directed toward methods and systems for data de-duplication. More particularly, in various embodiments, the present invention provides systems and methods for data de-duplication that may utilize a vectoring method for data de-duplication wherein a stream of data is divided into “data sets” or blocks. For each block, a code, such as a hash or cyclic redundancy code may be calculated and stored. The first block of the set may be written normally and its address and hash can be stored and noted. Subsequent block hashes may be compared with previously written block hashes.
1. A computerized method for vectored data de-duplication, comprising:
comparing a de-duplication code for a first block of data to a de-duplication code for a previously processed block of data, where the first block of data was received in an input stream of computer-readable data;
upon determining that the de-duplication code for the first block of data matches the de-duplication code for the previously processed block of data:
identifying a number of copies of the first block of data that are available in an output stream of computer-readable data produced from the input stream;
upon determining that the number of copies of the first block of data that are available in the output stream does not satisfy a pre-determined threshold number of blocks:
storing the first block of data in the output stream;
upon determining that the number of copies of the first block of data that are available in the output stream does satisfy the pre-determined threshold number of blocks:
storing, in the output stream, in a location where the first block of data would have been placed if the number of copies of the first block of data satisfied the pre-determined threshold number of blocks, a vector that includes data for locating the previously processed block of data in the output stream, or a vector that includes data for locating a duplicate of the previously processed block of data in the output stream;
where the input stream can be recreated from the output stream without reference to other de-duplication data structures, and where the output stream includes self-describing data.
2. The method of claim 1 , where the pre-determined number is an odd number.
3. The method of claim 1 , where the pre-determined number is three.
4. The method of claim 1 , where the vector is a pointer to a physical block in the output stream or is a pointer to a logical block in the output stream.
5. The method of claim 1 , where the vector includes pointers to two or more blocks or two or more vectors in the output stream.
6. The method of claim 1 , where the vector includes data for locating a number of blocks in the output stream that equals or exceeds the pre-determined threshold number of blocks.
7. The method of claim 1 , where the vector includes a capsule number, a vessel number, a block number, or an offset, where a vessel is a non-empty set of capsules, and where a capsule is a non-empty set of blocks.
8. The method of claim 1 , where the vector includes a start address and an offset.
9. The method of claim 1 , where the output stream is stored in a server, a disk array, or a tape server.
10. The method of claim 1 , comprising:
upon determining that the de-duplication code for the first block of data matches the de-duplication code for the previously processed block of data, verifying that the first block of data matches the previously processed block of data.
11. The method of claim 10 , where verifying that the first block of data matches the previously processed block of data includes performing a bit-by-bit comparison of the first block of data and the previously processed block of data, a byte-by-byte comparison of the first block of data and the previously processed block of data, or a word-by-word comparison of the first block of data and the previously processed block of data.
12. The method of claim 1 , comprising:
accessing a candidate block in the output stream;
determining a number of redundant copies of the candidate block that are present in the output stream;
upon determining that the number of redundant copies exceeds a threshold number of redundant copies:
determining how many of the redundant copies are identical;
upon determining that a majority of the redundant copies are identical:
determining whether the candidate block matches the majority; and
upon determining that the candidate block matches the majority:
identifying the candidate block as a valid block.
13. The method of claim 12 , where determining the number of redundant copies of the candidate block that are present in the output stream includes locating blocks that are related to the candidate block using vectors associated with the candidate block.
14. The method of claim 12 , comprising:
upon determining that the candidate block does not match the majority, replacing the candidate block with a member of the majority.
15. The method of claim 12 , comprising:
upon determining that a majority of the redundant copies of the candidate block are not identical, reporting an error concerning the candidate block.
16. The method of claim 12 , where the threshold number of redundant copies of the candidate block is an odd number.
17. The method of claim 12 , where the threshold number of redundant copies of the candidate block is three.
18. An apparatus, comprising:
a processor;
a memory;
a set of computer hardware components that perform vector based de-duplication for an input stream of computer-readable data blocks and produces an output stream of computer-readable data blocks, where the input stream can be recreated from the output stream without reference to other de-duplication data structures, and where the output stream includes self-describing data; and
an interface that connects the processor, the memory, and the set of computer hardware components,
where the set of computer hardware components include:
a first hardware component that identifies whether a candidate block of data in the input stream is a duplicate of a block of data in the output stream based on de-duplication codes;
a second hardware component that selectively adds a vector to the output stream in place of the candidate block of data; and
a third hardware component that selectively adds up to a threshold number of redundant copies of the candidate block of data to the output stream.
19. The apparatus of claim 18 ,
where the second hardware component adds the vector to the output stream upon determining that the output stream includes the threshold number of redundant copies of the candidate block of data, and
where the third hardware component adds the candidate block of data to the output stream upon determining that the output stream does not include the threshold number of redundant copies of the candidate block of data.
20. The apparatus of claim 19 , where the set of computer hardware components includes a fourth hardware component that identifies a read block of data in the output stream as a valid block of data upon determining that the read block of data matches a majority of redundant copies of the read data block in the output stream.