System and method for data deduplication for disk storage subsystems
A method for data deduplication includes the following steps. First, segmenting an original data set into a plurality of data segments. Next, transforming the data in each data segment into a transformed data representation that has a band-type structure for each data segment. The band-type structure includes a plurality of bands. Next, selecting a first set of bands, grouping them together and storing them with the original data set. The first set of bands includes non-identical transformed data for each data segment. Next, selecting a second set of bands and grouping them together. The second set of bands includes identical transformed data for each data segment. Next, applying a hash function onto the transformed data of the second set of bands and thereby generating transformed data segments indexed by hash function indices. Finally, storing the hash function indices and the transformed data representation of one representative data segment in a deduplication database.
1. A method for data deduplication comprising:
segmenting an original data set into a plurality of data segments;
applying a transform function to the data in each data segment of the plurality of data segments and transforming the data in said each data segment into a transformed data representation that comprises a band-type structure for said each data segment, wherein said band-type structure comprises a plurality of bands;
partitioning the plurality of bands into a first set of one or more bands of the plurality of bands and a second set of one or more bands of the plurality of bands, wherein the plurality of bands for each data segment of the plurality of data segments encodes structural information about said each data segment, wherein said first set of one or more bands is different in the transformed data representations of the plurality of data segments, wherein said second set of one or more bands, in the transformed data representations of the plurality of data segments, is identical transformed data indicating that the plurality of data segments comprise structurally identical data having transformed data representations that behave similarly and wherein the plurality of data segments are not identical;
applying a hash function onto the identical transformed data of the second set of one or more bands for the plurality of data segments and thereby generating transformed data segments indexed by hash function indices, wherein said hash function is not applied to the transformed data of the first set of one or more bands that is different in the transformed data representations of the plurality of data segments; and
storing a single copy of the identical transformed data with an associated hash value generated by the hash function for the identical transformed data in a deduplication database.
2. The method of claim 1 further comprising storing the hash function indices and the first set of bands with the original data set.
3. The method of claim 1 , wherein each of the plurality of data segments is transformed into a transform data representation via a Haar transform function.
4. The method of claim 3 , wherein said Haar transform function comprises one of a basic integer Haar transform function or a packet transform function.
5. The method of claim 1 , wherein each of the plurality of data segments is transformed into a transform data representation via a wavelet transform function.
6. The method of claim 5 , wherein said wavelet transform function comprises one of Haar transform function, a Daubechies 5/3 transform function or algorithmic variations thereof.
7. The method of claim 1 wherein said original data set comprises one of text, image, audio, financial data, or application binaries.
8. A method for data deduplication comprising:
segmenting an original data set into a plurality of data segments;
applying a transform function to the data in each data segment of the plurality of data segments and transforming the data in said each data segment into a transformed data representation comprising a band-type structure including a plurality of bands, wherein the plurality of bands for each data segment of the plurality of data segments encodes structural information about said each data segment;
removing one or more of the plurality of bands from each of the transformed data representations of the plurality of data segments thereby resulting in remaining transformed data representations for the plurality of data segments, wherein said one or more of the plurality of bands removed are different in the transformed data representations and wherein the remaining transformed data representations for the plurality of data segments are identical remaining transformed data representations indicating that the plurality of data segments comprise structurally identical data having transformed data representations that behave similarly and wherein the plurality of data segments are not identical;
applying a hash function onto the identical remaining transformed data representations and thereby generating transformed data representations indexed by hash function indices; and
storing a single copy of the identical remaining transformed data representation with an associated hash value generated by the hash function for the identical remaining transformed data representation in a deduplication database.
9. The method of claim 8 , further comprising storing the one or more of the plurality of bands removed from the transformed data representations for the plurality of data segments in a database with the original data set.
10. The method of claim 9 further comprising storing the hash function indices in the database with the original set of data.
11. A system for data deduplication comprising:
one or more computing host systems;
a deduplication engine configured to segment an original data set into a plurality of data segments, configured to apply a transform function to the data in each data segment of the plurality of segments and to transform the data in said each data segment into a transformed data representation comprising a band-type structure including a plurality of bands encoding structural information about said each data segment, configured to remove one or more of the plurality of bands from each of the transformed data representations of said plurality of data segments thereby resulting in remaining transformed data representations for the plurality of data segments, wherein said one or more of the plurality of bands removed are different in the transformed data representations and wherein the remaining transformed data representations for the plurality of data segments are identical remaining transformed data representations indicating that the plurality of data segments comprise structurally identical data having transformed data representation that behave similarly and wherein the plurality of data segments are not identical, configured to apply a hash function onto the identical remaining transformed data representations and thereby to generate transformed data representations indexed by hash function indices; and
a storage device that stores a single copy of the identical remaining transformed data representation with an associated hash value generated by the hash function for the identical remaining transformed data representation.
12. The system of claim 11 , wherein said deduplication engine is further configured to store the one or more removed bands from the transformed data representations for plurality of data segments in a database with the original set of data and is further configured to store the hash function indices in the database with the original set of data.
13. The system of claim 11 , wherein said original data set comprises one of text, image, audio, financial data, or application binaries.
14. A system for data deduplication comprising:
one or more computing host systems;
a deduplication engine configured to segment an original data set into a plurality of data segments, configured to apply a transform function to the data in each data segment in the plurality of data segments and to transform the data in said each data segment into a transformed data representation that comprises a band-type structure, wherein said band-type structure for said each data segment comprises a plurality of bands encoding structural information about said each data segment, configured to partition the plurality of bands into a first set of one or more bands of the plurality of bands and a second set of one or more bands of the plurality of bands, wherein said first set of one or more bands is different in the plurality of data segments, wherein said second set of one or more bands, for the plurality of data segments, is identical transformed data indicating that the plurality of data segments comprise structurally identical data having transformed data representations that behave similarly and wherein the plurality of data segments are not identical, configured to apply a hash function onto the identical transformed data of the second set of one or more bands and thereby to generate transformed data segments indexed by hash function indices, wherein said hash function is not applied to the transformed data of the first set of one or more bands that is different in the transformed data representations of the plurality of data segments; and
a database that stores a single copy of the identical transformed data with an associated hash value generated by the hash function for the identical transformed data.
15. The system of claim 14 , further comprising a storage device that stores the hash function indices and wherein the storage device comprises one or more virtual storage appliances provided by a cloud storage system.
16. The system of claim 14 , further comprising a cache and a local storage device and wherein said deduplication engine is located behind said cache and said local storage device.
17. The system of claim 14 , wherein said one or more computing host systems connect to said deduplication engine via an Internet Small Computer System Interface (iSCSI).
18. The system of claim 14 further comprising a bi-directional network connection between said deduplication engine and a cloud storage appliance.
19. The system of claim 14 further comprising a cloud layout module and a cloud provider application programming interface (API).
20. The method of claim 1 , further comprising:
storing a representation of the original data set that is a deduplicated representation of the original data set, said representation of the original data set including, for each of the plurality of data segments, the first set of one or more bands for said each data segment and a hash value generated by the hash function for the second set of one or more bands for said each data segment, wherein a database is indexed by hash values said hash value being included in an index.
21. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method for data deduplication comprising:
segmenting an original data set into a plurality of data segments;
applying a transform function to the data in each data segment of the plurality of data segments and transforming the data in said each data segment into a transformed data representation that comprises a band-type structure for said each data segment, wherein said band-type structure comprises a plurality of bands;
partitioning the plurality of bands into a first set of one or more bands of the plurality of bands and a second set of one or more bands of the plurality of bands, wherein the plurality of bands for each data segment of the plurality of data segments encodes structural information about said each data segment, wherein said first set of one or more bands is different in the transformed data representations of the plurality of data segments, wherein said second set of one or more bands, in the transformed data representations of the plurality of data segments, is identical transformed data indicating that the plurality of data segments comprise structurally identical data having transformed data representations that behave similarly and wherein the plurality of data segments are not identical;
applying a hash function onto the identical transformed data of the second set of one or more bands for the plurality of data segments and thereby generating transformed data segments indexed by hash function indices, wherein said hash function is not applied to the transformed data of the first set of one or more bands that is different in the transformed data representations of the plurality of data segments; and
storing a single copy of the identical transformed data with an associated hash value generated by the hash function for the identical transformed data in a deduplication database.
22. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method for data deduplication comprising:
segmenting an original data set into a plurality of data segments;
applying a transform function to the data in each data segment of the plurality of data segments and transforming the data in said each data segment into a transformed data representation comprising a band-type structure including a plurality of bands, wherein the plurality of bands for each data segment of the plurality of data segments encodes structural information about said each data segment;
removing one or more of the plurality of bands from each of the transformed data representations of the plurality of data segments thereby resulting in remaining transformed data representations for the plurality of data segments, wherein said one or more of the plurality of bands removed are different in the transformed data representations and wherein the remaining transformed data representations for the plurality of data segments are identical remaining transformed data representations indicating that the plurality of data segments comprise structurally identical data having transformed data representations that behave similarly and wherein the plurality of data segments are not identical;
applying a hash function onto the identical remaining transformed data representations and thereby generating transformed data representations indexed by hash function indices; and
storing a single copy of the identical remaining transformed data representation with an associated hash value generated by the hash function for the identical remaining transformed data representation in a deduplication database.