Method and apparatus for content-aware and adaptive deduplication
A method, a system, an apparatus, and a computer readable medium for transmission of data across a network are disclosed. The method includes receiving a data stream, analyzing the received data stream to determine a starting location and an ending location of each zone within the received data stream, based on the starting and ending locations, generating a zone stamp identifying the zone, the zone stamp includes a sequence of contiguous characters representing at least a portion of data in the zone, wherein the order of characters in the zone stamp corresponds to the order of data in the zone, comparing the zone stamp with another zone stamp of another zone in any data stream received, determining whether the zone is substantially similar to another zone by detecting that the zone stamp is substantially similar to another zone stamp, delta-compressing zones within any data stream received that have been determined to have substantially similar zone stamps, thereby deduplicating zones having substantially similar zone stamps within any data stream received, and transmitting the deduplicated zones across the network from one storage location to another storage location.
1. A system, comprising
a first deduplication and storage appliance including a first deduplication processor and a first memory communicatively coupled to the first deduplication processor;
the first deduplication and storage appliance receiving a data stream from at least one server in a plurality of servers, the plurality of servers being communicatively coupled to the first deduplication and storage appliance, the data stream including a plurality of zones, each zone in the plurality of zone being represented by a zone stamp and being characterized by a predetermined minimum and maximum zone size and a predetermined minimum and maximum zone stamp length;
the first deduplication processor delta-compressing zones in the received data stream based on a determination that a zone in the plurality of zones
is substantially similar to another zone upon detecting that a zone stamp representing the zone is substantially similar to another zone stamp representing the another zone; and
has a size greater than the predetermined minimum zone size and less than the predetermined maximum size and a stamp length greater than the predetermined minimum zone stamp length;
the first memory storing zones delta-compressed by the first deduplication processor.
2. The system according to claim 1 , further comprising a second deduplication and storage appliance communicatively coupled to the first deduplication and storage appliance;
the second deduplication appliance receiving zones delta-compressed zones by the first deduplication and storage appliance.
3. The system according to claim 2 , wherein at least one of the first deduplication and storage appliance and the second deduplication and storage appliance are a public storage provider.
4. The system according to claim 2 , further comprising a third deduplication and storage appliance communicatively coupled to the second deduplication and storage appliance;
the third deduplication appliance receiving zones, delta-compressed zones by the first deduplication and storage appliance, from the second deduplication and storage appliance.
5. The system according to claim 2 , further comprising a third deduplication and storage appliance communicatively coupled to the first deduplication and storage appliance;
the third deduplication appliance receiving zones, delta-compressed zones by the first deduplication and storage appliance, from the first deduplication and storage appliance in parallel with the second deduplication and storage appliance receiving delta-compressed zones from the first deduplication and storage appliance.
6. The system according to claim 1 , further comprising
a plurality of first deduplication and storage appliances;
a plurality of second deduplication and storage appliances communicatively coupled to the plurality of first deduplication and storage appliances;
at least one second deduplication and storage appliance in the plurality of second deduplication and storage appliances receiving zones delta-compressed zones by at least one first deduplication and storage appliance in the plurality of first deduplication and storage appliances.
7. The system according to claim 1 , further comprising a plurality of second deduplication and storage appliances communicatively coupled to the first deduplication and storage appliance;
the plurality of second deduplication appliance receiving zones delta-compressed zones by the first deduplication and storage appliance.
8. The system according to claim 2 , wherein the second deduplication and storage appliance is located remotely from the first deduplication and storage appliance.
9. The system according to claim 2 , wherein at least one of the first deduplication and storage appliance and the second deduplication and storage appliance receive the data stream from the at least one server in the plurality of servers.
10. The system according to claim 1 , wherein the first memory stores zone stamps of zones in the plurality of zones in a zone database, wherein the zone database stores zone stamps of the zones within any data stream received by the first deduplication and storage appliance.
11. The system according to claim 1 , wherein the first deduplication processor, during delta-compression, eliminates redundant data from any data stream received by the first deduplication and storage appliance.
12. The system according to claim 1 , wherein the first deduplication processor defines an end zone located at an end of any data stream received by the first deduplication and storage appliance.
13. The system according to claim 12 , wherein the first deduplication processor aggregates the defined end zone with a zone preceding the defined end zone for delta-compression when size of the defined end zone is less than the predetermined minimum zone size.
14. The system according to claim 12 , wherein the first deduplication processor aggregates the defined end zone with a zone preceding the defined end zone for delta-compressing when a total combined length of zone stamps for the defined end zone and a zone preceding the defined zone is less than the predetermined maximum zone stamp length.
15. The system according to claim 12 , wherein the first deduplication processor creates a separate end zone stamp for the defined end zone for delta-compressing when size of the defined end zone is greater than the predetermined minimum zone size and the predetermined minimum zone stamp length.
16. The system according to claim 15 , wherein the zone stamp of the preceding zone remains unchanged.
17. The system according to claim 1 , wherein the zone stamp has at least one of the following lengths: a fixed length and a variable length.
18. The system according to claim 1 , wherein the zone is a low entropy data zone, wherein the low entropy data zone includes homogeneous data within any data stream received by the first deduplication and storage appliance.
19. The system according to claim 1 , wherein the first deduplication processor compares zone stamps of zones having a difference in respective zone sizes that is less than or equal to a predetermined zone size difference;
wherein zones having a difference in respective zone sizes that is greater than the predetermined zone size difference are not compared to each other.
20. The system according to claim 19 , wherein upon determining that compared zones are substantially similar to one another, the first deduplication processor discontinues comparison of at least one compared zone to other zones in any data stream received by the first deduplication and storage appliance.
21. The system according to claim 20 , wherein the first deduplication processor computes an edit distance score between the compared zone stamps, wherein the edit distance score reflects the number of differences between the compared zone stamps, wherein each compared zone stamp is limited by a predetermined maximum zone stamp length.
22. The system according to claim 21 , wherein the first deduplication processor
compares the computed edit distance score to a predetermined threshold distance score; and,
determines that the compared zones are substantially similar when the computed edit distance score is less than the predetermined threshold distance score.
23. The system according to claim 22 , wherein the computed edit distance score closer to zero indicates that the compared zones are substantially similar.
24. The system according to claim 22 , wherein the predetermined threshold distance score is a function of a length of a shorter zone stamp of the zone stamps being compared.
25. The system according to claim 22 , wherein the first deduplication process processor, upon comparison of zone stamps of zones, selects a best match between a pair of zones in the received data stream for comparison with other zones in any data stream received by the first deduplication and storage appliance.