Multi-level deduplication
A method, a system, and a computer-implemented method for performing multi-level deduplication of data are disclosed. A zone stamp is generated for each zone in a plurality of zones contained in at least one data stream. The zone stamp is compared to another zone stamp. The zone stamp and another zone stamp represent zones in the plurality of zones. The comparison is performed for zones at corresponding zone levels based on a determination that a zone stamp of a zone of a preceding zone level is not similar to another zone stamp of another preceding zone level. The zone at the preceding zone level includes at least one zone of a next zone level having a size smaller than or equal to a size of the zone of the preceding zone level. The zone and another zone are deduplicated based on a determination that the zone stamp is similar to another zone stamp.
1. A computer-implemented method, comprising:
determining at least one first zone in a plurality of zones of a data stream not matching at least another zone in the plurality of zones;
splitting the at least one first zone into a plurality of second zones, each second zone being smaller in size than the at least one first zone;
upon determining that at least one second zone substantially matches at least one zone in the plurality of zones and has a substantially similar size as the at least one second zone, delta-compressing and deduplicating the at least one second zone; and
upon determining that at least one second zone does not substantially match at least one zone in the plurality of zones, splitting the at least one second zone into a plurality of third zones, each third zone being smaller in size than the at least one second zone.
2. The method according to claim 1 , wherein the splitting is performed using a rolling hash function.
3. The method according to claim 1 , wherein the splitting is performed based on a predetermined zone size for each of the first, second, and third zone.
4. The method according to claim 1 , further comprising
generating a zone stamp for each zone in the plurality of zones;
comparing a zone stamp to another zone stamp, the zone stamp representing a zone in the plurality of zones and the another zone stamp representing another zone in the plurality of zones;
deduplicating the zone and the another zone based on a determination that the zone stamp is similar to the another zone stamp.
5. The method according to claim 1 , wherein the deduplicating further comprises transmitting the delta-compressed zone across a network from a first storage location to a second storage location.
6. The method according to claim 1 , wherein upon determining that at least one third zone does not substantially match at least one zone in the plurality of zones,
data-compressing the at least one third zone; and
transmitting the data-compressed third zone across a network from a first storage location to a second storage location.
7. The method according to claim 1 , further comprising determining all zones in the plurality of zones simultaneously.
8. The method according to claim 1 , wherein for each zone in the plurality of zones, storing a zone entry in a table, the zone entry including
a starting location and a size of the zone,
a zone stamp identifying the zone based on the starting location and the size of the zone;
an identifier identifying location of the zone of the at least one data stream; and
a data stream identifier.
9. The method according to claim 8 , wherein upon determination that a zone stamp of a zone in the plurality of zones is similar to another zone stamp of another zone in the plurality of zones,
delta-compressing the zone;
storing the delta-compressed zone; and
removing a zone entry from a table corresponding to the zone.
10. The method according to claim 8 , wherein upon determination that a zone stamp of a zone in the plurality of zones is not similar to a zone stamp of any other zone in the plurality of zones,
data-compressing the zone;
storing the data-compressed zone; and
removing a zone entry from the table corresponding to the zone.
11. The method according to claim 1 , further comprising marking the at least one first zone deletion.
12. The method according to claim 11 , further comprising deleting, based on the marking, the plurality of second zones in the first zone upon determination that all next level zones contained within the preceding level zone have been deleted.
13. The method according to claim 11 , further comprising persisting the at least one first zone upon determination that at least one second zone in the plurality of second zones contained within the at least one first zone have not been marked for deletion.
14. The method according to claim 1 , wherein each zone has at least one of the following: a fixed length and a variable length.
15. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
determining at least one first zone in a plurality of zones of a data stream not matching at least another zone in the plurality of zones;
splitting the at least one first zone into a plurality of second zones, each second zone being smaller in size than the at least one first zone;
upon determining that at least one second zone substantially matches at least one zone in the plurality of zones and has a substantially similar size as the at least one second zone, delta-compressing and deduplicating the at least one second zone; and
upon determining that at least one second zone does not substantially match at least one zone in the plurality of zones, splitting the at least one second zone into a plurality of third zones, each third zone being smaller in size than the at least one second zone.
16. The system according to claim 15 , wherein the splitting is performed using a rolling hash function.
17. The system according to claim 15 , wherein the splitting is performed based on a predetermined zone size for each of the first, second, and third zone.
18. The system according to claim 15 , further comprising
generating a zone stamp for each zone in the plurality of zones;
comparing a zone stamp to another zone stamp, the zone stamp representing a zone in the plurality of zones and the another zone stamp representing another zone in the plurality of zones;
deduplicating the zone and the another zone based on a determination that the zone stamp is similar to the another zone stamp.
19. The system according to claim 15 , wherein the deduplicating further comprises transmitting the delta-compressed zone across a network from a first storage location to a second storage location.
20. The system according to claim 15 , wherein upon determining that at least one third zone does not substantially match at least one zone in the plurality of zones,
data-compressing the at least one third zone; and
transmitting the data-compressed third zone across a network from a first storage location to a second storage location.
21. The system according to claim 15 , further comprising determining all zones in the plurality of zones simultaneously.
22. The system according to claim 15 , wherein for each zone in the plurality of zones, storing a zone entry in a table, the zone entry including
a starting location and a size of the zone,
a zone stamp identifying the zone based on the starting location and the size of the zone;
an identifier identifying location of the zone of the at least one data stream; and
a data stream identifier.
23. The system according to claim 22 , wherein upon determination that a zone stamp of a zone in the plurality of zones is similar to another zone stamp of another zone in the plurality of zones,
delta-compressing the zone;
storing the delta-compressed zone; and
removing a zone entry from a table corresponding to the zone.
24. The system according to claim 22 , wherein upon determination that a zone stamp of a zone in the plurality of zones is not similar to a zone stamp of any other zone in the plurality of zones,
data-compressing the zone;
storing the data-compressed zone; and
removing a zone entry from the table corresponding to the zone.
25. The system according to claim 15 , further comprising marking the at least one first zone deletion.
26. The system according to claim 25 , further comprising deleting, based on the marking, the plurality of second zones in the first zone upon determination that all next level zones contained within the preceding level zone have been deleted.
27. The system according to claim 25 , further comprising persisting the at least one first zone upon determination that at least one second zone in the plurality of second zones contained within the at least one first zone have not been marked for deletion.
28. The system according to claim 15 , wherein each zone has at least one of the following: a fixed length and a variable length.
29. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
determining at least one first zone in a plurality of zones of a data stream not matching at least another zone in the plurality of zones;
splitting the at least one first zone into a plurality of second zones, each second zone being smaller in size than the at least one first zone;
upon determining that at least one second zone substantially matches at least one zone in the plurality of zones and has a substantially similar size as the at least one second zone, delta-compressing and deduplicating the at least one second zone; and
upon determining that at least one second zone does not substantially match at least one zone in the plurality of zones, splitting the at least one second zone into a plurality of third zones, each third zone being smaller in size than the at least one second zone.
30. The computer program product according to claim 29 , wherein the splitting is performed using a rolling hash function.
31. The computer program product according to claim 29 , wherein the splitting is performed based on a predetermined zone size for each of the first, second, and third zone.
32. The computer program product according to claim 29 , further comprising
generating a zone stamp for each zone in the plurality of zones;
comparing a zone stamp to another zone stamp, the zone stamp representing a zone in the plurality of zones and the another zone stamp representing another zone in the plurality of zones;
deduplicating the zone and the another zone based on a determination that the zone stamp is similar to the another zone stamp.
33. The computer program product according to claim 29 , wherein the deduplicating further comprises transmitting the delta-compressed zone across a network from a first storage location to a second storage location.
34. The computer program product according to claim 29 , wherein upon determining that at least one third zone does not substantially match at least one zone in the plurality of zones,
data-compressing the at least one third zone; and
transmitting the data-compressed third zone across a network from a first storage location to a second storage location.
35. The computer program product according to claim 29 , further comprising determining all zones in the plurality of zones simultaneously.
36. The computer program product according to claim 29 , wherein for each zone in the plurality of zones, storing a zone entry in a table, the zone entry including
a starting location and a size of the zone,
a zone stamp identifying the zone based on the starting location and the size of the zone;
an identifier identifying location of the zone of the at least one data stream; and
a data stream identifier.
37. The computer program product according to claim 36 , wherein upon determination that a zone stamp of a zone in the plurality of zones is similar to another zone stamp of another zone in the plurality of zones,
delta-compressing the zone;
storing the delta-compressed zone; and
removing a zone entry from a table corresponding to the zone.
38. The computer program product according to claim 36 , wherein upon determination that a zone stamp of a zone in the plurality of zones is not similar to a zone stamp of any other zone in the plurality of zones,
data-compressing the zone;
storing the data-compressed zone; and
removing a zone entry from the table corresponding to the zone.
39. The computer program product according to claim 29 , further comprising marking the at least one first zone deletion.
40. The computer program product according to claim 39 , further comprising deleting, based on the marking, the plurality of second zones in the first zone upon determination that all next level zones contained within the preceding level zone have been deleted.
41. The computer program product according to claim 39 , further comprising persisting the at least one first zone upon determination that at least one second zone in the plurality of second zones contained within the at least one first zone have not been marked for deletion.
42. The computer program product according to claim 29 , wherein each zone has at least one of the following: a fixed length and a variable length.