Deduplication and compression of data segments in a data storage system
View Patent ↗Techniques for performing data deduplication and compression in data storage systems. Data deduplication is performed in a deduplication domain on a segment-by-segment basis to obtain a plurality of deduplicated data segments. Deduplicated data segments are grouped together to form a plurality of compression groups. Data compression is performed on each compression group, and the compressed group is stored on spinning media. By performing data deduplication on a segment-by-segment basis, the size of each segment can be reduced to increase the effectiveness of data deduplication. By performing data compression on compression groups, the size of each compression domain can be increased to increase the effectiveness of data compression. By storing deduplicated data segments as a compressed group on the spinning media, a sequential nature of the segments can be preserved to reduce a seek time/rotational latency of the spinning media and a number of IOPS handled by the data storage system.
1. A method of data deduplication and compression in a data storage system including a data storage resource, comprising:
obtaining a data stream including a plurality of data slices;
performing, on a slice-by-slice basis, a data deduplication operation on the plurality of data slices to obtain a plurality of deduplicated data slices;
grouping the plurality of deduplicated data slices to form a plurality of compression groups, each of the plurality of compression groups corresponding to an uncompressed group of deduplicated data slices;
performing a data compression operation on each of the plurality of compression groups as a whole to obtain a plurality of compressed groups of deduplicated data slices;
storing the plurality of compressed groups of deduplicated data slices on the data storage resource;
performing at least the data deduplication operation in an in-line fashion as the data stream including the plurality of data slices is being obtained; and
having performed at least the data deduplication operation in the in-line fashion, performing a background process comprising:
obtaining the deduplicated data slices;
performing, on a slice-by-slice basis, a second data deduplication operation on the plurality of deduplicated data slices to obtain a second plurality of deduplicated data slices;
grouping the second plurality of deduplicated data slices to form a plurality of second compression groups, each of the plurality of second compression groups corresponding to an uncompressed group of deduplicated data slices;
performing a second data compression operation on each of the plurality of second compression groups as a whole to obtain a plurality of second compressed groups of deduplicated data slices; and
storing the plurality of second compressed groups of deduplicated data slices on the data storage resource.
2. The method of claim 1 further comprising:
slicing the data stream into the plurality of data slices, the plurality of data slices including a plurality of fixed-length data slices.
3. The method of claim 2 wherein the performing of the data deduplication operation includes performing the data deduplication operation on the plurality of fixed-length data slices to obtain a plurality of fixed-length deduplicated data slices.
4. The method of claim 3 wherein the grouping of the plurality of deduplicated data slices includes grouping the plurality of fixed-length deduplicated data slices to form a plurality of fixed-length compression groups.
5. The method of claim 1 further comprising:
slicing the data stream into the plurality of data slices, the plurality of data slices including a plurality of variable-length data slices.
6. The method of claim 5 wherein the performing of the data deduplication operation includes performing the data deduplication operation on the plurality of variable-length data slices to obtain a plurality of variable-length deduplicated data slices.
7. The method of claim 6 wherein the grouping of the plurality of deduplicated data slices includes grouping the plurality of variable-length deduplicated data slices to form a plurality of variable-length compression groups.
8. The method of claim 1 further comprising:
generating a metadata map that represents relationships between (i) a plurality of data slice indices for the plurality of deduplicated data slices, respectively, and (ii) a compression group identifier for each of the plurality of compression groups, the plurality of data slice indices pointing to locations within the plurality of compression groups where the plurality of deduplicated data slices reside.
9. The method of claim 1 further comprising:
performing the data deduplication operation and the data compression operation in an in-line fashion as the data stream including the plurality of data slices is being obtained.
10. The method of claim 9 wherein the performing of the data deduplication operation and the data compression operation in the in-line fashion comprises:
performing, in the in-line fashion:
(i) on a slice-by-slice basis, the data deduplication operation on the plurality of data slices to obtain a plurality of deduplicated data slices;
(ii) a staging operation to stage a fixed or variable number of the plurality of deduplicated data slices in a memory buffer to form each of the plurality of compression groups; and
(iii) the data compression operation on each of the plurality of compression groups as a whole in the memory buffer to obtain the plurality of compressed groups of deduplicated data slices.
11. A data storage system comprising:
a data storage resource;
a memory; and
a processor configured to execute instructions out of the memory:
to obtain a data stream including a plurality of data slices;
to perform, on a slice-by-slice basis, a data deduplication operation on the plurality of data slices to obtain a plurality of deduplicated data slices;
to group the plurality of deduplicated data slices to form a plurality of compression groups, each of the plurality of compression groups corresponding to an uncompressed group of deduplicated data slices;
to perform a data compression operation on each of the plurality of compression groups as a whole to obtain a plurality of compressed groups of deduplicated data slices;
to store the plurality of compressed groups of deduplicated data slices on the data storage resource;
to perform at least the data deduplication operation in an in-line fashion as the data stream including the plurality of data slices is being obtained; and
having performed at least the data deduplication operation in the in-line fashion, to perform a background process comprising:
obtaining the deduplicated data slices;
performing, on a slice-by-slice basis, a second data deduplication operation on the plurality of deduplicated data slices to obtain a second plurality of deduplicated data slices;
grouping the second plurality of deduplicated data slices to form a plurality of second compression groups, each of the plurality of second compression groups corresponding to an uncompressed group of deduplicated data slices;
performing a second data compression operation on each of the plurality of second compression groups as a whole to obtain a plurality of second compressed groups of deduplicated data slices; and
storing the plurality of second compressed groups of deduplicated data slices on the data storage resource.
12. The data storage system of claim 11 wherein the processor is further configured to execute the instructions out of the memory to slice the data stream into the plurality of data slices, wherein the plurality of data slices include a plurality of fixed-length data slices.
13. The data storage system of claim 12 wherein the processor is further configured to execute the instructions out of the memory to perform the data deduplication operation on the plurality of fixed-length data slices to obtain a plurality of fixed-length deduplicated data slices.
14. The data storage system of claim 13 wherein the processor is further configured to execute the instructions out of the memory to group the plurality of fixed-length deduplicated data slices to form a plurality of fixed-length compression groups.
15. The data storage system of claim 11 wherein the processor is further configured to execute the instructions out of the memory to slice the data stream into the plurality of data slices, wherein the plurality of data slices include a plurality of variable-length data slices.
16. The data storage system of claim 15 wherein the processor is further configured to execute the instructions out of the memory to perform the data deduplication operation on the plurality of variable-length data slices to obtain a plurality of variable-length deduplicated data slices, and to group the plurality of variable-length deduplicated data slices to form a plurality of variable-length compression groups.
17. The data storage system of claim 11 wherein the processor is further configured to execute the instructions out of the memory to generate a metadata map that represents relationships between (i) a plurality of data slice indices for the plurality of deduplicated data slices, respectively, and (ii) a compression group identifier for each of the plurality of compression groups, wherein the plurality of data slice indices point to locations within the plurality of compression groups where the plurality of deduplicated data slices reside.
18. A computer program product having a non-transitory computer readable medium that stores a set of instructions to perform data deduplication and compression in a data storage system that includes a data storage resource, the set of instructions, when carried out by computerized circuitry, causing the computerized circuitry to perform a method of:
obtaining a data stream including a plurality of data slices;
performing, on a slice-by-slice basis, a data deduplication operation on the plurality of data slices to obtain a plurality of deduplicated data slices;
grouping the plurality of deduplicated data slices to form a plurality of compression groups, each of the plurality of compression groups corresponding to an uncompressed group of deduplicated data slices;
performing a data compression operation on each of the plurality of compression groups as a whole to obtain a plurality of compressed groups of deduplicated data slices;
storing the plurality of compressed groups of deduplicated data slices on the data storage resource
performing at least the data deduplication operation in an in-line fashion as the data stream including the plurality of data slices is being obtained; and
having performed at least the data deduplication operation in the in-line fashion, performing a background process comprising:
obtaining the deduplicated data slices;
performing, on a slice-by-slice basis, a second data deduplication operation on the plurality of deduplicated data slices to obtain a second plurality of deduplicated data slices;
grouping the second plurality of deduplicated data slices to form a plurality of second compression groups, each of the plurality of second compression groups corresponding to an uncompressed group of deduplicated data slices;
performing a second data compression operation on each of the plurality of second compression groups as a whole to obtain a plurality of second compressed groups of deduplicated data slices; and
storing the plurality of second compressed groups of deduplicated data slices on the data storage resource.