Techniques for generating data sets with specified compression and deduplication ratios
Techniques for generating data sets may include: receiving an initial buffer that achieves a compression ratio responsive to compression processing using a compression algorithm, the initial buffer including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and generating a data set of buffers using the initial buffer. The data set may be expected to achieve a specified deduplication ratio responsive to deduplication processing and to achieve the compression ratio responsive to compression processing using the compression algorithm. Generating the data set may include generating a first plurality of buffers where each buffer of the first plurality is not a duplicate of another buffer in the first plurality, and generating a second plurality of duplicate buffers. Each duplicate buffer may be a duplicate of a buffer in the first plurality of buffers.
1. A method of generating data sets comprising:
receiving, at a system using a processor, initial data stored in an initial buffer, wherein compressing the initial data, by applying a first compression algorithm using a processor, generates a compressed form of the initial data having a first compression ratio, the initial buffer of the initial data including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and
generating, using a processor and the initial data, a data set, wherein performing a combination of deduplicating the data set, by performing deduplication processing using a processor, and compressing the data set, by applying the first compression algorithm using a processor, generates a resulting data set that is expected to have a specified deduplication ratio and is expected to have the first compression ratio, wherein said generating the data set using the initial data includes:
generating, using a processor, a first plurality of data stored in a first plurality of buffers, wherein each of the first plurality of buffers is unique with respect to its content and is not a duplicate with respect to content of any another buffer in the first plurality, wherein a counter associated with the initial buffer is set to a first value and the first value is stored at the first position in a first buffer of the first plurality, and wherein the second content of the initial buffer is stored at the second position in the first buffer; and
generating, using a processor, a second plurality of data stored in a second plurality of duplicate buffers, wherein each duplicate buffer of the second plurality is a duplicate of data of one of the first plurality of buffers of the data set, wherein a data value stored at the first position of a first duplicate buffer of the second plurality is included in a range of prior counter values of the counter, and wherein each prior counter value in the range is stored at the first position in a different one of the first plurality of buffers.
2. The method of claim 1 , wherein the range of prior counter values is from zero (0) to a maximum value, inclusively.
3. The method of claim 1 , wherein a total data reduction ratio of the data set as a result of deduplication processing and compression processing using the first compression algorithm is expected to be a mathematical product of the specified deduplication ratio multiplied by the first compression ratio.
4. The method of claim 3 , wherein a first number of duplicate buffers in the data set is in accordance with the specified deduplication ratio, and wherein the second plurality of duplicate buffers includes the first number of duplicate buffers.
5. The method of claim 4 , wherein N denotes a total number of buffers of data in the data set, wherein the specified deduplication ratio denotes a percentage of the total number of buffers of the data set that are duplicates, wherein the mathematical product of N and the percentage is equal to the first number that denotes a number of buffers of data in the second plurality of duplicate buffers, wherein a second number is equal to the difference between N and the first number, and wherein the second number denotes a number of buffers of data in the first plurality of buffers.
6. The method of claim 1 , wherein the data value stored at the first position of a first duplicate buffer of the second plurality is randomly selected from the range of prior counter values of the counter.
7. The method of claim 6 , wherein the data value is selected using a pseudo-random number generator.
8. The method of claim 1 , wherein the first position of each buffer in the data set corresponds to a plurality of bytes at one end of said each buffer.
9. The method of claim 8 , wherein the first position of each buffer in the data set corresponds to 8 bytes located at a beginning of said each buffer.
10. The method of claim 1 , wherein said generating the first plurality of data stored in the first plurality of buffers includes:
after generating the first buffer of data of the first plurality, incrementing the counter by one from the first value to a second value;
generating a second buffer of data of the first plurality, wherein the second value of the counter associated with the initial buffer is stored at the first position in the second buffer of the first plurality, and wherein the second content of the initial buffer is stored at the second position in the second buffer;
after generating the second buffer of data of the first plurality, incrementing the counter by one from the second value to a third value; and
generating a third buffer of data of the first plurality, wherein the third value of the counter associated with the initial buffer is stored at the first position in the third buffer of the first plurality, and wherein the second content of the initial buffer is stored at the second position in the third buffer.
11. The method of claim 1 , wherein the initial buffer is a first initial buffer of a first plurality of initial buffers of data associated with the first compression algorithm, and wherein each of the initial buffers of data in the first plurality of initial buffers, when compressed by applying the first compression algorithm using a processor, generates compressed data having a different compression ratio than any other initial buffer of data in the first plurality of initial buffers.
12. The method of claim 11 , wherein a second plurality of initial buffers is associated with a second compression algorithm that is different than the first compression algorithm, and wherein each of the initial buffers of data in the second plurality of initial buffers, when compressed by applying the second compression algorithm using a processor, generates compressed data having a different compression ratio than any other initial buffer of data in the second plurality of initial buffers.
13. The method of claim 12 , wherein the first compression algorithm and the second compression algorithm are different compression algorithms selectable for processing performed as part of inline processing of the data path in a data storage system.
14. The method of claim 1 , wherein each buffer in the data set is a same size.
15. The method of claim 14 , wherein the same size is a value that is at least 4K bytes and wherein the first position of each buffer of the data set corresponds to a same 8 byte region at one end of said each buffer.
16. The method of claim 3 , further comprising:
generating the resulting data set as a result of performing, using a processor, data deduplication processing and compression processing with the first compression algorithm on the data set; and
verifying that the total data reduction ratio is achieved with respect to the resulting data set and the data set.
17. A system comprising:
one or more processors; and
a memory comprising code stored thereon that, when executed, performs a method of generating data sets comprising:
receiving, at a system using a processor, initial data stored in an initial buffer, wherein compressing the initial data, by applying a first compression algorithm using a processor, generates a compressed form of the initial data having a first compression ratio, the initial buffer of the initial data including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and
generating, using a processor and the initial data, a data set, wherein performing a combination of deduplicating the data set, by performing deduplication processing using a processor, and compressing the data set, by applying the first compression algorithm using a processor, generates a resulting data set that is expected to have a specified deduplication ratio and is expected to have the first compression ratio, wherein said generating the data set using the initial data includes:
generating, using a processor, a first plurality of data stored in a first plurality of buffers, wherein each of the first plurality of buffers is unique with respect to its content and is not a duplicate with respect to content of any another buffer in the first plurality, wherein a counter associated with the initial buffer is set to a first value and the first value is stored at the first position in a first buffer of the first plurality, and wherein the second content of the initial buffer is stored at the second position in the first buffer; and
generating, using a processor, a second plurality of data stored in a second plurality of duplicate buffers, wherein each duplicate buffer of the second plurality is a duplicate of data of one of the first plurality of buffers of the data set, wherein a data value stored at the first position of a first duplicate buffer of the second plurality is included in a range of prior counter values of the counter, and wherein each prior counter value in the range is stored at the first position in a different one of the first plurality of buffers.
18. A non-transitory computer readable medium comprising code stored thereon that, when executed by one or more processors, performs a method of generating data sets comprising:
receiving, at a system, initial data stored in an initial buffer, wherein compressing the initial data, by applying a first compression algorithm using a processor, generates a compressed form of the initial data having a first compression ratio, the initial buffer of the initial data including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and
generating, using the initial data, a data set, wherein performing a combination of deduplicating the data set, by performing deduplication processing using a processor, and compressing the data set, by applying the first compression algorithm using a processor, generates a resulting data set that is expected to have a specified deduplication ratio and is expected to have the first compression ratio, wherein said generating the data set using the initial data includes:
generating a first plurality of data stored in a first plurality of buffers, wherein each of the first plurality of buffers is unique with respect to its content and is not a duplicate with respect to content of any another buffer in the first plurality, wherein a counter associated with the initial buffer is set to a first value and the first value is stored at the first position in a first buffer of the first plurality, and wherein the second content of the initial buffer is stored at the second position in the first buffer; and
generating a second plurality of data stored in a second plurality of duplicate buffers, wherein each duplicate buffer of the second plurality is a duplicate of data of one of the first plurality of buffers of the data set, wherein a data value stored at the first position of a first duplicate buffer of the second plurality is included in a range of prior counter values of the counter, and wherein each prior counter value in the range is stored at the first position in a different one of the first plurality of buffers.
19. The method of claim 1 , wherein the resulting data set is expected to have the specified deduplication ratio, within specified tolerances or bounds, and is expected to have the first compression ratio, within specified tolerances or bounds.
20. The method of claim 16 , wherein said verifying includes:
determining an achieved data reduction ratio for the resulting data set;
determining, within specified tolerances or bounds, whether there is a difference between the achieved data reduction ratio and the total data reduction ratio;
responsive to determining within the specified tolerances or bounds that there is no difference between the achieved data reduction ratio and the total data reduction ratio, successfully verifying and validating data reduction processing performed to generate the resulting data set, wherein said data reduction processing includes the data deduplication processing of the data set and the compression processing of the data set by applying the first compression algorithm as performed in said generating the resulting data set; and
responsive to determining that there is a difference between the achieved data reduction ratio and the total data reduction ratio and wherein the difference is not within the specified tolerances or bounds, determining that verification and validation of the data reduction processing performed to generate the resulting data set has failed.
21. The method of claim 1 , wherein the initial buffer of data, when compressed by applying the first compression algorithm using a processor, generates a compressed initial buffer of data having the first compression ratio, within specified tolerances and bounds.