Information processing device and non-transitory computer-readable storage medium
An information processing device includes a processor. The processor configured to extract part of data from estimation target data as a plurality of pieces of sample data and manage a number of duplications of the extracted plurality of pieces of sample data, classify the plurality of pieces of sample data into a first group and a second group based on the number of duplications, the first group having a number of duplications equal to or less that a predetermined number, the second group having a number of duplications more than a predetermined number, specify a first deduplication rate for sample data classified into the first group, specify a second deduplication rate for sample data classified into the second group, and specify a deduplication rate of the estimation target data based on the first deduplication rate and the second deduplication rate.
1. An information processing device comprising:
an interface circuit configured to be coupled to a storage device;
a memory; and
a processor coupled to the memory, the processor being configured to execute a process to:
extract, as a plurality of chunks, part of data from estimation target data stored in the storage device coupled to the information processing device via the interface circuit, by partial scanning the estimation target data at a predetermined ratio;
count, for each chunk of the plurality of chunks, a number of duplications of the chunk by comparing respective hash values calculated from the chunk and other chunks of the plurality of chunks using a predetermined hash function, to store information indicating the counted number of duplications of the chunk in the memory;
classify each chunk of the plurality of chunks into any one of a first group or a second group based on the number of duplications for the chunk, the first group having a number of duplications equal to or less that a predetermined number, the second group having a number of duplications more than the predetermined number;
apply, for each chunk of the plurality of chunks, any one of a first estimation scheme or a second estimation scheme based on whether the chunk is classified into the first group or the second group, the first estimation scheme including measuring a deduplication rate with a first accuracy, the second estimation scheme including measuring a deduplication rate with a second accuracy that is less than the first accuracy of the first estimation scheme, the second estimation scheme requiring a processor load that is less than a processor load required in applying the first estimation scheme, the applying of the any one of the first estimation scheme or the second estimation scheme including:
determining, for first chunks among the plurality of chunks, a first deduplication rate by applying the first estimation scheme to the first chunks, the first chunks being chunks classified into the first group among the plurality of chunks, and
determining, for second chunks among the plurality of chunks, a second deduplication rate by applying the second estimation scheme to the second chunks, the second estimation scheme including measuring the second deduplication rate with an accuracy less than the first estimation scheme, the second chunks being chunks classified into the second group among the plurality of chunks; and
output, as a deduplication rate of whole of the estimation target data, a rate obtained based on the first deduplication rate and the second deduplication rate.
2. The information processing device according to claim 1 , wherein the processor is further configured to
estimate an amount of data for each degree of duplication from one to the predetermined number using a plurality of pieces of sample data classified into the first group,
wherein the determining of the first deduplication rate includes calculating the first deduplication rate based on the estimated amount of data for each degree of duplication.
3. The information processing device according to claim 2 , wherein the processor is further configured to estimate the amount of data for each degree of duplication from one to the predetermined number based on an expected value of the amount of data for every degree of duplication of the plurality of pieces of sample data.
4. The information processing device according to claim 3 , wherein the processor is further configured to
create simultaneous equations with the amount of data for each degree of duplication as a variable for the expected value, and
solve the created simultaneous equations to estimate the amount of data for each degree of duplication from one to the predetermined number.
5. The information processing device according to claim 1 , wherein the determining of the second deduplication rate includes determining the second deduplication rate based on a total number and types of sample data classified into the second group.
6. A non-transitory computer-readable storage medium storing a program that causes a processor included in an information processing device to execute a process, the process comprising:
extracting, as a plurality of chunks, part of data from estimation target data stored in a storage device coupled to the information processing device via an interface circuit, by partial scanning the estimation target data at a predetermined ratio;
counting, for each chunk of the plurality of chunks, a number of duplications of the chunk by comparing respective hash values calculated from the chunk and other chunks of the plurality of chunks using a predetermined hash function, to store information indicating the counted number of duplications of the chunk in a memory of the information processing device;
classifying each chunk of the plurality of chunks into any one of a first group or a second group based on the number of duplications for the chunk, the first group having a number of duplications equal to or less that a predetermined number, the second group having a number of duplications more than the predetermined number;
applying, for each chunk of the plurality of chunks, any one of a first estimation scheme or a second estimation scheme based on whether the chunk is classified into the first group or the second group, the first estimation scheme including measuring a deduplication rate with a first accuracy, the second estimation scheme including measuring a deduplication rate with a second accuracy that is less than the first accuracy of the first estimation scheme, the second estimation scheme requiring a processor load that is less than a processor load required in applying the first estimation scheme, the applying of the any one of the first estimation scheme or the second estimation scheme including:
determining, for first chunks among the plurality of chunks, a first deduplication rate by applying the first estimation scheme to the first chunks, the first chunks being chunks classified into the first group among the plurality of chunks, and
determining, for second chunks among the plurality of chunks, a second deduplication rate by applying the second estimation scheme to the second chunks, the second estimation scheme including measuring the second deduplication rate with an accuracy less than the first estimation scheme, the second chunks being chunks classified into the second group among the plurality of chunks; and
outputting, as a deduplication rate of whole of the estimation target data, a rate obtained based on the first deduplication rate and the second deduplication rate.
7. The non-transitory computer-readable storage medium according to claim 6 , wherein the process further includes
estimating an amount of data for each degree of duplication from one to the predetermined number using a plurality of pieces of sample data classified into the first group, wherein
the determining of the first deduplication rate includes calculating the first deduplication rate based on the estimated amount of data for each degree of duplication.
8. The non-transitory computer-readable storage medium according to claim 7 , wherein the estimating includes estimating the amount of data for each degree of duplication from one to the predetermined number based on an expected value of the amount of data for every degree of duplication of the plurality of pieces of sample data.
9. The non-transitory computer-readable storage medium according to claim 8 , wherein the process further includes
creating simultaneous equations with the amount of data for each degree of duplication as a variable for the expected value, wherein
the estimating includes estimating the amount of data for each degree of duplication from one to the predetermined number by solving the created simultaneous equations.