Data reduction method, apparatus, computing device, and storage medium for forming index information based on fingerprints

1. A data reduction method, comprising:

obtaining fingerprints of to-be-reduced data blocks;

forming an index set based on the fingerprints of the to-be-reduced data blocks by using index information of data blocks with identical fingerprints, the index set comprising the index information of the data blocks, and the index information including addresses of the data blocks; and

performing, in the to-be-reduced data blocks based on the fingerprints of the to-be-reduced data blocks, data reduction processing on data blocks with index information in a same index set,

wherein the fingerprints of the data blocks are similar fingerprints or to-be-deduplicated fingerprints, the similar fingerprints are for determining whether similar deduplication can be performed on the data blocks, and the to-be-deduplicated fingerprints are for determining whether the data blocks can be deduplicated, and

wherein the index information of each of the data blocks is indicated by a key-value pair comprising a key and a value corresponding to the key, and

wherein, in the key-value pair indicating each of the data blocks, the key is the similar fingerprint of the data block, and the value comprises both the address of the data block and the to-be-deduplicated fingerprint of the data block.

2. The method according to claim 1 , wherein the forming an index set by using index information of data blocks with identical fingerprints comprises:

forming, in the to-be-reduced data blocks based on a log-structured merge (LSM) tree and/or a key-value pair (K-V) tree, the index set by using the index information of the data blocks with identical fingerprints.

3. The method according to claim 2 , wherein when the fingerprints are similar fingerprints, the index information comprises one or more addresses of at least one of the data blocks, or both one or more addresses of at least one of the data blocks and at least one of the to-be-deduplicated fingerprints; and when the fingerprints are to-be-deduplicated fingerprints, the index information comprises one or more addresses of at least one of the data blocks.

4. The method according to claim 2 , wherein

the forming an index set by using index information of the data blocks with identical fingerprints comprises:

forming a first index set by using first index information of the data blocks with identical similar fingerprints, and forming a second index set by using second index information of the data blocks with identical to-be-deduplicated fingerprints, wherein the first index information comprises one or more addresses of at least one of the data blocks, and the second index information comprises one or more addresses of at least one of the data blocks; and

forming a third index set by using third index information of the data blocks with identical similar fingerprints, wherein the third index information comprises one or more addresses of at least one of the data blocks and at least one of the to-be-deduplicated fingerprints.

5. The method according to claim 2 , wherein the forming, in the to-be-reduced data blocks based on an LSM tree, the index set by using the index information of the data blocks with identical fingerprints comprises:

when a merging condition for a level n and a level n+1 of the LSM tree is met, merging, in the to-be-reduced data blocks, index information of the data blocks with identical fingerprints at the level n and the level n+1 into the level n+1, and after the index information is merged into the level n+1, forming the index set by using the index information of the data blocks with identical fingerprints, wherein n is a natural number.

6. The method according to claim 2 , wherein when the index set is formed, in the to-be-reduced data blocks based on the LSM tree and the K-V tree, by using the index information of the data blocks with identical fingerprints, the performing, based on the fingerprints of the to-be-reduced data blocks, data reduction processing on data blocks with index information in a same index set comprises:

when a load of at least one processor that performs the method is greater than or equal to a first value, performing, based on the fingerprints of the to-be-reduced data blocks in the LSM tree, data reduction processing on the data blocks with the index information in the same index set; and

when the load of the at least one processor is less than the first value or the to-be-reduced data blocks correspond to a target identifier, performing, based on the fingerprints of the to-be-reduced data blocks in the K-V tree, data reduction processing on the data blocks with the index information in the same index set, wherein the target identifier indicates to perform reduction processing on the data blocks in real time.

7. The method according to claim 5 , further comprising:

obtaining a load of at least one processor that performs the method; and

the performing, based on the fingerprints of the to-be-reduced data blocks, data reduction processing on data blocks with index information in a same index set comprises:

creating, based on the fingerprints of the to-be-reduced data blocks, a reduction task for the data blocks of which the index information in the same index set, and executing the reduction task when the load of the at least one processor is less than a second value; and

when the load of the at least one processor is less than a second value, creating, based on the fingerprints of the to-be-reduced data blocks, a reduction task for the data blocks of which the index information in the same index set, and executing the reduction task.

8. A data reduction apparatus, comprising:

an interface; and

at least one processor coupled to the interface and configured to:

obtain fingerprints of to-be-reduced data blocks;

form an index set based on fingerprints of the to-be-reduced data blocks by using index information of data blocks with identical fingerprints, the index set comprising the index information of the data blocks, and the index information including addresses of the data blocks; and

perform, in the to-be-reduced data blocks based on the fingerprints of the to-be-reduced data blocks, data reduction processing on data blocks with index information in a same index set,

wherein the fingerprints of the data blocks are similar fingerprints or to-be-deduplicated fingerprints, the similar fingerprints are for determining whether similar deduplication can be performed on the data blocks, and the to-be-deduplicated fingerprints are for determining whether the data blocks can be deduplicated, and

wherein the index information of each of the data blocks is indicated by a key-value pair comprising a key and a value corresponding to the key, and

wherein, in the key-value pair indicating each of the data blocks, the key is the similar fingerprint of the data block, and the value comprises both the address of the data block and the to-be-deduplicated fingerprint of the data block.

9. The data reduction apparatus according to claim 8 , wherein the at least one processor is further configured to:

form, in the to-be-reduced data blocks based on a log-structured merge (LSM) tree and/or a key-value pair (K-V) tree, the index set by using the index information of the data blocks with identical fingerprints.

10. The data reduction apparatus according to claim 8 , wherein when the fingerprints are similar fingerprints, the index information comprises one or more addresses of at least one of the data blocks, or both one or more addresses of at least one of the data blocks and the to-be-deduplicated fingerprints; and when the fingerprints are to-be-deduplicated fingerprints, the index information comprises one or more addresses of at least one of the data blocks.

11. The data reduction apparatus according to claim 9 , wherein

the at least one processor is further configured to:

form a first index set by using first index information of the data blocks with identical similar fingerprints, and form a second index set by using second index information of the data blocks with identical to-be-deduplicated fingerprints, wherein the first index information comprises one or more addresses of at least one of the data blocks, and the second index information comprises one or more addresses of at least one of the data blocks; and

form a third index set by using third index information of the data blocks with identical similar fingerprints, wherein the third index information comprises one or more addresses of at least one of the data blocks and the to-be-deduplicated fingerprints.

12. The data reduction apparatus according to claim 9 , wherein the at least one processor is further configured to:

when a merging condition for a level n and a level n+1 of the LSM tree is met, merge, in the to-be-reduced data blocks, index information of the data blocks with identical fingerprints at the level n and the level n+1 into the level n+1, and after the index information is merged into the level n+1, form the index set by using the index information of the data blocks with identical fingerprints, wherein n is a natural number.

13. The data reduction apparatus according to claim 9 , wherein the at least one processor is further configured to:

when a load of the at least one processor is greater than or equal to a first value, perform, based on the fingerprints of the to-be-reduced data blocks in the LSM tree, data reduction processing on the data blocks with the index information in the same index set; and

when the load of the at least one processor is less than the first value or the to-be-reduced data blocks correspond to a target identifier, perform, based on the fingerprints of the to-be-reduced data blocks in the K-V tree, data reduction processing on the data blocks with the index information in the same index set, wherein the target identifier indicates to perform reduction processing on the data blocks in real time.

14. The data reduction apparatus according to claim 12 , wherein the at least one processor is further configured to:

obtain a load of the at least one processor;

create, based on the fingerprints of the to-be-reduced data blocks, a reduction task for the data blocks with the index information in the same index set, and execute the reduction task when the load of the at least one processor is less than a second value; and

when the load of the at least one processor is less than a second value, create, based on the fingerprints of the to-be-reduced data blocks, a reduction task for the data blocks with the index information in the same index set, and execute the reduction task.

15. A non-transitory storage medium storing information comprising instructions that, when executed by at least one processor, control the at least one processor to perform operations comprising:

obtaining fingerprints of to-be-reduced data blocks;

forming an index set based on the fingerprints of the to-be-reduced data blocks by using index information of data blocks with identical fingerprints, the index set comprising the index information of the data blocks, and the index information including addresses of the data blocks; and

performing, in the to-be-reduced data blocks based on the fingerprints of the to-be-reduced data blocks, data reduction processing on data blocks with index information in a same index set,

wherein the fingerprints of the data blocks are similar fingerprints or to-be-deduplicated fingerprints, the similar fingerprints are for determining whether similar deduplication can be performed on the data blocks, and the to-be-deduplicated fingerprints are for determining whether the data blocks can be deduplicated, and

wherein the index information of each of the data blocks is indicated by a key-value pair comprising a key and a value corresponding to the key, and

wherein, in the key-value pair indicating each of the data blocks, the key is the similar fingerprint of the data block, and the value comprises both the address of the data block and the to-be-deduplicated fingerprint of the data block.

16. The non-transitory storage medium according to claim 15 , wherein the forming an index set by using index information of data blocks with identical fingerprints comprises:

forming, in the to-be-reduced data blocks based on a log-structured merge (LSM) tree and/or a key-value pair (K-V) tree, the index set by using the index information of the data blocks with identical fingerprints.

17. The non-transitory storage medium according to claim 16 , wherein when the fingerprints are similar fingerprints, the index information comprises one or more addresses of at least one of the data blocks, or both one or more addresses of at least one of the data blocks and at least one of the to-be-deduplicated fingerprints; and when the fingerprints are to-be-deduplicated fingerprints, the index information comprises one or more addresses of at least one of the data blocks.