Granted Patent B2

US 12,430,306 · App. 18/939,942 · Granted Sep 30, 2025

Inventor: Peilin Mao (Beijing, CN)

Assignee: New H3C Cloud Technologies Co., Ltd.

G06F16/215

Abstract

Claims (69)

1. A method for purging data fingerprint, wherein the method is applied to a storage engine in a distributed storage system, the method comprising a fingerprint purging procedure performed by the storage engine under a synchronized control of a master node, the fingerprint purging procedure for each of the storage engines comprising:

performing a deduplication service suspending process;

performing a fingerprint scanning process on a local Redirect-On-Write (ROW) database, for identifying valid fingerprints in a local fingerprint table, in response to a determination that all storage engines in the distributed storage system complete the deduplication service suspending process;

performing a fingerprint purging process for deleting invalid fingerprints from the local fingerprint table after all the storage engines complete the fingerprint scanning process;

resuming deduplication services after all the storage engines complete the fingerprint purging process;

wherein the method further comprises: triggering the master node to instruct each of all the storage engines to perform the fingerprint purging procedure, comprising:

determining, by the storage engine, a deduplication data overwrite counting value since a last fingerprint purging procedure is performed by a node to which the storage engine belongs, and sending the deduplication data overwrite counting value to the master node;

receiving, by the master node, deduplication data overwrite counting values sent by all of the storage engines, summing up the deduplication overwrite counting values, and

triggering the master node to instruct each of all the storage engines to perform the fingerprint purging procedure in response to a determination that a sum of the deduplication overwrite counting values reaches or exceeds a preset watermark value.

2. The method of claim 1 , wherein the fingerprint scanning process for identifying valid fingerprints in the local fingerprint table comprises:

identifying a fingerprint referenced by the local ROW database as a valid fingerprint by traversing the local ROW database;

sending the identified valid fingerprint to another storage engine to which the identified valid fingerprint belongs;

receiving a plurality of valid fingerprints belonging to the storage engine; and

identifying which fingerprint among all fingerprints belonging to the storage engine is valid.

3. The method of claim 2 , wherein

the ROW database is a key-value database for persisting a redirection aggregation mapping relationship between a small I/O data write and a large I/O data write;

a fingerprint referenced by the ROW database is a referenced fingerprint in a value field of a deduplication-type.

4. The method of claim 2 , wherein identifying which fingerprint among all fingerprints belonging to the storage engine is valid comprises:

generating a fingerprint validity bitmap based on the received valid fingerprints, and

using the fingerprint validity bitmap to identify a local valid fingerprint among all fingerprints belonging to the storage engine;

wherein each binary bit in the fingerprint validity bitmap corresponds to a fingerprint that belongs to the storage engine; a fingerprint corresponding to a binary bit 1 is valid; a fingerprint corresponding to a binary bit 0 is invalid.

5. The method of claim 4 , wherein the performing a fingerprint purging process for deleting invalid fingerprints from the local fingerprint table comprises:

traversing the local fingerprint table; and

deleting each fingerprint marked as invalid corresponding to a binary bit 0 in the fingerprint validity bitmap from the local fingerprint table.

6. The method of claim 4 , wherein

prior to the generating a fingerprint validity bitmap based on the received valid fingerprints, the method further comprises:

persisting the received valid fingerprints into a fingerprint reference table, wherein the fingerprint reference table is used to recover the fingerprint validity bitmap in case of power failure.

7. The method of claim 1 , wherein

the deduplication data overwrite counting value determined by the storage engine is persisted in this storage engine in response to a determination that the deduplication data overwrite counting value determined by the storage engine reaches or exceeds an integer multiple of a preset persistence threshold value.

8. The method of claim 1 , wherein:

states of all the storage engines in the distributed storage system during performing the fingerprint purging procedure are synchronized by the master node based on a fingerprint purging state machine.

9. An electronic device, comprising:

a processor;

a communication interface;

a storage; and

a communication bus; wherein the processor, the communication interface, the storage are intercommunicated via the communication bus;

the storage is to store machine-executable instructions;

the processor is to execute the machine-executable instructions to cause a storage engine in a distributed storage system to perform a fingerprint purging procedure under a synchronized control of a master node, wherein the fingerprint purging procedure comprises operations comprising:

performing a deduplication service suspending process;

performing a fingerprint scanning process on a local Redirect-On-Write (ROW) database, for identifying valid fingerprints in a local fingerprint table, in response to a determination that all storage engines in the distributed storage system complete the deduplication service suspending process;

performing a fingerprint purging process for deleting invalid fingerprints from the local fingerprint table after all storage engines complete the fingerprint scanning process;

resuming deduplication services after all the storage engines complete the fingerprint purging process;

wherein the operations further comprise: triggering the master node to instruct each of all the storage engines to perform the fingerprint purging procedure, comprising:

determining, by the storage engine, a deduplication data overwrite counting value since a last fingerprint purging procedure is performed by a node to which the storage engine belongs, and sending the deduplication data overwrite counting value to the master node;

receiving, by the master node, deduplication data overwrite counting values sent by all of the storage engines, summing up the deduplication overwrite counting values, and

triggering the master node to instruct each of all the storage engines to perform the fingerprint purging procedure in response to a determination that a sum of the deduplication overwrite counting values reaches or exceeds a preset watermark value.

10. The electronic device of claim 9 , wherein the fingerprint scanning process for identifying valid fingerprints in the local fingerprint table comprises:

identifying a fingerprint referenced by the local ROW database as a valid fingerprint by traversing the local ROW database;

sending the identified valid fingerprint to another storage engine to which the identified valid fingerprint belongs;

receiving a plurality of valid fingerprints belonging to the storage engine; and

identifying which fingerprint among all fingerprints belonging to the storage engine is valid.

11. The electronic device of claim 10 , wherein

the ROW database is a key-value database for persisting a redirection aggregation mapping relationship between a small I/O data write and a large I/O data write;

a fingerprint referenced by the ROW database is a referenced fingerprint in a value field of a deduplication-type.

12. The electronic device of claim 10 , wherein identifying which fingerprint among all fingerprints belonging to the storage engine is valid comprises:

generating a fingerprint validity bitmap based on the received valid fingerprints, and

using the fingerprint validity bitmap to identify a local valid fingerprint among all fingerprints belonging to the storage engine;

wherein each binary bit in the fingerprint validity bitmap corresponds to a fingerprint that belongs to the storage engine; a fingerprint corresponding to a binary bit 1 is valid; a fingerprint corresponding to a binary bit 0 is invalid.

13. The electronic device of claim 12 , wherein the performing a fingerprint purging process for deleting invalid fingerprints from the local fingerprint table comprises:

traversing the local fingerprint table; and

deleting each fingerprint marked as invalid corresponding to a binary bit 0 in the fingerprint validity bitmap from the local fingerprint table.

14. The electronic device of claim 12 , wherein

prior to the generating a fingerprint validity bitmap based on the received valid fingerprints, the operations further comprise:

persisting the received valid fingerprints into a fingerprint reference table, wherein the fingerprint reference table is used to recover the fingerprint validity bitmap in case of power failure.

15. The electronic device of claim 9 , wherein

the deduplication data overwrite counting value determined by the storage engine is persisted in this storage engine in response to a determination that the deduplication data overwrite counting value determined by the storage engine reaches or exceeds an integer multiple of a preset persistence threshold value.

16. The electronic device of claim 9 , wherein:

states of all the storage engines in the distributed storage system during performing the fingerprint purging procedure are synchronized by the master node based on a fingerprint purging state machine.

17. A non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores machine-executable instructions to be invoked and executed by a processor to cause the processor to perform the method of claim 1 .

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (9)