Data deduplication latency reduction
Aspects of the present disclosure relate to reducing the latency of data deduplication. In embodiments, an input/output (IO) workload received by a storage array is monitored. Further, at least one IO write operation in the IO workload is identified. A space-efficient probabilistic data structure is used to determine if a director board is associated with the IO write. Additionally, the IO write operation is processed based on the determination.
1. A method comprising:
monitoring an input/output (TO) workload received by a storage array;
identifying at least one IO write operation in the IO workload;
generating a unique fingerprint identifier for the IO write operation;
selecting a set of n-bits of the unique fingerprint to store in a space-efficient probabilistic data structure, wherein the space-efficient probabilistic data structure includes a cuckoo filter;
determining a director board is associated with the IO write operation using the space-efficient probabilistic data structure, wherein the unique fingerprint is generated such that the set of n-bits of the unique fingerprint identify one of the storage array's director boards associated with the unique fingerprint's related IO write operation; and
processing the IO write operation based on the determination.
2. The method of claim 1 , further comprising receiving the IO write operation by a first director board.
3. The method of claim 2 , wherein determining if the director board is associated with the IO write operation, includes:
determining if the first director board or a second director board remote from the first director board is related to the IO write operation, wherein the storage array houses the first and second director board.
4. The method of claim 1 , further comprising:
pre-allocating a portion of the storage array's global memory for each of the storage array's director boards; and
configuring each director board's pre-allocated global memory portion to store the space-efficient probabilistic data structure.
5. The method of claim 4 , wherein pre-allocating each director board's global memory portion, includes uniformly provisioning a global memory section for each director board.
6. The method of claim 1 , further comprising:
generating the unique fingerprint using hash processing.
7. The method of claim 1 , further comprising:
maintaining a copy of the space-efficient probabilistic data structure on each of the storage array's director boards.
8. The method of claim 7 , wherein maintaining the copy of the space-efficient probabilistic data structure, includes:
periodically identifying one or more fingerprints generated during a prior time window; and
updating each copy of the space-efficient probabilistic data structure with each generated fingerprint's set of n-bits.
9. An apparatus comprising a memory and processor configured to:
monitor an input/output (TO) workload received by a storage array;
identify at least one IO write operation in the IO workload;
generate a unique fingerprint identifier for the IO write operation;
select a set of n-bits of the unique fingerprint to store in a space-efficient probabilistic data structure, wherein the space-efficient probabilistic data structure includes a cuckoo filter;
determine a director board is associated with the IO write operation using the space-efficient probabilistic data structure, wherein the unique fingerprint is generated such that the set of n-bits of the unique fingerprint identify one of the storage array's director boards associated with the unique fingerprint's related IO write operation; and
process the IO write operation based on the determination.
10. The apparatus of claim 9 , further configured to receive the IO write operation by a first director board.
11. The apparatus of claim 10 , further configured to:
determine if the first director board or a second director board remote from the first director board is associated with the IO write operation, wherein the storage array houses the first and second director board.
12. The apparatus of claim 9 , further configured to:
pre-allocate a portion of the storage array's global memory for each of the storage array's director boards; and
configure each director board's pre-allocated global memory portion to store the space-efficient probabilistic data structure.
13. The apparatus of claim 12 , further configured to uniformly provision a global memory section of each director board's pre-allocated global memory portion for each director board.
14. The apparatus of claim 9 , further configured to:
generate the unique fingerprint using hash processing.
15. The apparatus of claim 9 , further configured to:
maintain a copy of the space-efficient probabilistic data structure on each of the storage array's director boards.
16. The apparatus of claim 15 , further configured to:
periodically identify one or more fingerprints generated during a prior time window; and
update each copy of the space-efficient probabilistic data structure with each generated fingerprint's set of n-bits.