Providing data deduplication in a data storage system with parallelized computation of crypto-digests for blocks of host I/O data
In response to a cache flush event indicating that host data accumulated in a cache of a storage processor of a data storage system is to be flushed to a lower deck file system, an aggregation set of blocks is formed within the cache, and a digest calculation group is selected from within the aggregation set. Hardware vector processing logic is caused to simultaneously calculate crypto-digests from the blocks in the digest calculation group. If one of the resulting crypto-digests matches a previously generated crypto-digest, deduplication is performed that i) causes the lower deck file system to indicate the block of data from which the previously generated crypto-digest was generated and ii) discards the block that corresponds to the matching crypto-digest. Objects required by a digest generation component may be allocated in a just in time manner to avoid having to manage a pool of pre-allocated objects.
1. A method comprising:
accumulating data indicated by host I/O requests into a cache;
in response to a cache flush event, forming an aggregation set of blocks of data within the accumulated data in the cache;
selecting a digest calculation group of blocks from within the aggregation set; and
causing hardware vector processing logic to simultaneously calculate a plurality of crypto-digests, each crypto-digest in the plurality of crypto-digests being calculated from a corresponding one of the blocks in the digest calculation group of blocks.
2. The method of claim 1 , wherein causing the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests further comprises:
allocating, in a function call stack, a plurality of context objects, each one of the context objects corresponding to one of the blocks in the digest calculation group of blocks selected from within the aggregation set, wherein each one of the context objects contains workspace memory for use by a component of a shareable library while using the hardware vector processing logic to calculate a crypto-digest from the corresponding one of the blocks of data in the digest calculation group of blocks selected from the aggregation set; and
passing the plurality of context objects to the component of the shareable library to cause the component of the shareable library to use the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests.
3. The method of claim 2 , wherein causing the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests further comprises:
passing an indication of the corresponding one of the blocks in the digest calculation group of blocks to the component of the shareable library with each one of context objects in the plurality of context objects;
in response to detecting that all crypto-digests in the plurality of crypto-digests simultaneously calculated by the hardware vector processing logic have been completely calculated, extracting the crypto-digests simultaneously calculated by the hardware vector processing logic from the plurality of context objects; and
in response to all of the crypto-digests having been extracted from the plurality of context objects, deallocating the plurality of context objects from the function call stack.
4. The method of claim 3 , wherein causing the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests further comprises:
allocating, in the function call stack, a context manager object, wherein the context manager object corresponds to the digest calculation group of blocks selected from within the aggregation set, and wherein the context manager object contains workspace memory for use by the component of the shareable library while using the hardware vector processing logic to calculate the plurality of crypto-digests from the digest calculation group of blocks selected from the aggregation set.
5. The method of claim 4 , wherein causing the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests further comprises:
passing an indication of the context manager object to the component of the shareable library with each one of the context objects in the plurality of context objects; and
in response to all of the crypto-digests having been extracted from the plurality of context objects, deallocating the context manager object from the function call stack.
6. The method of claim 5 , further comprising:
wherein the hardware vector processing logic is part of a processing unit;
wherein the hardware vector processing logic includes at least one vector register; and
wherein causing the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests further comprises the component of the shareable library iteratively storing portions of each block in the digest calculation group into corresponding vector elements in the vector register to cause the hardware vector processing logic to perform vector operations that simultaneously process the stored portions of each block.
7. The method of claim 6 , wherein causing the hardware vector processing logic within the storage processor to simultaneously calculate the plurality of crypto-digests comprises using the hardware vector processing logic within the storage processor to perform a secure hash function on each one of the blocks in the digest calculation group; and
wherein the plurality of crypto-digests comprise digests that are outputs resulting from using the hardware vector processing logic within the storage processor to perform the secure hash function on the blocks in the digest calculation group.
8. The method of claim 7 , wherein the cache flush event indicates that a plurality of blocks of data accumulated in the cache are to be flushed to at least one lower deck file system, wherein the lower deck file system maps files to blocks in an address space of the lower deck file system, wherein each file implements a corresponding data object that is accessible to at least one host computer, and wherein blocks in the address space of the lower deck file system also correspond to non-volatile storage provisioned from a set of storage devices that provide storage for the lower deck file system.
9. The method of claim 1 , further comprising:
comparing each one of the plurality of crypto-digests to at least one previously generated crypto-digest that was previously generated from a block of data that was previously flushed from the cache; and
in response to detecting that one of the crypto-digests in the plurality of crypto-digests matches the previously generated crypto-digest, performing a deduplication operation on the one of the blocks in the digest calculation group of blocks that corresponds to the matching crypto-digest.
10. A storage processor in a data storage system, the storage processor including a set of processing units and memory, coupled to the set of processing units, the memory storing instructions, wherein the instructions, when executed by the set of processing units, cause the set of processing units to:
accumulate data indicated by host I/O requests into a cache;
in response to a cache flush event, form an aggregation set of blocks of data within the accumulated data in the cache;
select a digest calculation group of blocks from within the aggregation set; and
cause hardware vector processing logic to simultaneously calculate a plurality of crypto-digests, each crypto-digest in the plurality of crypto-digests being calculated from a corresponding one of the blocks in the digest calculation group of blocks.
11. The storage processor of claim 10 , wherein the instructions, when executed by the set of processing units, further cause the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests at least in part by causing the set of processing units to:
allocate, in a function call stack, a plurality of context objects, each one of the context objects corresponding to one of the blocks in the digest calculation group of blocks selected from within the aggregation set, wherein each one of the context objects contains workspace memory for use by a component of a shareable library while using the hardware vector processing logic to calculate a crypto-digest from the corresponding one of the blocks of data in the digest calculation group of blocks selected from the aggregation set; and
pass the plurality of context objects to the component of the shareable library to cause the component of the shareable library to use the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests.
12. The storage processor of claim 11 , wherein the instructions, when executed by the set of processing units, further cause the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests at least in part by causing the set of processing units to:
pass an indication of the corresponding one of the blocks in the digest calculation group of blocks to the component of the shareable library with each one of context objects in the plurality of context objects;
in response to detecting that all crypto-digests in the plurality of crypto-digests simultaneously calculated by the hardware vector processing logic have been completely calculated, extract the crypto-digests simultaneously calculated by the hardware vector processing logic from the plurality of context objects; and
in response to all of the crypto-digests having been extracted from the plurality of context objects, deallocate the plurality of context objects from the function call stack.
13. The storage processor of claim 12 , wherein the instructions, when executed by the set of processing units, further cause the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests at least in part by causing the set of processing units to:
allocate, in the function call stack, a context manager object, wherein the context manager object corresponds to the digest calculation group of blocks selected from within the aggregation set, wherein the context manager object contains workspace memory for use by the component of the shareable library while using the hardware vector processing logic to calculate the plurality of crypto-digests from the digest calculation group of blocks selected from the aggregation set.
14. The storage processor of claim 13 , wherein the instructions, when executed by the set of processing units, further cause the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests at least in part by causing the set of processing units to:
pass an indication of the context manager object to the component of the shareable library with each one of the context objects in the plurality of context objects; and
in response to all of the crypto-digests having been extracted from the plurality of context objects, deallocate the context manager object from the function call stack.
15. The storage processor of claim 14 , further comprising:
wherein the hardware vector processing logic is part of at least one of the processing units;
wherein the hardware vector processing logic includes at least one vector register; and
wherein the component of the shareable library uses the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests at least in part by storing portions of each block in the digest calculation group into corresponding vector elements in the vector register to cause the hardware vector processing logic to perform vector operations that simultaneously process the stored portions of each block.
16. The storage processor of claim 15 , wherein the component of the shareable library uses the hardware vector processing logic to simultaneously calculate the plurality of crypto-digests by using the hardware vector processing logic within the storage processor to perform a secure hash function on each one of the blocks in the digest calculation group; and
wherein the plurality of crypto-digests comprise digests that are outputs resulting from using the hardware vector processing logic within the storage processor to perform the secure hash function on the blocks in the digest calculation group.
17. The storage processor of claim 16 , wherein the cache flush event indicates that a plurality of blocks of data accumulated in the cache are to be flushed to at least one lower deck file system, wherein the lower deck file system maps files to blocks in an address space of the lower deck file system, wherein each file implements a corresponding data object that is accessible to at least one host computer, and wherein blocks in the address space of the lower deck file system also correspond to non-volatile storage provisioned from a set of storage devices that provide storage for the lower deck file system.
18. The storage processor of claim 10 , wherein the instructions, when executed by the set of processing units, further cause the set of processing units to:
compare each one of the plurality of crypto-digests to at least one previously generated crypto-digest that was previously generated from a block of data that was previously flushed from the cache; and
in response to detecting that one of the crypto-digests in the plurality of crypto-digests matches the previously generated crypto-digest, perform a deduplication operation on the one of the blocks in the digest calculation group of blocks that corresponds to the matching crypto-digest.
19. A non-transitory, computer-readable medium including instructions which, when executed by a set of processing units, cause the set of processing units to perform a method comprising the steps of:
accumulating data indicated by host I/O requests into a cache;
in response to a cache flush event, forming an aggregation set of blocks of data within the accumulated data in the cache;
selecting a digest calculation group of blocks from within the aggregation set; and
causing hardware vector processing logic to simultaneously calculate a plurality of crypto-digests, each crypto-digest in the plurality of crypto-digests being calculated from a corresponding one of the blocks in the digest calculation group of blocks.