Coalescing multiple small writes to large files or multiple writes to a number of small files to generate larger compressible chunks for inline compression
Systems and methods for coalescing writes to facilitate generation of larger compression groups for use during inline compression are provided. According to one embodiment, inline compression performed by a storage system is improved by temporarily staging writes to in-memory data structures (e.g., inline storage efficiency (ISE) index nodes (inodes)) and performing coalescing in a deferred manner to generate larger compression groups for use during performance of inline compression. In one example, all files may be treated in the same manner, for example, by staging writes within a staging area and then processing the staged data by an inline compression workflow. In another example, the staging processing for small and large file may be different. For instance, the data blocks associated with small files may be staged separately from data blocks associated with large files and/or data blocks of multiple small files may be staged within the same ISE inode.
1 . A method comprising:
as data blocks of a plurality of write requests are received from a client of a storage system;
initializing a timer for each large file of a plurality of large files to which a write request of the plurality of write requests is directed; and
for a given file of the plurality of large files, staging a plurality of non-duplicative data blocks of the data blocks associated with the given file until a trigger event occurs, wherein each large file of the plurality of large files has a size greater than or equal to a predetermined or configurable number of kilobytes (KB) corresponding to a compression group size utilized by the storage system for performing inline compression, wherein the trigger event comprises:
the timer for the given file indicating a predefined or configurable time threshold has expired; or
a number of the staged plurality of the non-duplicative data blocks associated with the given file meeting a predetermined or configurable threshold;
after occurrence of the trigger event, generating an inline compression group of the compression group size by coalescing a group of multiple of the staged plurality of the non-duplicative data blocks associated with the given file; and
prior to persisting data associated with the inline compression group to a data storage medium of the storage system, performing compression based on the inline compression group.
2 . The method of claim 1 , wherein each of the write requests includes one or two data blocks having a size of less than 32 kilobytes (KB) and wherein the compression group size is greater than or equal to 32 KB.
3 . The method of claim 1 , wherein the staging includes creating a list of inline storage efficiency (ISE) index nodes (inodes) to track compression work in a memory of the storage system, wherein there is a one-to-one correspondence between the ISE inodes and files for which write have been received.
4 . A method comprising:
as data blocks of a plurality of write requests are received from a client of a storage system;
initializing a timer for a plurality of small files to which write requests of the plurality of write requests are directed; and
staging a plurality of non-duplicative data blocks of the data blocks spanning the plurality of files, until a trigger event occurs, wherein each small file of the plurality of small files has a size of less than or equal to a predetermined or configurable number of kilobytes (KB) corresponding to a compression group size utilized by the storage system for performing inline compression, wherein the trigger event comprises:
the timer indicating a predefined or configurable time threshold has elapsed since a first of the write requests was received; or
a number of the staged plurality of the non-duplicative data blocks associated with the plurality of small files meeting a predetermined or configurable threshold;
after occurrence of the trigger event, generating an inline compression group of the compression group size by coalescing a group of multiple of the staged plurality of the non-duplicative data blocks; and
prior to persisting data associated with the inline compression group to a data storage medium of the storage system, performing compression based on the inline compression group.
5 . The method of claim 4 , wherein each of the write requests includes one or two data blocks having a size of less than 32 kilobytes (KB) and wherein the compression group size is greater than or equal to 32 KB.
6 . The method of claim 4 , wherein the staging includes creating a list of inline storage efficiency (ISE) index nodes (inodes) to track compression work in a memory of the storage system.
7 . The method of claim 6 , wherein a number of the ISE inodes is limited and is potentially fewer than a number of files for which writes have been received.
8 . A non-transitory machine readable medium storing instructions, which when executed by one or more processing resources of a storage system, cause the storage system to:
as data blocks of a plurality of write requests are received from a client of the storage system;
initializing a timer for each large file of a plurality of large files to which a write request of the plurality of write requests is directed; and
for a given file of the plurality of large files, stage a plurality of non-duplicative data blocks of the data blocks associated with the given file until a trigger event occurs, wherein each large file of the plurality of large files has a size greater than or equal to a predetermined or configurable number of kilobytes (KB) corresponding to a compression group size utilized by the storage system for performing inline compression, wherein the trigger event comprises:
the timer for the given file indicating a predefined or configurable time threshold has expired; or
a number of the staged plurality of the non-duplicative data blocks associated with the given file meeting a predetermined or configurable threshold;
after occurrence of the trigger event, generate an inline compression group of the compression group size by coalescing a group of multiple of the staged plurality of the non-duplicative data blocks associated with the given file; and
prior to persisting data associated with the inline compression group to a data storage medium of the storage system, perform compression based on the inline compression group.
9 . The non-transitory machine readable medium of claim 8 , wherein each of the write requests includes one or two data blocks having a size of less than 32 kilobytes (KB) and wherein the compression group size is greater than or equal to 32 KB.
10 . The non-transitory machine readable medium of claim 8 , wherein the staging includes creating a list of inline storage efficiency (ISE) index nodes (inodes) to track compression work in a memory of the storage system, wherein there is a one-to-one correspondence between the ISE inodes and files for which writes have been received.
11 . A storage system comprising:
one or more processing resources; and
instructions that when executed by the one or more processing resources cause the storage system to:
as data blocks of a plurality of write requests are received from a client of a storage system;
initialize a timer for a plurality of small files to which write requests of the plurality of write requests are directed; and
stage a plurality of non-duplicative data blocks of the data blocks spanning the plurality of files, until a trigger event occurs, wherein each small file of the plurality of small files has a size of less than or equal to a predetermined or configurable number of kilobytes (KB) corresponding to a compression group size utilized by the storage system for performing inline compression, wherein the trigger event comprises:
the timer satisfies a predefined or configurable time threshold; or
a number of the staged plurality of the non-duplicative data blocks associated with the plurality of small files meeting a predetermined or configurable threshold;
after occurrence of the trigger event, generate an inline compression group of the compression group size by coalescing a group of multiple of the staged plurality of the non-duplicative data blocks; and
prior to persisting data associated with the inline compression group to a data storage medium of the storage system, perform compression based on the inline compression group.
12 . The storage system of claim 11 , wherein each of the write requests includes one or two data blocks having a size of less than 32 kilobytes (KB) and wherein the compression group size is greater than or equal to 32 KB.
13 . The storage system of claim 11 , wherein the staging includes creating a list of inline storage efficiency (ISE) index nodes (inodes) to track compression work in a memory of the storage system.
14 . The storage system of claim 13 , wherein a number of the ISE inodes is limited and is potentially fewer than a number of files for which writes have been received.