Adaptively over-allocating storage space for compressed data
A technique for managing storage of compressed data includes generating and enforcing a minimum slot size requirement. The minimum slot size is based at least in part on collected performance metrics that indicate a degree of compressibility of data received, compressed, and written by the data storage system. As new data arrive, the new data are compressed and stored in slots at least as big as the minimum slot size, in many cases effecting an over-allocation of storage space and improving the likelihood that subsequent overwrites will fit into existing slots.
1. A method of managing storage of compressed data in a data storage system, the method comprising:
processing a set of overwrite requests, each of the set of overwrite requests directed to a respective address of the data storage system and specifying a block of data to be written, in compressed form, as a replacement for a compressed block already stored at the respective address, each compressed block stored in a respective slot of allocated storage space;
while processing the set of overwrite requests, determining compressed sizes of data received, compressed, and written by the data storage system; and
enforcing an allocation policy to maintain a minimum slot size when allocating slots for storing newly arriving data, the minimum slot size based at least in part on the compressed sizes determined while processing the set of overwrite requests and sized to provide a margin of over-allocated storage space of at least one sector (512 bytes) for at least some of the newly arriving data,
wherein the method further comprises:
initially disabling the allocation policy for a particular LUN;
monitoring a fragment-fill failure ratio (FFFR) for the particular LUN, the FFFR providing a ratio of slot-overwrite failures to total overwrite attempts, each slot-overwrite failure representing an overwrite of compressed data that is too big to fit into a slot that stores respective data being replaced; and
selectively enabling the allocation policy for the particular LUN in response to the FFFR for the particular LUN exceeding a predetermined threshold.
2. The method of claim 1 , further comprising:
receiving an overwrite request specifying a new block of data directed to an address mapped to a target slot that was allocated in accordance with the allocation policy, the target slot storing an existing compressed block having a size;
compressing the new block to produce a new compressed block, the new compressed block having a size that is larger than the size of the existing compressed block by at least one sector; and
replacing the existing compressed block in the target slot with the new compressed block, the new compressed block fitting into the target slot on account of the over-allocation of storage space effected when allocating the target slot.
3. The method of claim 2 , further comprising generating, based on the determined compressed sizes, a representative difference in size between compressed blocks being written as specified by the set of overwrite requests and the compressed data being replaced by the overwrite requests, wherein the minimum slot size is based at least in part on the representative difference.
4. The method of claim 2 , wherein generating the minimum slot size is performed on a per-LUN (Logical UNit) basis, such that each of multiple LUNs hosted by the data storage system has its own respective minimum slot size to be used when allocating slots into which to place compressed data of the respective LUN.
5. The method of claim 4 , further comprising updating the minimum slot size for a particular LUN over time to reflect changes in compressibility of data being written to the particular LUN.
6. The method of claim 5 , wherein updating the minimum slot size for the particular LUN is performed only after the data storage system has received a predetermined minimum number of I/O requests directed to the particular LUN and specifying overwrites.
7. The method of claim 5 , wherein generating the minimum slot size for the particular LUN is based upon both:
(i) a representative size of compressed blocks written to the particular LUN; and
(ii) a representative difference in size between compressed blocks written to the particular LUN and respective, compressed data being replaced by the compressed blocks written to the particular LUN.
8. The method of claim 7 , wherein the representative difference is generated as a median size difference expressed as an integer number of sectors.
9. The method of claim 5 , wherein the predetermined threshold is a first threshold and wherein the method further comprises selectively disabling the allocation policy for the particular LUN in response to the FFFR for the particular LUN falling below a second threshold that is less than the first threshold.
10. The method of claim 5 , wherein the data storage system realizes the particular LUN within a file of a file system, and wherein the method further comprises persistently storing metadata indicative of the minimum slot size of the particular LUN in a superblock of the file system.
11. The method of claim 10 , wherein storing the metadata in the superblock is performed only when performing an operation that takes the particular LUN offline.
12. The method of claim 1 , further comprising dynamically varying the minimum slot size over time based at least in part on the compressed sizes determined while processing the set of overwrite requests.
13. A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
process a set of overwrite requests, each of the set of overwrite requests directed to a respective address of the data storage system and specifying a block of data to be written, in compressed form, as a replacement for a compressed block already stored at the respective address, each compressed block stored in a respective slot of allocated storage space;
while processing the set of overwrite requests, determine compressed sizes of data received, compressed, and written by the data storage system; and
enforce an allocation policy to maintain a minimum slot size when allocating slots for storing newly arriving data, the minimum slot size based at least in part on the compressed sizes determined while processing the set of overwrite requests and sized to provide a margin of over-allocated storage space of at least one sector (512 bytes) for at least some of the newly arriving data,
wherein the control circuitry is further constructed and arranged to:
initially disable the allocation policy for a particular LUN;
monitor a fragment-fill failure ratio (FFFR) for the particular LUN, the FFFR providing a ratio of slot-overwrite failures to total overwrite attempts, each slot-overwrite failure representing an overwrite of compressed data that is too big to fit into a slot that stores respective data being replaced; and
selectively enable the allocation policy for the particular LUN in response to the FFFR for the particular LUN exceeding a predetermined threshold.
14. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method for managing storage of compressed data, the method comprising:
processing a set of overwrite requests, each of the set of overwrite requests directed to a respective address of the data storage system and specifying a block of data to be written, in compressed form, as a replacement for a compressed block already stored at the respective address, each compressed block stored in a respective slot of allocated storage space;
while processing the set of overwrite requests, determining compressed sizes of data received, compressed, and written by the data storage system; and
enforcing an allocation policy to maintain a minimum slot size when allocating slots for storing newly arriving data, the minimum slot size based at least in part on the compressed sizes determined while processing the set of overwrite requests and sized to provide a margin of over-allocated storage space of at least one sector (512 bytes) for at least some of the newly arriving data,
wherein the method further comprises:
initially disabling the allocation policy for a particular LUN;
monitoring a fragment-fill failure ratio (FFFR) for the particular LUN, the FFFR providing a ratio of slot-overwrite failures to total overwrite attempts, each slot-overwrite failure representing an overwrite of compressed data that is too big to fit into a slot that stores respective data being replaced; and
selectively enabling the allocation policy for the particular LUN in response to the FFFR for the particular LUN exceeding a predetermined threshold.
15. The computer program product of claim 14 , wherein the method further comprises generating, based on the determined compressed sizes, a representative difference in size between compressed blocks being written as specified by the set of overwrite requests and the compressed data being replaced by the overwrite requests, wherein the minimum slot size is based at least in part on the representative difference.
16. The computer program product of claim 14 , wherein generating the minimum slot size is performed on a per-LUN (Logical UNit) basis, such that each of multiple LUNs hosted by the data storage system has its own respective minimum slot size to be used when allocating slots into which to place compressed data of the respective LUN.
17. The computer program product of claim 16 , wherein the method further comprises updating the minimum slot size for a particular LUN over time to reflect changes in compressibility of data being written to the particular LUN.
18. The computer program product of claim 16 , wherein generating the minimum slot size for the particular LUN is based upon both:
(i) a representative size of compressed blocks written to the particular LUN; and
(ii) a representative difference in size between compressed blocks written to the particular LUN and respective, compressed data being replaced by the compressed blocks written to the particular LUN.
19. The computer program product of claim 16 , wherein the predetermined threshold is a first threshold and wherein the method further comprises selectively disabling the allocation policy for the particular LUN in response to the FFFR for the particular LUN falling below a second threshold that is less than the first threshold.