Throttling writes with asynchronous flushing
View Patent ↗Embodiments are directed to techniques for allowing a data storage system to be able to flush data to underlying storage when the bandwidth is high without excessively impacting the maximum latency. This may be accomplished by utilizing asynchronous flushing and by throttling incoming writes by preventing too many asynchronous flushes from happening when the amount of available cache space is too small. In addition, an improved system employing Copy on First Write (CoFW) may initiate write throttling only once the amount of available write cache drops below a dynamically-calculated threshold that accounts for an amount of space actually needed to store CoFW buffers. In addition, only a portion of the write caching process is placed under the protection of a mutex or a spinlock, allowing a significant portion of the write caching for any given write operation to be performed without needing the mutex or spinlock, allowing some increased parallelization.
1. A method of efficiently processing a set of write requests received by a computerized data storage system, each write request identifying a respective set of block-aligned data to be written to a respective contiguous region of a file, the method comprising, for each of the set of write requests:
copying the block-aligned data of that write request into a file-specific ordered list of buffers;
after copying, performing a decision operation configured to (a) produce a first result when an amount of memory of the computerized data storage system available for storing buffers drops below a threshold amount and (b) produce a second result when the amount of memory of the computerized data storage system available for storing buffers exceeds the threshold amount;
in response to the decision operation producing the first result when processing a first write request, entering a waiting state before proceeding, the waiting state suspending processing of the first write request until a currently-busy asynchronous flushing thread becomes idle;
in response to the decision operation producing the second result when processing a second write request, proceeding to process the second write request without entering the waiting state;
upon proceeding for both the first write request and the second write request, configuring an available asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to underlying persistent storage; and
in response to configuring the asynchronous flushing thread, sending an acknowledgment of completion of that write request back to a client which issued that write request.
2. The method of claim 1 ,
wherein the method further comprises:
for each of the set of write requests, performing a testing operation, the testing operation configured to (a) produce a third result when a length of the file-specific ordered list of buffers exceeds a threshold length and (b) produce a fourth result when the length of the file-specific ordered list of buffers does not exceed the threshold length; and
in response to the testing operation producing the fourth result when processing a third write request, directly sending acknowledgment of completion of the fourth write request back to a client which issued the fourth write request without configuring any asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to underlying persistent storage; and
wherein the decision operation is performed in response to the testing operation producing the third result when processing the first and second write requests.
3. The method of claim 2 wherein the method further comprises, for each of the set of write requests, prior to copying the block-aligned data of that write request into the file-specific ordered list of buffers and upon determining that the length of the file-specific ordered list of buffers exceeds a threshold length, pausing that write request until the length of the file-specific ordered list of buffers no longer exceeds the threshold length.
4. The method of claim 1 wherein performing the decision operation includes:
calculating the amount of memory of the computerized data storage system available for storing buffers by:
determining a net Copy on First Write (CoFW) reservation by subtracting (i) an amount of memory space already allocated towards CoFW buffers from (ii) an amount of memory space that would be required to store copies of all uncached shared data blocks that have been overwritten and not yet flushed; and
subtracting the determined net CoFW reservation from a total amount of free memory, yielding the amount of memory of the computerized data storage system available for storing buffers; and
comparing the calculated amount of memory of the computerized data storage system available for storing buffers to the threshold amount, the threshold amount being a threshold percentage of total system memory.
5. The method of claim 4 wherein the threshold percentage is within a range from 40 percent to 50 percent.
6. The method of claim 5 ,
wherein the decision operation is further configured to (c) produce a third result when the amount of memory of the computerized data storage system available for storing buffers drops below a criticality threshold, the criticality threshold being within a range of 15 percent to 25 percent of total system memory; and
wherein the method further comprises, in response to the decision operation producing the third result when processing a third write request, triggering a synchronous flush without proceeding to configure an available asynchronous flushing thread to asynchronously flush data buffers.
7. The method of claim 1 ,
wherein copying the block-aligned data of that write request into the file-specific ordered list of buffers includes:
performing a looped operation, wherein performing the looped operation includes, for each block-aligned page of that write request, starting with a block-aligned page corresponding to an initial address of the contiguous region and proceeding with consecutively-addressed block-aligned pages of that write request:
copying data of that block-aligned page into a buffer; and
inserting that buffer onto a tail of a temporary list of buffers specific to that write request;
subsequent to performing the looped operation, obtaining a lock on the file of that write request; and
subsequent to obtaining the lock, merging the temporary list of buffers into the file-specific ordered list of buffers; and
the method further comprises, subsequent to sending the acknowledgment of completion of that write request back to the client which issued that write request, releasing the lock on the file of that write request.
8. The method of claim 7 wherein the lock is a mutex.
9. The method of claim 7 wherein the lock is a spinlock.
10. The method of claim 7 wherein performing the looped operation further includes, for a particular block-aligned page of that write request:
determining that the particular block-aligned page corresponds to a region of the file of that write request that shares its data with a checkpointed version of the file; and
in response to determining, increasing a Copy on First Write (CoFW) reservation, the CoFW reservation ensuring that a CoFW read buffer will be available upon flushing to temporarily store the shared data until it can be inserted directly into the checkpointed version of the file.
11. An apparatus comprising:
persistent storage hardware;
network interface circuitry for communicating with a remote host; and
multi-core processing circuitry coupled to memory configured to efficiently process a write request received by the network interface circuitry from the remote host, the write request identifying a respective set of block-aligned data to be written to a respective contiguous region of a file, by:
copying the block-aligned data of the write request into a file-specific ordered list of buffers;
after copying, performing a decision operation configured to (a) produce a first result when an amount of memory of the apparatus available for storing buffers drops below a threshold amount and (b) produce a second result when the amount of memory of the apparatus available for storing buffers exceeds the threshold amount;
when the decision operation produces the first result, entering a waiting state before proceeding, the waiting state suspending processing of the write request until a currently-busy asynchronous flushing thread becomes idle;
when the decision operation produces the second result, proceeding to process the write request without entering the waiting state;
upon proceeding, configuring an available asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to the persistent storage hardware; and
in response to configuring the asynchronous flushing thread, sending an acknowledgment of completion of the write request back to the remote host.
12. The apparatus of claim 11 ,
wherein the multi-core processing circuitry coupled to memory is further configured to efficiently process the write request received by the network interface circuitry from the remote host by:
for each of the set of write requests, performing a testing operation, the testing operation configured to (a) produce a third result when a length of the file-specific ordered list of buffers exceeds a threshold length and (b) produce a fourth result when the length of the file-specific ordered list of buffers does not exceed the threshold length; and
in response to the testing operation producing the fourth result when processing a third write request, directly sending acknowledgment of completion of the fourth write request back to the remote host without configuring any asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to underlying persistent storage; and
wherein the decision operation is performed in response to the testing operation producing the third result when processing the first and second write requests.
13. The apparatus of claim 11 wherein performing the decision operation includes:
calculating the amount of memory of the apparatus available for storing buffers by:
determining a net Copy on First Write (CoFW) reservation by subtracting (i) an amount of memory space already allocated towards CoFW buffers from (ii) an amount of memory space that would be required to store copies of all uncached shared data blocks that have been overwritten and not yet flushed; and
subtracting the determined net CoFW reservation from a total amount of free memory, yielding the amount of memory of the apparatus available for storing buffers; and
comparing the calculated amount of memory of the apparatus available for storing buffers to the threshold amount, the threshold amount being a threshold percentage of total system memory.
14. The apparatus of claim 11 ,
wherein copying the block-aligned data of that write request into the file-specific ordered list of buffers includes:
performing a looped operation, wherein performing the looped operation includes, for each block-aligned page of that write request, starting with a block-aligned page corresponding to an initial address of the contiguous region and proceeding with consecutively-addressed block-aligned pages of that write request:
copying data of that block-aligned page into a buffer; and
inserting that buffer onto a tail of a temporary list of buffers specific to that write request;
subsequent to performing the looped operation, obtaining a lock on the file of that write request; and
subsequent to obtaining the lock, merging the temporary list of buffers into the file-specific ordered list of buffers; and
wherein the multi-core processing circuitry coupled to memory is further configured to efficiently process the write request received by the network interface circuitry from the remote host by, subsequent to sending the acknowledgment of completion of that write request back to the remote host, releasing the lock on the file of that write request.
15. A computer program product comprising a tangible non-transitory computer-readable storage medium storing instructions, which, when executed by a computerized data storage system, cause the computerized data storage system to efficiently process a write request received from a remote host, the write request identifying a respective set of block-aligned data to be written to a respective contiguous region of a file, by:
copying the block-aligned data of the write request into a file-specific ordered list of buffers;
after copying, performing a decision operation configured to (a) produce a first result when an amount of memory of the apparatus available for storing buffers drops below a threshold amount and (b) produce a second result when the amount of memory of the apparatus available for storing buffers exceeds the threshold amount;
when the decision operation produces the first result, entering a waiting state before proceeding, the waiting state suspending processing of the write request until a currently-busy asynchronous flushing thread becomes idle;
when the decision operation produces the second result, proceeding to process the write request without entering the waiting state;
upon proceeding, configuring an available asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to underlying persistent storage; and
in response to configuring the asynchronous flushing thread, sending an acknowledgment of completion of the write request back to the remote host.
16. The computer program product of claim 15 ,
wherein the instructions, when executed by the computerized data storage system, further cause the computerized data storage system to efficiently process the write request received from the remote host by:
for each of the set of write requests, performing a testing operation, the testing operation configured to (a) produce a third result when a length of the file-specific ordered list of buffers exceeds a threshold length and (b) produce a fourth result when the length of the file-specific ordered list of buffers does not exceed the threshold length; and
in response to the testing operation producing the fourth result when processing a third write request, directly sending acknowledgment of completion of the fourth write request back to the remote host without configuring any asynchronous flushing thread to asynchronously flush data buffers from the file-specific ordered list to underlying persistent storage; and
wherein the decision operation is performed in response to the testing operation producing the third result when processing the first and second write requests.
17. The computer program product of claim 15 wherein performing the decision operation includes:
calculating the amount of memory of the computerized data storage system available for storing buffers by:
determining a net Copy on First Write (CoFW) reservation by subtracting (i) an amount of memory space already allocated towards CoFW buffers from (ii) an amount of memory space that would be required to store copies of all uncached shared data blocks that have been overwritten and not yet flushed; and
subtracting the determined net CoFW reservation from a total amount of free memory, yielding the amount of memory of the computerized data storage system available for storing buffers; and
comparing the calculated amount of memory of the computerized data storage system available for storing buffers to the threshold amount, the threshold amount being a threshold percentage of total system memory.
18. The computer program product of claim 15 ,
wherein copying the block-aligned data of that write request into the file-specific ordered list of buffers includes:
performing a looped operation, wherein performing the looped operation includes, for each block-aligned page of that write request, starting with a block-aligned page corresponding to an initial address of the contiguous region and proceeding with consecutively-addressed block-aligned pages of that write request:
copying data of that block-aligned page into a buffer; and
inserting that buffer onto a tail of a temporary list of buffers specific to that write request;
subsequent to performing the looped operation, obtaining a lock on the file of that write request; and
subsequent to obtaining the lock, merging the temporary list of buffers into the file-specific ordered list of buffers; and
wherein the instructions, when executed by the computerized data storage system, further cause the computerized data storage system to efficiently process the write request received from the remote host by, subsequent to sending the acknowledgment of completion of that write request back to the remote host, releasing the lock on the file of that write request.