Reducing file write latency
Reducing file write latency includes receiving incoming data, from a data source, for storage in a file and a target storage location for the incoming data, and determining whether the target storage location corresponds to a cache entry. Based on at least the target storage location not corresponding to a cache entry, the incoming data is written to a block pre-allocated for cache misses and the writing of the incoming data to the pre-allocated block is journaled. The writing of the incoming data is acknowledged to the data source. A process executing in parallel with the above commits the incoming data in the pre-allocated block with the file. Using this parallel process to commit the incoming data in the file removes high-latency operations (e.g., reading pointer blocks from the storage media) from a critical input/output path and results in more rapid write acknowledgement.
1. A method of reducing file write latency, the method comprising:
pre-allocating at least one block for cache misses;
receiving, from a data source, (a) incoming data for storage in a file and (b) a target storage location for the incoming data;
determining whether the target storage location corresponds to any cache entry of a plurality of cache entries;
based on at least the target storage location not corresponding to any of the plurality of cache entries:
writing the incoming data to the pre-allocated block, and
journaling the writing of the incoming data to a journal; and
acknowledging, to the data source, the writing of the incoming data, wherein a process executing in parallel with the method is configured to commit the incoming data in the pro-allocated block with the file.
2. The method of claim 1 , wherein pre-allocating at least one block for cache misses comprises:
pre-allocating a pool of blocks for the cache misses; and
dynamically adjusting a sit c5f the pre-allocated pool based on a latency associated with availability of at least one of the blocks within the pre-allocated pool for storing the incoming data.
3. The method of claim 1 , further comprising:
based on at least the target storage location corresponding to a cache entry, writing the incoming data to a block indicated by the cache entry.
4. The method of claim 1 , further comprising:
determining whether the incoming data is aligned to write to an entire block; and
based on at least the incoming data not being aligned to write to an entire block, recording a length of the incoming data and an internal block offset within the journaling.
5. The method of claim 1 , wherein commuting the incoming data comprises:
changing a pointer block entry from pointing to a prior block of the file to pointing to the pre-allocated block; or
copying the incoming data from the pre-allocated block to the prior block.
6. The method of claim 5 , further comprising:
based at least on completing the committing, freeing the pre-allocated block or the prior block.
7. The method of claim 1 , further comprising:
monitoring for a commit trigger event;
detecting the commit trigger event; and
performing the committing in response to the detected commit trigger event.
8. A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code executable by the processor, the program code causing the processor to:
pre-allocate at least one block for cache misses;
receive, from a data source, (a) incoming data for storage in a file and (b) a target storage location for the incoming data;
determine whether the target storage location corresponds to any cache entry of a plurality of cache entries;
based on at least the target storage location not corresponding to any of the plurality of cache entries:
write the incoming data to the pre-allocated block, and
journal the writing of the incoming data to a journal; and
acknowledge, to the data source, the writing of the incoming data, wherein a process executing in parallel with the program code is configured to commit the incoming data in the pre-allocated block with the file.
9. The computer system of claim 8 , wherein the program code causes the processor to pre-allocate at least one block for cache misses by:
pre allocating a pool of blocks for the cache misses; and
dynamically adjusting a size of the pre-allocated pool based on a latency associated with availability of at least one of the blocks within the 3re-allocated pool for storing the incoming data.
10. The computer system of claim 8 , wherein the program code is further operative to:
based on at least the target storage location corresponding to a cache entry, write the incoming, data to a block indicated by the cache entry.
11. The computer system of claim 8 , wherein committing the incoming data comprises:
changing a pointer block entry from pointing to a prior block of the file to pointing to the pre- allocated block; or
copying the incoming data from the pre-allocated block to the prior block.
12. The computer system of claim 11 , wherein the program code is further operative to:
based at least on completing the committing, free the pre-allocated block or the prior block.
13. The computer system of claim 8 , wherein the program code is further operative to:
monitor for a commit trigger event;
detect the commit trigger event; and
perform the commit in response to the detected commit trigger event.
14. The computer system of claim 13 , wherein the commit trigger event comprises a count of pending journal entries meeting a commit trigger threshold, and wherein the program code is further operative to:
dynamically adjust the commit trigger threshold based on burstiness of the incoming data.
15. A non-transitory computer storage medium having stored thereon program code executable by a processor, the program code embodying a method comprising:
pre-allocating at least one block for cache misses;
receiving, from a data source, (a) incoming data for storage in a file and (b) a target storage location for the incoming data;
determining whether the target storage location corresponds to any cache entry of a plurality of cache entries;
based on at least the target storage location not corresponding to any of the plurality of cache entries:
writing the incoming data to the pre-allocated block, and
journaling the writing of the incoming data to a journal; and
acknowledging, to the data source, the writing of the incoming data, wherein a process executing in parallel with the program code is configured to commit the incoming data in the pre-allocated block with the file.
16. The computer storage medium of claim 15 , wherein pre-allocating at least one block for cache misses comprises:
pre-allocating a pool of blocks for the cache misses; and
dynamically adjusting a size of the pre-allocated pool based on a latency associated with availability of at least one of the blocks within the pre-allocated pool for storing the incoming data.
17. The computer storage medium of claim 15 , wherein the program code further comprises:
based on at least the target storage location corresponding to a cache entry, writing the incoming data to a block indicated by the cache entry.
18. The computer storage medium of claim 15 , wherein the program code further comprises:
determining whether the incoming data is aligned to write to an entire block; and
based on at least the incoming data not being aligned to write to an entire block, recording a length of the incoming data and an internal block offset within the journaling.
19. The computer storage medium of claim 15 , wherein committing the incoming data comprises:
changing a pointer block entry from pointing to a prior block of the file to pointing to the pre-allocated block; or
copying the incoming data from the pre-allocated block to the prior block.
20. The computer storage medium of claim 19 , wherein the program code further comprises:
based at least on completing the committing, freeing the pre allocated block or the prior block.