Management of data staging for writing to object based storage
A method, computing device, and non-transitory machine-readable medium for managing incoming/outgoing flow through a staging area. A request to write data to an object based storage is received. A determination is made that the data is to be first staged within a transfer data structure. The transfer data structure is in a first storage tier and the object based storage is in a second storage tier. A determination is made that an amount of storage space used in the transfer data structure exceeds a start throttle threshold. In response to the determination that the amount of storage space exceeds the start throttle threshold, the request is sent into a queue selected from a set of throttling queues, wherein requests in the queue are handled in a first in, first out (FIFO) order. The request in the queue is processed based on the FIFO order and a dynamic throttle rate.
1 . A method implemented by a processor executing instructions out of a memory for a storage operating system, the method comprising:
receiving a request to write data to an object based storage;
determining that the data is to be staged within a transfer data structure before being written to the object based storage,
wherein the transfer data structure is in a first storage tier and the object based storage is in a second storage tier;
determining that an amount of storage space used in the transfer data structure exceeds a start throttle threshold;
sending, in response to a determination the amount of storage space used exceeds the start throttle threshold, the request into a queue selected from a set of throttling queues, wherein requests in the queue are handled in a first in, first out (FIFO) order;
processing the request in the queue based on the FIFO order and a dynamic throttle rate; and
adjusting a drain throttle rate at which the data is transferred out of the transfer data structure based on whether the data to be written is first stored in a physical storage in the first storage tier separate from the transfer data structure.
2 . The method of claim 1 , further comprising:
halting throttling in response to determining that the amount of storage space used in the transfer data structure has fallen below a stop throttle threshold; and
processing any remaining requests in the set of throttling queues according to the FIFO order.
3 . The method of claim 1 , wherein the set of throttling queues includes a normal queue and a high priority queue and wherein the normal queue is processed at a slower rate than the high priority queue.
4 . The method of claim 1 , wherein the set of throttling queues includes a normal queue and a high priority queue; wherein a first amount of space within the transfer data structure is designated for processing requests in the normal queue and a second amount of space within the transfer data structure is designated for processing requests in the high priority queue; and wherein the second amount of space is greater than the first amount of space.
5 . The method of claim 1 , further comprising:
determining that a source of the data is one of a set of sources designated as being subject to throttling prior to sending the request into the queue selected from the set of throttling queues.
6 . The method of claim 1 , further comprising:
computing a dynamic drain rate for the transfer data structure; and
identifying the dynamic throttle rate based on the dynamic drain rate.
7 . The method of claim 1 , wherein processing the request comprises:
processing the request in the queue based on the FIFO order, the dynamic throttle rate, and a priority type of the queue.
8 . The method of claim 1 , wherein the request is a write request received from a client.
9 . The method of claim 1 , wherein, prior to be written to the transfer data structure, the data is stored in the physical storage in the first storage tier and the object based storage comprises a cloud storage.
10 . The method of claim 1 , wherein the request is received from a software-based process that comprises at least one of an application, a process internal to the storage operating system, a process associated with the storage operating system, a third-party process, a piece of code, a model, an algorithm, or a policy-based process.
11 . A computing device comprising:
a memory containing a machine-readable medium having instructions stored thereon comprising machine executable code; and
a processor coupled to the memory, the processor configured to execute the machine executable code to:
receive a request to write data to an object based storage;
determine that the data is to be staged within a transfer data structure before being written to the object based storage,
wherein the transfer data structure is in a first storage tier and the object based storage is in a second storage tier;
determine that an amount of storage space used in the transfer data structure exceeds a start throttle threshold;
send, in response to a determination the amount of storage space used exceeds the start throttle threshold, the request into a queue selected from a set of throttling queues;
process the request in the queue based on a dynamic throttle rate; and
adjust a drain throttle rate at which the data is transferred out of the transfer data structure based on whether the data to be written is first stored in a physical storage in the first storage tier separate from the transfer data structure.
12 . The computing device of claim 11 , wherein the processor is further configured to halt throttling in response to determining that the amount of storage space used in the transfer data structure has fallen below a stop throttle threshold; and process any remaining requests in the set of throttling queues according to a first in, first out (FIFO) order.
13 . The computing device of claim 11 , wherein the set of throttling queues includes a normal queue and a high priority queue and wherein the normal queue is processed at a slower rate than the high priority queue.
14 . The computing device of claim 11 , wherein the set of throttling queues includes a normal queue and a high priority queue; wherein a first amount of space within the transfer data structure is designated for processing requests in the normal queue and a second amount of space within the transfer data structure is designated for processing requests in the high priority queue; and wherein the second amount of space is greater than the first amount of space.
15 . The computing device of claim 11 , wherein the processor is further configured to determine that a source of the data is one of a set of sources designated as being subject to throttling prior to sending the request into the queue selected from the set of throttling queues.
16 . The computing device of claim 11 , wherein the dynamic throttle rate is identified based on a dynamic drain rate that is computed for the transfer data structure.
17 . The computing device of claim 11 , wherein processing the request comprises:
processing the request in the queue based on the dynamic throttle rate, a first in, first out (FIFO) order, and a priority type of the queue.
18 . A non-transitory machine-readable medium having stored thereon instructions for performing a method comprising machine-executable code which, when executed by at least one machine, causes the at least one machine to:
receive a request to stage data in a transfer data structure in a first storage tier before the data is written to object based storage in a second storage tier;
determine that an amount of storage space used in the transfer data structure exceeds a start throttle threshold;
send, in response to a determination the amount of storage space used exceeds the start throttle threshold, the request into a queue selected from a set of throttling queues;
identify a dynamic throttle rate based on a dynamic drain rate computed for the transfer data structure;
process the request in the queue based on the dynamic throttle rate, a priority type of the queue, and a first in, first out (FIFO) order; and
adjust a drain throttle rate at which the data is transferred out of the transfer data structure based on whether the data to be written is first stored in a physical storage in the first storage tier separate from the transfer data structure.
19 . The non-transitory machine-readable medium of claim 18 , wherein the machine-executable code which, when executed by the at least one machine, further causes the at least one machine to halt throttling in response to determining that the amount of storage space used in the transfer data structure has fallen below a stop throttle threshold; and process any remaining requests in the set of throttling queues according to the FIFO order.
20 . The non-transitory machine-readable medium of claim 18 , wherein the set of throttling queues includes a normal queue and a high priority queue and wherein the normal queue is processed at a slower rate than the high priority queue.