Flow control with a dynamic window in a storage system with latency guarantees

1. A method comprising:

receiving one or more input/output (I/O) requests by a storage system having at least one storage cluster, each of the received I/O requests associated with a storage cluster;

performing the I/O request with dynamic flow control by:

determining a latency associated with the one or more received I/O requests during at least one monitoring interval;

tracking I/O requests to the storage cluster;

if a received I/O request exceeds a choker threshold value of the storage cluster, queuing the received I/O request, otherwise, performing the received I/O request,

wherein determining the latency associated with the one or more received I/O requests during at least one monitoring interval comprises:

determining an end-to-end latency of each of the one or more received I/O requests performed for the storage cluster during the monitoring interval; and

determining latency for the storage cluster as at least one of: a peak latency for the storage cluster during the monitoring interval, and an average latency for the storage cluster during the monitoring interval; and

based upon the determined latency for the storage cluster, adjusting the choker threshold value associated with the storage cluster, wherein adjusting the choker threshold value comprises:

if the determined latency for the storage cluster is below a first latency threshold value, increasing the choker threshold value by a first step value; and

if the determined latency for the storage cluster is above a second latency threshold value, decreasing the choker threshold value by a second step value,

wherein the choker threshold value of the storage cluster comprises one of: a maximum number of concurrent I/O requests that can be processed by the storage cluster, and a maximum amount of data that can be processed by the storage cluster.

2. The method of claim 1 , wherein the first latency threshold value and the second latency threshold value comprise the same value.

3. The method of claim 2 , wherein the second step value is larger than the first step value.

4. The method of claim 1 , wherein queuing the received I/O request comprises:

adding the received I/O request to a queue corresponding to the storage cluster associated with the I/O request; and

rejecting any queued I/O requests that have reached a retention interval value associated with the storage cluster.

5. The method of claim 1 , further comprising, based upon one or more operating conditions of the storage system, adjusting the retention interval value.

6. A system comprising:

a processor; and

memory storing computer program code that when executed on the processor causes the processor to execute an input/output (I/O) request received by a storage system having at least one storage cluster operable to perform the operations of:

performing the I/O request with dynamic flow control by:

determining a latency associated with the one or more received I/O requests during at least one monitoring interval;

tracking I/O requests to the storage cluster; and

if a received I/O request exceeds a choker threshold value of the storage cluster, queuing the received I/O request, otherwise, performing the received I/O request, wherein determining the latency associated with the one or more received I/O requests during at least one monitoring interval comprises:

determining an end-to-end latency of each of the one or more received I/O requests performed for the storage cluster during the monitoring interval; and

determining latency for the storage cluster as at least one of: a peak latency for the storage cluster during the monitoring interval, and an average latency for the storage cluster during the monitoring interval; and

based upon the determined latency for the storage cluster, adjusting the choker threshold value associated with the storage cluster, wherein adjusting the choker threshold value comprises:

if the determined latency for the storage cluster is below a first latency threshold value, increasing the choker threshold value by a first step value; and

if the determined latency for the storage cluster is above a second latency threshold value, decreasing the choker threshold value by a second step value,

wherein the choker threshold value of the storage cluster comprises one of: a maximum number of concurrent I/O requests that can be processed by the storage cluster, and a maximum amount of data that can be processed by the storage cluster.

7. The system of claim 6 , wherein the first latency threshold value and the second latency threshold value comprise the same value.

8. The system of claim 6 , wherein the second step value is larger than the first step value.

9. The system of claim 6 , wherein the system is further operable to perform the operations of:

adding the received I/O request to a queue corresponding to the storage cluster;

rejecting any queued I/O requests that have reached a retention interval value associated with the storage cluster; and

based upon one or more operating conditions of the storage system, adjusting the retention interval value.

10. A computer program product including a non-transitory computer readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to execute an input/output (I/O) request received by a storage system having at least one storage cluster, the computer program product comprising:

computer program code for receiving one or more input/output (I/O) requests; and

computer program code for performing the I/O request with dynamic flow control by:

determining a latency associated with the one or more received I/O requests during at least one monitoring interval;

tracking I/O requests to the storage cluster; and

if a received I/O request exceeds a choker threshold value of the storage cluster, queuing the received I/O request, otherwise, performing the received I/O request, wherein determining the latency associated with the one or more received I/O requests during at least one monitoring interval comprises:

determining an end-to-end latency of each of the one or more received I/O requests performed for the storage cluster during the monitoring interval; and

determining latency for the storage cluster as at least one of: a peak latency for the storage cluster during the monitoring interval, and an average latency for the storage cluster during the monitoring interval; and

based upon the determined latency for the storage cluster, adjusting the choker threshold value associated with the storage cluster, wherein adjusting the choker threshold value comprises:

if the determined latency for the storage cluster is below a first latency threshold value, increasing the choker threshold value by a first step value; and

if the determined latency for the storage cluster is above a second latency threshold value, decreasing the choker threshold value by a second step value,

wherein the choker threshold value of a storage cluster comprises one of: a maximum number of concurrent I/O requests that can be processed by the storage cluster, and a maximum amount of data that can be processed by the storage cluster.

11. The computer program product of claim 10 , further comprising:

computer program code for adding the received I/O request to a queue corresponding to the storage cluster;

computer program code for rejecting any queued I/O requests that have reached a retention interval value associated with the storage cluster; and

computer program code for adjusting the retention interval value based upon one or more operating conditions of the storage system.