Method and apparatus for dynamic flow control in distributed storage systems

1. A method for processing I/O requests that are received at a distributed storage system including a plurality of receiver nodes, a plurality of first nodes, and a plurality of second nodes, the method comprising:

receiving, at a receiver node, an I/O request and executing the I/O request by decomposing the I/O request into a plurality of first internal commands and providing each of the first internal commands to any of a plurality of first nodes, each of the plurality of first nodes being configured to decompose any received first internal commands into one or more second internal commands and provide, based on a table, each of the second internal commands to one of a plurality of second nodes, the table being arranged to map each of a plurality of logical address spaces to a respective one of the plurality of second nodes that is assigned to manage the logical address space, each of the plurality of logical address spaces including a set of addresses that are managed by the same second node;

receiving, at the receiver node, one or more first latency metrics, each of the first latency metrics being generated by a respective one of the plurality of first nodes and indicating, at least partially, a time taken by the respective one of the plurality of first nodes to execute a respective first internal command that is provided to the respective one of the plurality of first nodes by the receiver node;

receiving, at the receiver node, one or more second latency metrics, each of the second latency metrics being generated by a respective one of the plurality of second nodes and indicating, at least partially, a time taken by the respective one of the plurality of second nodes to execute a respective second internal command that is provided to the respective one of the plurality of second nodes by one of the plurality of first nodes, the respective second internal command being generated by decomposing one of the plurality of first internal commands; and

reconfiguring the storage system based on any of the first latency metrics and the second latency metrics, wherein reconfigured the storage system includes changing a size of at least one of the plurality of logical address spaces.

2. The method of claim 1 , wherein:

the receiver node is configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be processed concurrently by the receiver node,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size of the receiver node based on the plurality of path-specific latency metrics.

3. The method of claim 2 , wherein the window size of the receiver node is decreased based on a smallest one of the plurality of path-specific latency metrics being lower than a predetermined threshold.

4. The method of claim 1 , wherein:

the receiver node is configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be processed concurrently by the receiver node,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size of the receiver node based on the plurality of path-specific latency metrics.

5. The method of claim 4 , wherein the window size of the receiver node is decreased based on a smallest one of the plurality of path-specific latency metrics exceeding a predetermined threshold.

6. An apparatus for use in a distributed storage system, comprising:

a memory, and

one or more processors operatively coupled to the memory, the one or more processors being configured to perform a method comprising the steps of:

receiving an I/O request and executing the I/O request by decomposing the I/O request into a plurality of first internal commands and providing each of the first internal commands to any of a plurality of first nodes, each of the plurality of first nodes being configured to decompose any received first internal commands into one or more second internal commands and provide, based on a table, each of the second internal commands to one of a plurality of second nodes, the table being arranged to map each to the plurality of logical address spaces to a respective one of the plurality of second nodes that is assigned to manage the logical address space, each of the plurality of logical address spaces including a set of addresses that are managed by the same second node;

receiving one or more first latency metrics, each of the first latency metrics being generated by a respective one of the plurality of first nodes and indicating, at least partially, a time taken by the respective one of the plurality of first nodes to execute a respective first internal command that is provided to the respective one of the plurality of first nodes by the one or more processors;

receiving one or more second latency metrics, each of the second latency metrics being generated by a respective one of the plurality of second nodes and indicating, at least partially, a time taken by the respective one of the plurality of second nodes to execute a respective second internal command that is provided to the respective one of the plurality of second nodes by one of the plurality of first nodes, the respective second internal command being generated by decomposing one of the plurality of first internal commands; and

reconfiguring the storage system based on any of the first latency metrics and the second latency metrics, wherein reconfiguring the storage system includes changing a size of at least one of the plurality of logical address spaces.

7. The apparatus of claim 6 , wherein:

the one or more processors are configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be queued for further processing by the one or more processors,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size based on the plurality of path-specific latency metrics.

8. The apparatus of claim 7 , wherein the window size is decreased based on a smallest one of the plurality of path-specific latency metrics being lower than a predetermined threshold.

9. The apparatus of claim 6 , wherein:

the one or more processors are configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be processed concurrently by the one or more processors,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size based on the plurality of path-specific latency metrics.

10. The apparatus of claim 9 , wherein the window size is decreased based on a smallest one of the plurality of path-specific latency metrics exceeding a predetermined threshold.

11. A non-transitory computer-readable medium configured to store one or more processor-executable instructions, which when executed by at least one processor causes the at least one processor to perform a method comprising the steps of:

receiving, at a receiver node in a distributed storage system, an I/O request and executing the I/O request by decomposing the I/O request into a plurality of first internal commands and providing each of the first internal commands to any of a plurality of first nodes, each of the plurality of first nodes being configured to decompose any received first internal commands into one or more second internal commands and provide, based on a table, each of the second internal commands to one of a plurality of second nodes, the table being arranged to map each of plurality of logical address spaces to a respective one of the plurality of second nodes that is assigned to manage the logical address space, each of the plurality of logical address spaces including a set of addresses that are managed by the same second node;

receiving, at the receiver node, one or more first latency metrics, each of the first latency metrics being generated by a respective one of the plurality of first nodes and indicating, at least partially, a time taken by the respective one of the plurality of first nodes to execute a respective first internal command that is provided to the respective one of the plurality of first nodes by the receiver node;

receiving, at the receiver node, one or more second latency metrics, each of the second latency metrics being generated by a respective one of the plurality of second nodes and indicating, at least partially, a time taken by the respective one of the plurality of second nodes to execute a respective second internal command that is provided to the respective one of the plurality of second nodes by one of the plurality of first nodes, the respective second internal command being generated by decomposing one of the plurality of first internal commands; and

reconfiguring the storage system based on any of the first latency metrics and the second latency metrics, wherein reconfiguring the storage system includes changing a size of at least one of the plurality of logical address spaces.

12. The non-transitory computer-readable medium of claim 11 , wherein:

the receiver node is configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be processed concurrently by the receiver node,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size of the receiver node based on the plurality of path-specific latency metrics.

13. The non-transitory computer-readable medium of claim 12 , wherein the window size of the receiver node is decreased based on a smallest one of the plurality of path- specific latency metrics being lower than a predetermined threshold.

14. The non-transitory computer-readable medium of claim 11 , wherein:

the receiver node is configured to receive the I/O request in accordance with a window size that specifies a maximum number of I/O requests that can be processed concurrently by the receiver node,

the I/O request is executed using a plurality of processing paths, each processing path including a different one of the plurality of first nodes and a different one of the plurality of second nodes; and

configuring at least some of the storage system based on the first latency metrics and the second latency metrics includes detecting a plurality of path-specific latency metrics, such that each path-specific metric corresponds to a different one of the processing paths, and decreasing the window size of the receiver node based on the plurality of path-specific latency metrics.

15. The non-transitory computer-readable medium of claim 14 , wherein the window size of the receiver node is decreased based on a smallest one of the plurality of path- specific latency metrics exceeding a predetermined threshold.