NVMEoF flow control from initiator based on transaction latency
A storage array that uses NVMEoF to interconnect compute nodes with NVME SSDs via a fabric and NVME offload engines implements flow control based on transaction latency. Transaction latency is the elapsed time between the send side completion message and receive side completion message for a single transaction. Counts of total transactions and over-latency-limit transactions are accumulated over a time interval. If the over limit rate exceeds a threshold, then the maximum allowed number of enqueued pending transactions is reduced. The maximum allowed number of enqueued pending transactions is periodically restored to a default value.
1. An internal input-output (TO) flow control method implemented in a storage array with a plurality of non-volatile solid-state drives as first IO end points and a plurality of interconnected compute nodes as second IO end points, wherein the compute nodes access the drives via a fabric and offload engines that are not IO end points and for which utilization of processor and memory resources is not accounted for by any end-to-end remote direct memory access (RDMA) flow control, the offload engines including cache and being configured to perform error correction or deduplication, the method comprising:
performing end-to-end RDMA flow control between the first IO end points and the second IO end points; and
performing initiator-based non-volatile memory express over fabric (NVMEoF) flow control by:
monitoring transactions between ones of the compute nodes and ones of the offload engines to determine transaction latency;
determining transaction latency as time between send side completion and receive side completion based on messages sent by ones of the offload engines; and
adjusting a number of pending transactions based on the transaction latency.
2. The method of claim 1 comprising maintaining a count of transactions that exceed a latency limit during a checking interval.
3. The method of claim 2 comprising maintaining a count of monitored transactions during the checking interval.
4. The method of claim 3 comprising resetting both the count of transactions that exceed the latency limit during the checking interval and the count of monitored transactions during the checking interval after a max setting time has elapsed.
5. The method of claim 4 comprising restoring a default limit on the pending transactions after the max setting time has elapsed.
6. The method of claim 5 comprising lowering a limit on pending transactions responsive to a transaction latency over limit rate exceeding a predetermined threshold, wherein the transaction latency over limit rate equals the count of transactions that exceed the latency limit during the checking interval divided by the count of monitored transactions during the checking interval.
7. A non-transitory computer-readable storage medium that stores instructions that when executed by a compute node of a storage array cause the compute node to perform a method for flow control in the storage array where the compute node is a first internal input-output (TO) end point and a plurality of non-volatile solid-state drives are second internal IO end points, wherein the compute nodes access the drives via a fabric and offload engines that are not internal TO end points and for which utilization of processor and memory resources is not accounted for by any end-to-end remote direct memory access (RDMA) flow control, the offload engines including cache and being configured to perform error correction or deduplication, the method comprising:
performing end-to-end RDMA flow control between the first TO end points and the second TO end points; and
performing initiator-based non-volatile memory express over fabric (NVMEoF) flow control by:
monitoring transactions between the compute node and at least one of the offload engines via which the non-volatile drives are accessed to determine transaction latency;
determining transaction latency as time between send side completion and receive side completion based on messages sent by the offload engine; and
adjusting a number of pending transactions based on the transaction latency.
8. The computer-readable storage medium of claim 7 wherein the method comprises maintaining a count of transactions that exceed a latency limit during a checking interval.
9. The computer-readable storage medium of claim 8 wherein the method comprises maintaining a count of monitored transactions during the checking interval.
10. The computer-readable storage medium of claim 9 wherein the method comprises resetting both the count of transactions that exceed the latency limit during the checking interval and the count of monitored transactions during the checking interval after a max setting time has elapsed.
11. The computer-readable storage medium of claim 10 wherein the method comprises restoring a default limit on the pending transactions after the max setting time has elapsed.
12. The computer-readable storage medium of claim 11 wherein the method comprises lowering a limit on pending transactions responsive to a transaction latency over limit rate exceeding a predetermined threshold, wherein the transaction latency over limit rate equals the count of transactions that exceed the latency limit during the checking interval divided by the count of monitored transactions during the checking interval.
13. An apparatus comprising:
a plurality of non-volatile solid-state drives that are first internal input-output (TO) end points;
a plurality of interconnected compute nodes that are second internal TO end points, end-to- end remote direct memory access (RDMA) flow control being implemented between the first TO end points and the second TO end points, the plurality of interconnected compute nodes configured to access the drives via a fabric and ones of a plurality of offload engines that are not internal TO end points and for which utilization of processor and memory resources is not accounted for by any end-to-end remote direct memory access (RDMA) flow control, the offload engines including cache and being configured to perform error correction or deduplication; and
an initiator-based non-volatile memory express over fabric (NVMEoF) flow controller configured to monitor transactions between ones of the compute nodes and ones of the offload engines to determine transaction latency and adjust a number of pending transactions based on the transaction latency, wherein transaction latency indicates time between send side completion and receive side completion based on messages sent by ones of the offload engines.
14. The apparatus if claim 13 wherein the flow controller is configured to maintain a count of transactions that exceed a latency limit during a checking interval.
15. The apparatus if claim 14 wherein the flow controller is configured to maintain a count of monitored transactions during the checking interval.
16. The apparatus if claim 15 wherein the flow controller is configured to reset both the count of transactions that exceed the latency limit during the checking interval and the count of monitored transactions during the checking interval after a max setting time has elapsed.
17. The apparatus if claim 16 wherein the flow controller is configured to restore a default limit on the pending transactions after the max setting time has elapsed.
18. The apparatus if claim 17 wherein the flow controller is configured to lower a limit on pending transactions responsive to a transaction latency over limit rate exceeding a predetermined threshold, wherein the transaction latency over limit rate equals the count of transactions that exceed the latency limit during the checking interval divided by the count of monitored transactions during the checking interval.