Bandwidth allocation for storage system commands in peer-to-peer environment
Technology is disclosed for allocating PCIe bus bandwidth to storage commands in a peer-to-peer environment. A non-volatile storage system has a peer-to-peer connection with a host system and a target device, such as a GPU. A memory controller in the storage system monitors latency of PCIe transactions that are performed over a PCIe bus in order to transfer data for NVMe commands. The PCIe transactions may involve direct memory access (DMA) of memory in the host system or target device. There could be a significant difference in transaction latency depending on what memory is being accessed and/or what communication link is used to access the memory. The memory controller allocates bandwidth on a PCIe bus to the NVMe commands based on the latencies of the PCIe transactions. In an aspect, the memory controller groups the PCIe addresses based on the latencies of the PCIe transactions.
1. An apparatus, comprising:
a first communication interface configured to issue transactions over a communication bus in accordance with a first communication protocol to electronic devices connected to the communication bus, wherein the first communication protocol is a physical layer protocol;
a second communication interface configured to process storage commands of a second communication protocol received over the communication bus to control access to non-volatile memory in accordance with the second communication protocol, wherein the second communication protocol is a logical layer protocol above the first communication protocol; and
a control circuit coupled to the first communication interface and to the second communication interface, the control circuit configured to:
instruct the first communication interface to issue the transactions of the first communication protocol over the communication bus to perform a direct memory access (DMA) of the electronic devices connected to the communication bus;
determine latencies of the issued transactions, each transaction having a communication bus address; and
allocate bandwidth on the communication bus to the storage commands of the second communication protocol based on the latencies of the issued transactions of the first communication protocol.
2. The apparatus of claim 1 , wherein:
the first communication protocol is Peripheral Computer Interface Express (PCIe); and
the second communication protocol is Non-Volatile Memory Express (NVMe).
3. The apparatus of claim 1 , wherein the control circuit configured to:
estimate costs of transferring data associated with the storage commands across the communication bus based on the latencies of the transactions; and
regulate data transfers across the communication bus for the storage commands based on the estimated costs in order to allocate communication bus bandwidth to the storage commands.
4. The apparatus of claim 1 , wherein the control circuit is further configured to:
determine latencies of data control transactions;
determine latencies of data payload transactions; and
allocate bandwidth on the communication bus to storage commands of the second communication protocol based on dependencies between the data payload transactions and the data control transactions for the storage commands.
5. The apparatus of claim 1 , wherein the control circuit is further configured to:
measure latencies for transactions of the first communication protocol over the communication bus that store data to read data buffers, each read data buffer residing in memory of one of the electronic devices connected to the communication bus; and
allocate bandwidth on the communication bus to the storage commands of the second communication protocol based on the latencies of the transactions of the first communication protocol that access the read data buffers, wherein the storage commands comprise read commands.
6. The apparatus of claim 1 , wherein the control circuit is further configured to:
measure latencies for transactions of the first communication protocol over the communication bus that read data from write data buffers, each write data buffer residing in memory of one of the electronic devices connected to the communication bus, wherein the storage commands comprises write commands; and
allocate bandwidth on the communication bus to the storage commands of the second communication protocol based on the latencies of the transactions of the first communication protocol that access the write data buffers.
7. The apparatus of claim 1 , wherein the control circuit configured to:
determine a total latency for transactions of the first communication protocol needed for a storage command of the second communication protocol; and
schedule the storage command for access to the communication bus based on the total latency.
8. The apparatus of claim 1 , wherein the control circuit configured to:
measure latencies of transactions of the first communication protocol over the communication bus that access command submission queues of the second communication protocol, each command submission queue residing in memory of one of the electronic devices connected to the communication bus; and
allocate the bandwidth on the communication bus to the storage commands of the second communication protocol based on the latencies of the transactions of the first communication protocol that access command submission queues of the second communication protocol.
9. The apparatus of claim 1 , wherein the control circuit is further configured to:
measure latencies of transactions of the first communication protocol over the communication bus that access lists of pointers to data buffers, each list of pointers residing in memory of one of the electronic devices connected to the communication bus, the lists of pointers to data buffers being compliant with the second communication protocol; and
allocate bandwidth on the communication bus to the storage commands of the second communication protocol based on the latencies of the transactions of the first communication protocol that access the lists of pointers to data buffers.
10. The apparatus of claim 1 , wherein the electronic devices connected to the communication bus comprise:
a central processing unit (CPU); and
a graphics processing unit (GPU).
11. A method of operating a non-volatile storage system, the method comprising:
determining, by a memory controller of the non-volatile storage system, latencies of Peripheral Computer Interconnected Express (PCIe) transactions over a PCIe bus to directly access memory in respective PCIe devices connected to the PCIe bus, each PCIe transaction having a PCIe address;
grouping, by the memory controller, the PCIe addresses based on the latencies of the PCIe transactions;
scheduling, by the memory controller, Non-Volatile Memory Express (NVMe) commands to access the PCIe bus based on the latencies of the PCIe transactions; and
initiating, by the memory controller, PCIe transactions on the PCI bus to execute the NVMe commands in accordance with the schedule, including initiating multiple PCIe transactions for each NVMe command.
12. The method of claim 11 , wherein:
determining latencies of PCIe transactions over the PCIe bus to directly access memory in respective PCIe devices connected to the PCIe bus comprises measuring latencies of PCIe transactions that directly access NVMe submission queues, each NVMe submission queue residing in memory of one of the PCIe devices; and
grouping the PCIe addresses based on the latencies of the PCIe transactions comprises grouping the NVMe submission queues based on the PCIe transaction latencies to directly access the NVMe submission queues.
13. The method of claim 11 , wherein:
determining latencies of PCIe transactions over the PCIe bus to directly access memory in respective PCIe devices connected to the PCIe bus comprises measuring latencies of PCIe transactions that directly access NVMe submission queues, each command submission queue residing in memory of one of the PCIe devices measuring latencies of PCIe transactions that directly access NVMe data buffers, each data buffer residing in memory of one of the PCIe devices; and
grouping the PCIe addresses based on the latencies of the PCIe transactions comprises placing the PCIe addresses into different groups based on the latencies of the PCIe transactions that directly access the NVMe data buffers.
14. The method of claim 11 , wherein scheduling the NVMe commands to access the PCIe bus based on the latencies of the PCIe transactions comprises:
estimating a cost to implement an NVMe command based on the latencies of PCIe transactions to implement the NVMe command; and
determining whether the estimated cost exceeds a threshold.
15. A non-volatile memory system, comprising:
non-volatile memory;
a Peripheral Computer Interface Express (PCIe) interface configured to control PCIe transactions over a PCIe bus, the PCIe interface configured to issue PCIe transactions to directly access data buffers in a plurality of processors connected to the PCIe bus;
a Non-Volatile Memory Express (NVMe) interface configured to process NVMe commands received on the PCIe bus to access the non-volatile memory;
means for determining latencies of the PCIe transactions that directly access the data buffers in the plurality of processors, the PCIe transactions that directly access the data buffers comprise data payload transactions; and
means for regulating data transfers across the PCIe bus for the NVMe commands based on the latencies of the PCIe transactions that directly access the data buffers in the plurality of processors.
16. The non-volatile memory system of claim 15 , wherein the means for regulating data transfers across the PCIe bus for the NVMe commands based on the latencies of the PCIe transactions that directly access the data buffers in the plurality of processors is configured to:
estimate costs of accessing the data buffers associated with the NVMe commands based on the latencies of the PCIe transactions that directly access the data buffers in the plurality of processors; and
schedule the NVMe commands based on the estimated costs.
17. The non-volatile memory system of claim 15 , further comprising means for determining latencies of PCIe control transactions that directly access NVMe Input/Output (I/O) queues or physical region page (PRP) lists in memory of the plurality of processors connected to the PCIe bus; and
wherein the means for regulating data transfers across the PCIe bus for the NVMe commands is further for scheduling the NVMe commands based on the latencies of the PCIe control transactions.
18. The non-volatile memory system of claim 15 , wherein the means for determining latencies of the PCIe transactions that directly access the data buffers in the plurality of processors is configured to:
determine latencies of PCIe transactions that directly access data buffers in a central processing unit (CPU); and
determine latencies of PCIe transactions that directly access data buffers in a graphics processing unit (GPU).
19. The non-volatile memory system of claim 15 , wherein the means for regulating data transfers across the PCIe bus for the NVMe commands based on the latencies of the PCIe transactions that directly access the data buffers in the plurality of processors is configured to:
schedule PCI transactions that directly access the data buffers in a first processor of the plurality of processors.