Methods and apparatus for DMA engine descriptors for high speed data systems
Methods and apparatus for improved data movement operations through interconnect fabric. In one embodiment, Non-Transparent Bridge (NTB) technology used to perform data movement operations between a host and multiple peer devices using a DMA (direct memory access) engine and at least one descriptor ring having enhanced descriptor entries. In one implementation, descriptor ring entries include source and destination address information, address translation information, and fabric partition information. In one implementation, a DMA engine is configured directly access host memory and generate data packets using the descriptor entry information. In one embodiment, the descriptor ring is a virtual descriptor ring located on DMA hardware, host memory, or elsewhere in the NT fabric address space, and may be accessed by user processes.
1. A method of operating a data fabric, comprising:
receiving at a first address space and from a first device, a packet;
editing the received packet by at least:
(i) based at least on the first address space, editing a read address of the received packet to a first translated address; and
(ii) editing a first identifier associated with the received packet to a second identifier;
causing routing the edited received packet through the data fabric to a second device;
receiving a reply to the routed edited received packet; and
causing writing data associated with the reply to the first device;
wherein the first and second devices each comprise hosts in data communication with the data fabric, and at least the acts of receiving and editing are performed by a non-transparent endpoint (NT EP).
2. The method of claim 1 , wherein the receiving a packet at a first address space comprises receiving a transaction layer packet (TLP) addressed to a physical address of a BAR (base address register) of the NT EP.
3. The method of claim 2 , wherein the routing the edited received packet through the data fabric to a second device comprises routing the TLP to a second NT EP associated with the second device.
4. The method of claim 1 , wherein the editing a first identifier associated with the received packet to a second identifier comprises editing a requester identifier into a proxy identifier.
5. The method of claim 4 , wherein the receiving a reply to the routed edited received packet comprises receiving the reply based on the proxy identifier.
6. The method of claim 5 , wherein the writing data associated with the reply to the first device further comprises correlating the proxy identifier to the first device.
7. The method of claim 1 , wherein the editing, based at least on the first address space, of a read address of the received packet to a first translated address comprises using translation data within a lookup table (LUT) associated with a BAR space.
8. The method of claim 1 , wherein the method of operating comprises use of a PCIe (Peripheral Component Interconnect express) protocol on both (i) a first side of the NT EP, the first side associated with the first device, and (ii) a second side of the NT EP, the second side associated with the second device.
9. The method of claim 1 , wherein the method of operating is conducted without a protocol change between a first domain of the first device and a second domain of the second device.
10. A computerized apparatus comprising computerized logic configured to:
at a DMA (direct memory access) descriptor ring associated with a data fabric, receive from a first device an entry;
read the received entry using a DMA engine process;
based at least on the reading, access first data within a prescribed address range within a memory;
generate at least one transaction layer packet (TLP) based at least on the accessed first data;
cause transmission of the at least one TLP via at least the data fabric to a target;
receive a response to the at least one TLP, the response issued by the target;
evaluate at least a portion of the received response; and
based at least on the evaluation, cause provision of at least a portion of the received response to the first device;
wherein the generation of the at least one TLP comprises (i) utilization of a source ID value associated with the first device for the at least one TLP; and (ii) utilization of a particular NT EP (non-transparent endpoint) memory address as a destination address for the at least one TLP.
11. The computerized apparatus of claim 10 , wherein the computerized apparatus comprises the descriptor ring.
12. The computerized apparatus of claim 10 , wherein the data fabric comprises a non-transparent and PCIe (Peripheral Component Interconnect express) compliant fabric.
13. The computerized apparatus of claim 10 , wherein the first device and the target are associated with a first domain and a second domain, respectively, and the DMA descriptor ring is not part of either the first domain or the second domain.
14. The computerized apparatus of claim 13 , wherein the first domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol, and (ii) the second domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol.
15. The computerized apparatus of claim 13 , wherein the DMA engine process is configured to directly transact in at least one of an address domain or transaction domain of the data fabric.
16. The computerized apparatus of claim 10 , wherein the DMA engine process is configured to directly transact in at least one of an address domain or transaction domain of the data fabric.
17. The computerized apparatus of claim 16 , wherein the DMA engine process is associated with the DMA descriptor ring and the data fabric.
18. The computerized apparatus of claim 10 , wherein the generation of the at least one TLP further comprises a fabric ID value.
19. A computerized apparatus comprising computerized logic configured to:
receive, at a DMA (direct memory access) descriptor ring associated with a data fabric, an entry from a first device;
read the received entry using a DMA engine process;
based at least on the reading, access first data within a prescribed address range within a memory;
generate at least one transaction layer packet (TLP) based at least on the accessed first data;
cause transmission of the at least one TLP via at least the data fabric to a target;
receive a response to the at least one TLP issued by the target;
evaluate at least a portion of the received response; and
based at least on the evaluation, cause provision of the received response to the first device;
wherein:
the first device comprises a first host device associated with a first NT EP (non-transparent endpoint) of the fabric;
the target comprises a second host device associated with a second NT EP of the fabric; and
the causation of transmission of the at least one TLP via at least the data fabric to the target comprises causation by the DMA engine process of transmission of the at least one TLP to the second NT EP.
20. The computerized apparatus of claim 19 , wherein the computerized apparatus comprises the descriptor ring.
21. The computerized apparatus of claim 19 , wherein the data fabric comprises a non-transparent and PCIe (Peripheral Component Interconnect express) compliant fabric.
22. The computerized apparatus of claim 19 , wherein the first device and the target are associated with a first domain and a second domain, respectively, and the DMA engine is not part of either the first domain or the second domain.
23. The computerized apparatus of claim 22 , wherein the first domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol, and (ii) the second domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol.
24. The computerized apparatus of claim 22 , wherein the DMA engine process is configured to directly transact in at least one of an address domain or transaction domain of the data fabric.
25. The computerized apparatus of claim 19 , wherein the DMA engine process is configured to directly transact in at least one of an address domain or transaction domain of the data fabric.
26. The computerized apparatus of claim 25 , wherein the DMA engine process is associated with the data fabric, and the DMA descriptor ring is associated with the first domain.
27. The computerized apparatus of claim 19 , wherein the generation of the at least one TLP further comprises a fabric ID value.
28. A method of operating a data fabric to perform a read transaction, the method comprising:
receiving at a first address space and from a first device within a first PCIe (Peripheral Component Interconnect express) domain, a packet;
editing the received packet by at least:
(i) based at least on the first address space, editing a read address of the received packet to a first translated address; and
(ii) editing a first identifier associated with the received packet to a second identifier;
causing routing the edited received packet through the data fabric to a second device in a second PCIe domain;
receiving a reply to the routed edited received packet; and
causing writing data associated with the reply to the first device;
wherein the first and second devices each comprise computerized processes in data communication with the data fabric, and at least the acts of receiving and editing are performed by a by a DMA (direct memory access) engine associated with non-transparent endpoint (NT EP) located in neither the first PCIe domain nor the second PCIe domain.
29. A computerized apparatus comprising computerized logic configured to:
at a DMA (direct memory access) descriptor ring associated with a data fabric, receive from a first device an entry;
read the received entry using a DMA engine process;
based at least on the reading, access first data within a determined address range within a memory;
generate at least one transaction layer packet (TLP) based at least on the accessed first data;
cause transmission of the at least one TLP via at least the data fabric to a target;
receive a response to the at least one TLP, the response issued by the target;
evaluate at least a portion of the received response; and
based at least on the evaluation, cause provision of at least a portion of the received response to the first device;
wherein the generation of the at least one TLP comprises (i) utilization of a source ID value associated with the first device for the at least one TLP; and (ii) utilization of a particular NT EP (non-transparent endpoint) address to derive a destination address for the at least one TLP.
30. A computerized apparatus comprising computerized logic configured to:
access, via a DMA (direct memory access) engine, first data within a prescribed address range within a memory, the access based at least on a second data received from a first domain via a DMA descriptor ring, the DMA descriptor ring in data communication with the DMA engine;
generate at least one transaction layer packet (TLP) based at least on the accessed first data;
cause transmission of the at least one TLP via at least a data fabric to a target domain;
receive a response to the at least one TLP issued by the target domain;
evaluate at least a portion of a response to the transmitted at least one TLP, the response issued by the target domain; and
based at least on the evaluation, cause provision of at least part of the received response to the first domain;
wherein:
the first domain is associated with a first NT EP (non-transparent endpoint) of the fabric;
the target domain is associated with a second NT EP of the fabric; and
the causation of transmission of the at least one TLP via at least the data fabric to the target domain comprises causation of transmission of the at least one TLP to the second NT EP.
31. The computerized apparatus of claim 30 , wherein the DMA descriptor ring is not part of either the first domain or the target domain.
32. The computerized apparatus of claim 30 , wherein the first domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol, and (ii) the target domain is configured to operate using a PCIe (Peripheral Component Interconnect express) protocol.
33. The computerized apparatus of claim 32 , wherein the DMA engine process is configured to directly transact in at least one of an address domain or transaction domain of the data fabric.