Direct access to reconfigurable processor memory
A system is presented that includes two data processing systems that are coupled via a network, each data processing system including a reconfigurable processor with a reconfigurable processor memory, a host that is coupled to the reconfigurable processor and that includes a host processor and a host memory that is coupled to the host processor, and a network interface controller (NIC) that is operatively coupled to the reconfigurable processor and to the host processor. The reconfigurable processor of one of the data processing systems is configured to implement a virtual function that uses a virtual address for a memory access operation. An application programming interface (API) in the host processor translates the virtual address into a physical address, and the NIC uses the physical address to initiate a direct memory access operation at the reconfigurable processor memory or the host memory of the other data processing system.
1. A system, comprising:
a first data processing system, comprising:
a first reconfigurable processor with a first reconfigurable processor memory,
a first host that is operatively coupled to the first reconfigurable processor, comprising:
a first host processor including runtime logic with a user space and a kernel space, and
a first host memory that is coupled to the first host processor, and
a first network interface controller (NIC) that is operatively coupled to, a network, the first reconfigurable processor, and the first host processor; and
a second data processing system that is coupled via the network to the first data processing system, comprising:
a second reconfigurable processor with a second reconfigurable processor memory,
a second host that is operatively coupled to the second reconfigurable processor, comprising:
a second host processor, and
a second host memory that is coupled to the second host processor, and
a second network interface controller (NIC) that is operatively coupled to the network, the second reconfigurable processor, and the second host processor;
wherein the first reconfigurable processor is configured to implement a virtual function that uses a virtual address for a memory access operation ;
wherein the first host processor is configured to implement an application programming interface (API) within the runtime logic that includes a first module located in the user space, a second module executing in kernel space, a third module executing in the kernel space, and input/output memory management unit (IOMMU) page tables;
wherein the API translates the virtual address into a physical address using the first module and the second module for virtual addresses targeting the first host memory and additionally using the third module for virtual addresses targeting the first reconfigurable processor memory, wherein the third module translates the virtual address targeting the first reconfigurable processor memory to an input/output virtual address (IOVA), and then uses the IOMMU page tables to translate the IOVA to the physical address; and
wherein the first NIC uses the physical address to initiate a direct memory access operation at the second reconfigurable processor memory or the second host memory that moves data directly between the first reconfigurable processor and the second reconfigurable processor memory or the second host memory, wherein the data bypasses the first host processor and is transferred directly between the first reconfigurable processor and the first NIC.
2. The system of claim 1 , wherein the first and second reconfigurable processors comprise arrays of coarse-grained reconfigurable (CGR) units.
3. The system of claim 2 , wherein the arrays of CGR units of the first reconfigurable processor perform computational tasks in parallel to the direct memory access operation.
4. The system of claim 1 , wherein the direct memory access operation is a direct memory write access operation that moves the data directly from the first reconfigurable processor to the first-second reconfigurable processor memory or the second host memory.
5. The system of claim 1 , wherein the direct memory access operation is a direct memory read access operation that moves the data directly from the second reconfigurable processor memory or the second host memory to the first reconfigurable processor.
6. The system of claim 1 , wherein the API that translates the virtual address into the physical address comprises:
a first argument that comprises a list, wherein each element in the list comprises a virtual address page;
a second argument that comprises a number of elements in the list; and
a third argument that is set to a first value when the direct memory access operation targets the second reconfigurable processor memory and to a second value that is different than the first value when the direct memory access operation targets the second host memory.
7. The system of claim 6 , wherein the first data processing system further comprises:
a peripheral component interconnect express (PCIe) bus, that couples the first reconfigurable processor, the first host, and the first NIC, wherein a PCIe physical base address register region is memory-mapped into a virtual address space to create a virtual to physical memory address mapping.
8. The system of claim 7 , wherein the first reconfigurable processor implements a fixed-sized memory-mapped region that exposes a virtually contiguous window to the first reconfigurable processor memory, and wherein the virtually contiguous window is implemented using a list of physically discontinuous regions.
9. A method of operating a system that comprises a first data processing system comprising a first reconfigurable processor with a first reconfigurable processor memory, a first host that is operatively coupled to the first reconfigurable processor and comprising a first host processor, and a first host memory that is coupled to the first host processor, and a first network interface controller (NIC) that is operatively coupled to a network, the first reconfigurable processor, and the first host processor; and a second data processing system that is coupled via the network to the first data processing system, comprising a second reconfigurable processor with a second reconfigurable processor memory, a second host that is operatively coupled to the second reconfigurable processor and comprising a second host processor, and a second host memory that is coupled to the second host processor, and a second network interface controller (NIC) that is operatively coupled to the network, the second reconfigurable processor, and the second host processor, the method comprising:
executing runtime logic on the first host processor with a user space and a kernel space;
configuring the first reconfigurable processor to implement a virtual function that uses a virtual address for a memory access operation;
configuring the first host processor to implement an application programming interface (API) within the runtime logic that includes a first module located in the user space, a second module executing in kernel space, a third module executing in the kernel space, and input/output memory management unit (IOMMU) page tables;
translating, with the API, the virtual address into a physical address using the first module and the second module for virtual addresses targeting the first host memory and additionally using the third module for virtual addresses targeting the first reconfigurable processor memory, wherein the third module translates the virtual address targeting the first reconfigurable processor memory to an input/output virtual address (IOVA), and then uses the IOMMU page tables to translate the IOVA to the physical address; and
with the first NIC, using the physical address to initiate a direct memory access operation at the second host memory or the second reconfigurable processor memory that moves data directly between the first reconfigurable processor and the second host memory or the second reconfigurable processor memory, wherein the data bypasses the first host processor and is transferred directly between the first reconfigurable processor and the first NIC.
10. The method of claim 9 , wherein the direct memory access operation is one of a direct memory write access operation that moves the data directly from the first reconfigurable processor to the second host memory or the second reconfigurable processor memory or a direct memory read access operation that moves the data directly from the second host memory or the second reconfigurable processor memory to the first reconfigurable processor.
11. The method of claim 9 , wherein the first data processing system comprises a peripheral component interface express (PCIe) bus that couples the first host, the first reconfigurable processor, and the first NIC, and wherein translating, with the API, the virtual address into the physical address further comprises:
determining an offset of the virtual address from a start of an associated virtual memory region; and
adding the offset to the start of a base address register's physical address.
12. The method of claim 11 , wherein translating, with the API, the virtual address into the physical address further comprises:
receiving, with the API, a first argument that comprises a list, wherein each element in the list comprises a virtual address page; and
receiving, with the API, a second argument that comprises a number of elements in the list.
13. The method of claim 11 , further comprising:
receiving, with the API, a third argument that has a first value, if the virtual address belongs to the second host memory, and a second value that is different than the first value, if the virtual address belongs to the second reconfigurable processor memory.
14. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to operate a system that comprises a first data processing system comprising a first reconfigurable processor with a first reconfigurable processor memory, a first host that is operatively coupled to the first reconfigurable processor and comprising a first host processor, and a first host memory that is coupled to the first host processor, and a first network interface controller (NIC) that is operatively coupled to a network, the first reconfigurable processor, and the first host processor; and a second data processing system that is coupled via the network to the first data processing system, comprising a second reconfigurable processor with a second reconfigurable processor memory, a second host that is operatively coupled to the second reconfigurable processor and comprising a second host processor, and a second host memory that is coupled to the second host processor, and a second network interface controller (NIC) that is operatively coupled to the network, the second reconfigurable processor, and the second host processor, the instructions comprising:
executing runtime logic on the first host processor with a user space and a kernel space;
configuring the first reconfigurable processor to implement a virtual function that uses a virtual address for a memory access operation;
translating, with an application programming interface (API) within the runtime logic that includes a first module located in the user space, a second module executing in kernel space, a third module executing in the kernel space, and input/output memory management unit (IOMMU) page tables, the virtual address into a physical address using the first module and the second module for virtual addresses targeting the first host memory and additionally using the third module for virtual addresses targeting the first reconfigurable processor memory, wherein the third module translates the virtual address targeting the first reconfigurable processor memory to an input/output virtual address (IOVA), and then uses the IOMMU page tables to translate the IOVA to the physical address; and
initiating, using the first NIC, a direct memory access operation at the physical address of the second host memory or the second reconfigurable processor memory that moves data directly between the first reconfigurable processor and the second host memory or the second reconfigurable processor memory, wherein the data bypasses the first host processor and is transferred directly between the first reconfigurable processor and the first NIC.