Session-based remote direct memory access
Apparatus for data communication includes a network interface for connection to a packet data network and a host interface for connection to a host computer, which includes a central processing unit (CPU) and a host memory. Packet processing circuitry receives, via the host interface, from a kernel running on the CPU, associations between multiple remote direct memory access (RDMA) sessions and multiple different User Datagram Protocol (UDP) 5-tuple, which are assigned respectively to the RDMA sessions, and receives from an application running on the CPU a request to send an RDMA message, using a selected group of one or more of the RDMA sessions, to a peer application over the packet data network, and in response to the request, transmits, via the network interface, one or more data packets using a UDP 5-tuple that is assigned to one of the RDMA sessions in the selected group.
1. Apparatus for data communication, comprising:
a network interface for connection to a packet data network;
a host interface for connection to a host computer, which includes a central processing unit (CPU) and a host memory; and
packet processing circuitry to receive, via the host interface, from a kernel running on the CPU, associations between multiple remote direct memory access (RDMA) sessions and multiple different User Datagram Protocol (UDP) 5-tuples, which comprise different, respective UDP source ports and are assigned respectively to the RDMA sessions, and to receive from an application running on the CPU a request to send an RDMA message, using a selected group of one or more of the RDMA sessions, to a peer application over the packet data network, and in response to the request, to transmit to a destination node, via the network interface, one or more Internet Protocol (IP) data packets encapsulating respective RDMA payloads and having an IP header containing a selected one of the UDP 5-tuples that is assigned to one of the RDMA sessions in the selected group,
wherein a UDP source port field in the IP header contains a respective port number that is assigned to the one of the RDMA sessions in the selected group, and
wherein upon receiving an incoming IP packet from the packet data network, the packet processing circuitry identifies a UDP 5-tuple in a header of the incoming IP packet, recognizes an RDMA session to which the UDP 5-tuple in the header is assigned, wherein the UDP 5-tuple in the header includes the assigned port number that is inserted by the destination node in a UDP destination port field of the received incoming IP packet.
2. The apparatus according to claim 1 , wherein based on the recognized session, the packet processing circuitry decapsulates and delivers an RDMA payload of the incoming IP packet to the application.
3. The apparatus according to claim 2 , wherein the kernel assigns different ones of the RDMA sessions to work queues associated with different, respective applications running on the CPU, and wherein the packet processing circuitry delivers the RDMA payload to the application only after verifying that the recognized session is assigned to a work queue that is assigned to the application.
4. The apparatus according to claim 1 , wherein in response to instructions conveyed by the kernel, the packet processing circuitry stores assignments of different ones of the RDMA sessions to different, respective applications running on the CPU.
5. The apparatus according to claim 4 , wherein the packet processing circuitry stores an assignment of a group of two or more of the RDMA sessions to a single one of the applications.
6. The apparatus according to claim 4 , wherein the application enters the request to send the RDMA message in a work queue that is assigned to the application, and the work queue is bound by the kernel to an RDMA session that is assigned to the application, and wherein the packet processing circuitry stores records of the RDMA sessions and the work queues that are respectively bound to the RDMA sessions, and applies the records in validating RDMA messages from and to the application.
7. The apparatus according to claim 1 , wherein the packet processing circuitry stores respective cryptographic keys for one or more of the RDMA sessions, and applies the respective cryptographic keys in encrypting and decrypting RDMA payloads of data packets that are transmitted to and received from the packet data network in the one or more of the RDMA sessions.
8. A method for data communication, comprising:
coupling a network interface controller (NIC) between a packet data network and a host computer, which includes a central processing unit (CPU) and a host memory;
receiving in the NIC, from a kernel running on the CPU, a definition of multiple remote direct memory access (RDMA) sessions and multiple different User Datagram Protocol (UDP) 5-tuples, which comprise different, respective UDP source ports and which are assigned respectively to the RDMA sessions;
receiving in the NIC, from an application running on the CPU, a request to send an RDMA message, using a selected group of one or more of the RDMA sessions, to a peer application over the packet data network;
in response to the request, transmitting from the NIC to a destination node, via the packet data network, one or more Internet Protocol (IP) data packets encapsulating respective RDMA payloads and having an IP header containing a selected one of the UDP 5-tuples that is assigned to one of the RDMA sessions in the selected group,
wherein a UDP source port field in the IP header contains a respective port number that is assigned to the one of the RDMA sessions in the selected group;
receiving in the NIC an incoming IP packet from the packet data network;
identifying, by the NIC, a UDP 5-tuple in a header of the incoming IP packet; and
recognizing an RDMA session to which the UDP 5-tuple in the header is assigned,
wherein the UDP 5-tuple in the header includes the assigned port number that is inserted by the destination node in a UDP destination port field of the received incoming IP packet.
9. The method according to claim 8 , and comprising: based on the recognized session, decapsulating and delivering an RDMA payload of the incoming IP packet to the application.
10. The method according to claim 9 , and comprising receiving in the NIC, from the kernel, assignments of different ones of the RDMA sessions to work queues associated with different, respective applications running on the CPU, wherein delivering the RDMA payload comprises passing the RDMA payload to the application only after verifying that the recognized session is assigned to a work queue that is assigned to the application.
11. The method according to claim 8 , and comprising assigning, by the kernel, different ones of the RDMA sessions to different, respective applications running on the CPU.
12. The method according to claim 11 , wherein assigning the different ones of the RDMA sessions comprises assigning a group of two or more of the RDMA sessions to a single one of the applications.
13. The method according to claim 11 , wherein receiving the request comprises queueing a work request to send the RDMA message in a work queue that is assigned to the application, wherein the work queue is bound by the kernel to an RDMA session that is assigned to the application, and wherein the NIC stores records of the RDMA sessions and the work queues that are respectively bound to the RDMA sessions, and applies the records in validating RDMA messages from and to the application.
14. The method according to claim 10 , and comprising storing in the NIC respective cryptographic keys for one or more of the RDMA sessions, and applying the respective cryptographic keys in encrypting and decrypting RDMA payloads of data packets that are transmitted to and received from the packet data network in the one or more of the RDMA sessions.
15. A system for data communication, comprising:
a host memory;
a central processing unit (CPU), which is to run a kernel, which defines multiple remote direct memory access (RDMA) sessions and respectively assigns multiple different User Datagram Protocol (UDP) 5-tuples, which comprise different, respective UDP source ports, to the RDMA sessions, and an application, which generates a request to send an RDMA message, using a selected group of one or more of the RDMA sessions, to a peer application over a packet data network; and
a network interface controller (NIC), which is to transmit to a destination node, in response to the request, via the packet data network, one or more Internet Protocol (IP) data packets encapsulating respective RDMA payloads and having an IP header containing a selected one of the UDP 5-tuples that is assigned to one of the RDMA sessions in the selected group,
wherein a UDP source port field in the IP header contains a respective port number that is assigned to the one of the RDMA sessions in the selected group, and
wherein upon receiving an incoming IP packet from the packet data network, the NIC identifies a UDP 5-tuple in a header of the incoming IP packet, recognizes an RDMA session to which the UDP 5-tuple in the header is assigned, wherein the UDP 5-tuple in the header includes the assigned port number that is inserted by the destination node in a UDP destination port field of the received incoming IP packet.
16. The system according to claim 15 , wherein based on the recognized session, the NIC decapsulates and delivers an RDMA payload of the incoming IP packet to the application.
17. The system according to claim 16 , wherein the kernel assigns different ones of the RDMA sessions to work queues associated with different, respective applications running on the CPU, and wherein the NIC delivers the RDMA payload to the application only after verifying that the recognized session is assigned to a work queue that is assigned to the application.
18. The system according to claim 15 , wherein in response to instructions conveyed by the kernel, the NIC stores assignments of different ones of the RDMA sessions to different, respective applications running on the CPU.
19. The system according to claim 18 , wherein the application enters the request to send the RDMA message in a work queue that is assigned to the application, and the work queue is bound by the kernel to an RDMA session that is assigned to the application, and wherein the NIC stores records of the RDMA sessions and the work queues that are respectively bound to the RDMA sessions, and applies the records in validating RDMA messages from and to the application.
20. The system according to claim 15 , wherein the NIC stores respective cryptographic keys for one or more of the RDMA sessions, and applies the respective cryptographic keys in encrypting and decrypting RDMA payloads of [the] data packets that are transmitted to and received from the packet data network in the one or more of the RDMA sessions.