Merged hardware RDMA transport
A best efforts (BE) hardware remote direct memory access (RDMA) transport being performed by a smart network interface controller (NIC). Elements from RoCEv2 and iWARP are utilized in combination with extensions to improve flexibility and packet error recovery. Flexibility is provided by allowing RDMA roles to be individually specified. Flexibility is also provided by additional packet numbering options to allow interleaving of request and response messages at a packet boundary. Error recovery is improved by utilized new acknowledgement responses, SNAK provided for each new hole detected and RACK for each received packet after a SNAK. SNAK allows the indication of resource exhaustion at the receiver, causing entry into a recovery mode where only packets in a hole are transmitted until resources are recovered.
1 . A remote direct memory access (RDMA) network interface controller (NIC) for connection to a lossy Ethernet network to communicate to a remote RDMA NIC by exchanging packets with the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic which controls reception of packets from and transmission of packets to the network interface;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including RDMA packet processing programs for cooperating with the RDMA packet processing logic to perform transmission and reception of packets,
wherein the RDMA packet processing logic and the RDMA packet processing programs are configured, for a given RDMA connection, to perform NAK signaling using the reliability semantics of ROCE-RC, ACK and SACK signaling using reliability semantics of iWARP, symmetric reliability for request and response flows of iWARP, single-sided, one-phase transactions and dual-sided, one-phase transactions using the operations semantics of RoCE-RC and iWARP and single-sided, two-phase transactions using the operations semantics of ROCE-RC.
2 . The RDMA NIC of claim 1 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide a packet identifier in every request and response on the given RDMA connection.
3 . The RDMA NIC of claim 1 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to echo control information provided in request messages in related response messages on the given RDMA connection.
4 . The RDMA NIC of claim 1 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide target node RDMA addressing information in every single-sided, one-phase transaction on the given RDMA connection.
5 . The RDMA NIC of claim 1 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide message numbering and intra-message packet numbering in every dual-sided, one-phase transaction except transactions carrying addressing information on the given RDMA connection.
6 . The RDMA NIC of claim 1 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide response message numbering and intra-message packet numbering in every single-sided two-phase transaction on the given RDMA connection.
7 . A computer system comprising:
a computer including:
a computer processor;
a memory controller coupled to the computer processor;
computer memory coupled to the computer processor and the memory controller;
a computer peripheral device interface coupled to the computer processor and to the computer memory; and
computer non-transitory storage for programs to execute from computer memory on the computer processor; and
a remote direct memory access (RDMA) network interface controller (NIC) for connection to an Ethernet network to communicate to a remote RDMA NIC by exchanging packets with the remote RDMA NIC, the RDMA NIC including:
an RDMA NIC peripheral device interface for connection to the computer;
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic which controls reception of packets from and transmission of packets to the network interface;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing received and to be transmitted packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including RDMA packet processing programs for cooperating with the RDMA packet processing logic to perform transmission and reception of packets,
wherein the RDMA packet processing logic and the RDMA packet processing programs are configured, for a given RDMA connection, to perform NAK signaling using the reliability semantics of ROCE-RC, ACK and SACK signaling using reliability semantics of iWARP, symmetric reliability for request and response flows of iWARP-, single-sided, one-phase transactions and dual-sided, one-phase transactions using the operations semantics of ROCE-RC and iWARP and single-sided, two-phase transactions using the operations semantics of ROCE-RC.
8 . The computer system of claim 7 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide a packet identifier in every request and response on the given RDMA connection.
9 . The computer system of claim 7 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to echo control information provided in request messages in related response messages on the given RDMA connection.
10 . The computer system of claim 7 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide target node RDMA addressing information in every single-sided, one-phase transaction on the given RDMA connection.
11 . The computer system of claim 7 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide message numbering and intra-message packet numbering in every dual-sided, one-phase transaction except transactions carrying addressing information on the given RDMA connection.
12 . The computer system of claim 7 , wherein the RDMA packet processing logic and the RDMA packet processing programs are further configured to provide response message numbering and intra-message packet numbering in every single-sided two-phase transaction on the given RDMA connection.
13 . A method of operating a remote direct memory access (RDMA) network interface controller (NIC) for connection to a lossy Ethernet network to communicate to a remote RDMA NIC by exchanging packets with the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic which controls reception of packets from and transmission of packets to the network interface;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including RDMA packet processing programs for cooperating with the RDMA packet processing logic to perform transmission and reception of packets;
the method comprising, for a given RDMA connection:
performing NAK signaling using the reliability semantics of ROCE-RC;
performing ACK and SACK signaling using the reliability semantics of iWARP;
performing symmetric reliability for request and response flows of iWARP;
performing single-sided, one-phase transactions and dual-sided, one-phase transactions using the operations semantics of ROCE-RC and iWARP; and
performing single-sided, two-phase transactions using the operations semantics of RoCE-RC.
14 . The method of claim 13 , further comprising providing a packet identifier in every request and response on the given RDMA connection.
15 . The method of claim 13 , further comprising echoing control information provided in request messages in related response messages on the given RDMA connection.
16 . The method of claim 13 , further comprising providing target node RDMA addressing information in every single-sided, one-phase transaction on the given RDMA connection.
17 . The method of claim 13 , further comprising providing message numbering and intra-message packet numbering in every dual-sided, one-phase transaction except transactions carrying addressing information on the given RDMA connection.
18 . The method of claim 13 , further comprising providing response message numbering and intra-message packet numbering in every single-sided two-phase transaction on the given RDMA connection.