RDMA NIC utilizing packet level request and response interleaving
A best efforts (BE) hardware remote direct memory access (RDMA) transport being performed by a smart network interface controller (NIC). Elements from RoCEv2 and iWARP are utilized in combination with extensions to improve flexibility and packet error recovery. Flexibility is provided by allowing RDMA roles to be individually specified. Flexibility is also provided by additional packet numbering options to allow interleaving of request and response messages at a packet boundary. Error recovery is improved by utilized new acknowledgement responses, SNAK provided for each new hole detected and RACK for each received packet after a SNAK. SNAK allows the indication of resource exhaustion at the receiver, causing entry into a recovery mode where only packets in a hole are transmitted until resources are recovered.
1 . A remote direct memory access (RDMA) network interface controller (NIC) for connection to a network to communicate with a remote RDMA NIC by providing request messages and response messages to the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA header processing logic which builds RDMA packet headers to include values identifying individual packets in request packet flows and response packet flows so that request message packets can be interleaved with response message packets on a packet basis without ambiguity and allowing packet reliability operations;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA header processing logic and the network interface; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor.
2 . The RDMA NIC of claim 1 , wherein the RDMA header processing logic builds RDMA packet headers to include an added header and repurposes the packet sequence number (PSN) field of the base transport header (BTH) header so that one of the added header and the repurposed PSN field provides a message number and the other of the added header and the repurposed PSN field provides an intra message packet number, the message number incrementing with each new message and staying the same for each packet in the message and the intra message packet number incrementing for each packet in the message.
3 . The RDMA NIC of claim 2 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program; and
wherein the RDMA NIC reliability protocol program is configured to determine holes in a received stream of packets by examining the added header and the repurposed PSN field in the received packets.
4 . The RDMA NIC of claim 1 , wherein the RDMA header processing logic which builds RDMA packet headers so that request messages use a first sequence of packet sequence number (PSNs) and response messages utilize a second, different sequence of PSNs, the first sequence of PSNs increasing with each packet in a request message and the second sequence of PSNs increasing with each packet in a response message.
5 . The RDMA NIC of claim 4 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program; and
wherein the RDMA NIC reliability protocol program is configured to determine holes in a received stream of packets by examining the first sequence of PSNs and the second sequence of PSNs.
6 . A computer system comprising:
a computer including:
a computer processor;
a memory controller coupled to the computer processor;
computer memory coupled to the computer processor and the memory controller;
a computer peripheral device interface coupled to the computer processor and to the computer memory; and
computer non-transitory storage for programs to execute from computer memory on the computer processor; and
a remote direct memory access (RDMA) network interface controller (NIC) for connection to an Ethernet network to communicate with a remote RDMA NIC by providing request messages and response messages to the remote RDMA NIC, the RDMA NIC including:
an RDMA NIC peripheral device interface for connection to the computer;
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA header processing logic which builds RDMA packet headers to include values identifying individual packets in request packet flows and response packet flows so that request message packets can be interleaved with response message packets on a packet basis without ambiguity and allowing packet reliability operations;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA header processing logic and the network interface, the RDMA NIC memory; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor.
7 . The computer system of claim 6 , wherein the RDMA header processing logic builds RDMA packet headers to include an added header and repurposes the packet sequence number (PSN) field of the base transport header (BTH) header so that one of the added header and the repurposed PSN field provides a message number and the other of the added header and the repurposed PSN field provides an intra message packet number, the message number incrementing with each new message and staying the same for each packet in the message and the intra message packet number incrementing for each packet in the message.
8 . The computer system of claim 7 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program; and
wherein the RDMA NIC reliability protocol program is configured to determine holes in a received stream of packets by examining the added header and the repurposed PSN field in the received packets.
9 . The computer system of claim 6 , wherein the RDMA header processing logic which builds RDMA packet headers so that request messages use a first sequence of packet sequence number (PSNs) and response messages utilize a second, different sequence of PSNs, the first sequence of PSNs increasing with each packet in a request message and the second sequence of PSNs increasing with each packet in a response message.
10 . The computer system of claim 9 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program; and
wherein the RDMA NIC reliability protocol program is configured to determine holes in a received stream of packets by examining the first sequence of PSNs and the second sequence of PSNs.
11 . A method of operating a remote direct memory access (RDMA) network interface controller (NIC) for connection to a lossy Ethernet network to communicate with a remote RDMA NIC by providing request messages and response messages to the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA header processing logic which builds RDMA packet headers;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA header processing logic and the network interface; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor,
the method comprising:
including values in the RDMA packet headers identifying individual packets in request packet flows and response packet flows so that request message packets can be interleaved with response message packets on a packet basis without ambiguity and allowing packet reliability operations.
12 . The method of claim 11 , further comprising building RDMA packet headers to include an added header and repurposes the packet sequence number (PSN) field of the base transport header (BTH) header so that one of the added header and the repurposed PSN field provides a message number and the other of the added header and the repurposed PSN field provides an intra message packet number, the message number incrementing with each new message and staying the same for each packet in the message and the intra message packet number incrementing for each packet in the message.
13 . The method of claim 12 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program; and
the method further comprising determining holes in a received stream of packets by examining the added header and the repurposed PSN field in the received packets.
14 . The method of claim 11 , further comprising building RDMA packet headers so that request messages use a first sequence of packet sequence number (PSNs) and response messages utilize a second, different sequence of PSNs, the first sequence of PSNs increasing with each packet in a request message and the second sequence of PSNs increasing with each packet in a response message.
15 . The method of claim 14 , wherein the RDMA NIC non-transitory storage includes a RDMA NIC reliability protocol program, and
the method further comprising determining holes in a received stream of packets by examining the first sequence of PSNs and the second sequence of PSNs.