Hardware RDMA transport including new hole and received after a hole acknowledgements
A best efforts (BE) hardware remote direct memory access (RDMA) transport being performed by a smart network interface controller (NIC). Elements from RoCEv2 and iWARP are utilized in combination with extensions to improve flexibility and packet error recovery. Flexibility is provided by allowing RDMA roles to be individually specified. Flexibility is also provided by additional packet numbering options to allow interleaving of request and response messages at a packet boundary. Error recovery is improved by utilized new acknowledgement responses, SNAK provided for each new hole detected and RACK for each received packet after a SNAK. SNAK allows the indication of resource exhaustion at the receiver, causing entry into a recovery mode where only packets in a hole are transmitted until resources are recovered.
1 . A remote direct memory access (RDMA) network interface controller (NIC) for connection to a network to communicate with a remote RDMA NIC for receiving a stream of sequentially numbered packets from the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic coupled to the network interface which controls reception of packets from and transmission of packets to the network interface and provides reliability protocol elements;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing received packets of the stream of sequentially numbered packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including an RDMA NIC reliability protocol program,
wherein the RDMA NIC reliability protocol is configured to determine holes in the stream of sequentially numbered packets and packets received after a hole, to provide a new hole indication packet to the network interface upon determining a new hole and to provide a received packet after a hole packet to the network interface for packets received after a hole.
2 . The RDMA NIC of claim 1 , wherein the RDMA NIC reliability program is further configured to provide an acknowledgement of successful in-order reception to the network interface of at least one packet received before a hole is determined to have occurred and after all holes have been filled.
3 . The RDMA NIC of claim 2 , wherein the RDMA NIC reliability program is further configured to provide an acknowledgement of successful in-order reception of at least one packet upon receipt of a packet filling at least the earliest hole.
4 . The RDMA NIC of claim 3 , wherein the new hole indication packet can indicate one or more singular holes or ranges of holes.
5 . The RDMA NIC of claim 3 , wherein the earliest hole is the beginning of a range of holes or is a singular hole.
6 . The RDMA NIC of claim 1 , wherein the RDMA NIC memory further includes information on holes in the received stream of sequentially numbered packets,
wherein an allocated amount of the RDMA NIC memory is provided for the hole information, and
wherein the RDMA NIC reliability program is further configured to store hole information in the RDMA NIC memory, to determine when the allocated amount of RDMA NIC memory provided for hole information is exceeded and to provide a resources depleted packet to the network interface upon determining that the allocated amount of RDMA NIC memory provided for hole information has been exceeded.
7 . The RDMA NIC of claim 6 , wherein the RDMA NIC reliability program is further configured to provide a resources restored packet to the network interface after sufficient of the allocated memory provided for hole information has been cleared.
8 . A computer system comprising:
a computer including:
a computer processor;
a memory controller coupled to the computer processor;
computer memory coupled to the computer processor and the memory controller;
a computer peripheral device interface coupled to the computer processor and to the computer memory; and
computer non-transitory storage for programs to execute from computer memory on the computer processor; and
a remote direct memory access (RDMA) network interface controller (NIC) for connection to an Ethernet network to communicate with a remote RDMA NIC by exchanging packets with the remote RDMA NIC, the RDMA NIC including:
an RDMA NIC peripheral device interface for connection to the computer;
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic coupled to the network interface which controls reception of packets from and transmission of packets to the network interface and provides reliability protocol elements;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing received packets of a stream of sequentially numbered packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including an RDMA NIC reliability protocol program,
wherein the RDMA NIC reliability protocol is configured to determine holes in the stream of sequentially numbered packets and packets received after a hole, to provide a new hole indication packet to the network interface upon determining a new hole and to provide a received packet after a hole packet to the network interface for packets received after a hole.
9 . The computer system of claim 8 , wherein the RDMA NIC reliability program is further configured to provide an acknowledgement of successful in-order reception to the network interface of at least one packet received before a hole is determined to have occurred and after all holes have been filled.
10 . The computer system of claim 9 , wherein the RDMA NIC reliability program is further configured to provide an acknowledgement of successful in-order reception of at least one packet upon receipt of a packet filling at least the earliest hole.
11 . The computer system of claim 10 , wherein the new hole indication packet can indicate one or more singular holes or ranges of holes.
12 . The computer system of claim 10 , wherein the earliest hole is the beginning of a range of holes or is a singular hole.
13 . The computer system of claim 8 , wherein the RDMA NIC memory further includes information on holes in the received stream of sequentially numbered packets,
wherein an allocated amount of the RDMA NIC memory is provided for the hole information,
wherein the RDMA NIC reliability program is further configured to store hole information in the RDMA NIC memory, to determine when the allocated amount of RDMA NIC memory provided for hole information is exceeded and to provide a resources depleted packet to the network interface upon determining that the allocated amount of RDMA NIC memory provided for hole information has been exceeded.
14 . The computer system of claim 13 , wherein the RDMA NIC reliability program is further configured to provide a resources restored packet to the network interface after sufficient of the allocated memory provided for hole information has been cleared.
15 . A method of operating a remote direct memory access (RDMA) network interface controller (NIC) for connection to a network for receiving a stream of sequentially numbered packets from the remote RDMA NIC, the RDMA NIC comprising:
a network interface for connection to the network;
an RDMA NIC processor coupled to the network interface;
RDMA packet processing logic coupled to the network interface which controls reception of packets from and transmission of packets to the network interface and provides reliability protocol elements;
RDMA NIC memory coupled to the RDMA NIC processor, the RDMA packet processing logic and the network interface, the RDMA NIC memory including:
buffer storage for storing received packets of a stream of sequentially numbered packets; and
RDMA NIC non-transitory storage for programs to execute from the RDMA NIC memory on the RDMA NIC processor, the RDMA NIC non-transitory storage including an RDMA NIC reliability protocol program,
the method comprising:
determining holes in the stream of sequentially numbered packets and packets received after a hole;
providing a new hole indication packet to the network interface upon determining a new hole; and
providing a received packet after a hole packet to the network interface for packets received after a hole.
16 . The method of claim 15 , further comprising providing an acknowledgement of successful in-order reception to the network interface of at least one packet received before a hole is determined to have occurred and after all holes have been filled.
17 . The method of claim 16 , further comprising providing an acknowledgement of successful in-order reception of at least one packet upon receipt of a packet filling at least the earliest hole.
18 . The method of claim 17 , wherein the new hole indication packet can indicate one or more singular holes or ranges of holes.
19 . The method of claim 17 , wherein the earliest hole is the beginning of a range of holes or is a singular hole.
20 . The method of claim 15 , wherein the RDMA NIC memory further includes information on holes in the received stream of sequentially numbered packets, and
wherein an allocated amount of the RDMA NIC memory is provided for the hole information,
the method further comprising:
storing hole information in the RDMA NIC memory;
determining when the allocated amount of RDMA NIC memory provided for hole information is exceeded; and
providing a resources depleted packet to the network interface upon determining that the allocated amount of RDMA NIC memory provided for hole information has been exceeded.
21 . The method of claim 20 , further comprising providing a resources restored packet to the network interface after sufficient of the allocated memory provided for hole information has been cleared.