Reliable out-of order end-to-end protocol with robust window state overflow management and a multi-node system using same
In an embodiment, an out-of-order, reliable, end-to-end protocol is provided that can enable direct user-level data placement and atomic operations between nodes of a multi-node network. The protocol may be optimized for low-loss environments such as High Performance Computing (HPC) applications, and may enable loss detection and de-duplication of packets through the use of a robust window state manager at a target node. A multi-node network implementing the protocol may have increased system reliability, packet throughput, and increased tolerance for adaptively routed traffic, while still allowing atomic operations to be idempotently applied directly to a user memory location.
1. A method for performing inter-process communication between nodes of a multi-node system, the method comprising:
defining, by a circuit, a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
receiving, by the circuit, a connection initialization packet from a remote node;
sending, by the circuit, a response packet to the remote node including a maximum window size associated the window and a starting sequence number, wherein the response packet is configured to cause the remote node to assign sequence numbers to the plurality of packets sent therefrom with a starting sequence number set to the starting sequence number included in the response packet;
receiving, by the circuit, a plurality of packets from the remote node, each received packet having a unique sequence number;
determining, by the circuit, receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
causing, by the circuit, the remote node to suspend transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
transmitting, to the remote node by the circuit, an indication in response to determining receipt of the consecutive set of received packets.
2. The method of claim 1 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.
3. The method of claim 2 , wherein the special acknowledgement packet is transmitted periodically to the remote node based on a timer.
4. The method of claim 1 , further comprising:
in response to determining receipt of the consecutive set of received packets, shifting the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set.
5. The method of claim 1 , wherein a packet of the received plurality of packets comprises at least one memory operation, and wherein the method further comprises causing the at least one memory operation to be performed against a region of a memory.
6. The method of claim 5 , wherein the region of the memory comprises user-space memory.
7. The method of claim 5 , wherein the at least one memory operation includes an atomic operation.
8. The method of claim 1 , wherein the window size is at least 1000 packets.
9. The method of claim 1 , wherein determining receipt of the consecutive set of received packets within the range further comprises determining the consecutive set includes a minimum number of packets.
10. The method of claim 9 , wherein the minimum number of packets is a number less than a total number of consecutive packets represented by the window.
11. The method of claim 1 , wherein the circuit is integrated into a fabric interface circuit.
12. A system comprising:
a first node device comprising:
a memory;
circuitry configured to be communicatively coupled to a multi-node network, the circuitry being configured to:
define a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
receive a connection initialization packet from a remote node;
transmit a response packet to the remote node including a maximum window size and a starting sequence number;
receive a plurality of packets from the remote node, each received packet having a unique sequence number;
determine receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
suspend transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
transmit to the remote node an indication in response to determining receipt of the consecutive set of received packets.
13. The system of claim 12 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.
14. The system of claim 12 , wherein the circuitry of the first node device is further configured to:
in response to determining receipt of the consecutive set of received packets, shift the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set.
15. The system of claim 12 , wherein the circuitry of the first node device comprises a network interface circuit.
16. A node device comprising:
a memory;
a network interface circuit to communicate on a multi-node network and to receive a plurality of packets from a remote node, each received packet having a unique sequence number;
means for defining a window that represents a range of consecutively numbered packets, the window having a starting index number corresponding to the first packet of the range;
means for receiving a connection initialization packet from the remote node;
means for transmitting a response packet to the remote node including a maximum window size and a starting sequence number;
means for determining receipt of a consecutive set of received packets within the range of consecutively numbered packets, wherein the consecutive set of received packets has consecutive sequence numbers, and a first packet of the consecutive set has a sequence number corresponding to the starting index number of the window;
means for suspending transmitting packets before transmitting a packet with a sequence number that is equal to a max sequence number, the max sequence number being calculated as a sum of the starting sequence number and the maximum window size; and
means for shifting the window such that the starting index number of the window corresponds to a packet associated with a sequence number beyond the last packet of the consecutive set in response to determining receipt of the consecutive set of received packets.
17. The node device of claim 16 , wherein the network interface circuit is further configured to transmit an indication to a remote node in response to determining receipt of the consecutive set of received packets.
18. The node device of claim 16 , wherein the indication comprises a special acknowledgement packet, the special acknowledgment packet including a sequence number corresponding to the last packet of the consecutive set of received packets within the range.