System and method for tracking infiniband RDMA read responses
View Patent ↗A system and method for tracking responses to InfiniBand RDMA Reads. When an RDMA Read or Read request is issued by a transmit module, a receive module is informed of the packet sequence numbers (PSN) associated with the expected RDMA Read responses. The receive module maintains a linked list for each queue pair that issues RDMA Reads. Each entry in the linked list corresponds to one RDMA Read for the associated queue pair, and identifies the first and last PSN and includes a link to the next entry in the linked list. When the final RDMA Read response is received, the receive module notifies the transmit module, which can then retire the RDMA Read from its retry queue.
1. A method of processing traffic received from an InfiniBand node via a first queue pair, comprising:
selecting a traffic entry in an InfiniBand receive queue, wherein said traffic entry comprises one of:
a Send command comprising an encapsulated communication;
a Send command comprising a remote direct memory access (RDMA) Read descriptor; and
an RDMA Read response comprising a response to an RDMA Read request;
if said selected traffic entry comprises a Send command comprising an RDMA Read descriptor:
issuing a first RDMA Read request to retrieve one or more portions of a communication described by said RDMA Read descriptor;
in a linked list corresponding to the first queue pair, adding an entry corresponding to said first RDMA Read request, said entry comprising a first sequence number corresponding to a first expected response to said first RDMA Read request and a last sequence number corresponding to a last expected response to said first Read request, and optionally comprising a sequence number of the most recently received response or a link to the next entry in the linked list; and
in a retry queue, adding an entry corresponding to said first RDMA Read request; and
if said selected traffic entry comprises an RDMA Read response to said first RDMA Read request:
identifying a sequence number associated with said RDMA Read response;
comparing said sequence number to said range of sequence numbers;
storing said one or more portions of said described communication to facilitate assembly of said described communication in said queue; and
if said sequence number matches a final sequence number in said range, retiring in said retry queue said entry corresponding to said first RDMA Read request.
2. The method of claim 1 , further comprising:
forwarding a communication associated with said selected traffic entry, for transmission on an external communication link, wherein said communication is one of:
said encapsulated communication; and
said described communication, after said described communication is assembled.
3. The method of claim 1 , further comprising, if said selected traffic entry comprises an RDMA Read response to said first RDMA Read request:
if said sequence number does not match said final sequence number, updating said entry in said linked list to include said sequence number.
4. The method of claim 1 , further comprising:
maintaining a single memory structure comprising multiple linked list, including said linked list;
wherein each linked list stores entries associated with RDMA Read requests for a different InfiniBand queue pair.
5. The method of claim 1 , further comprising:
maintaining a single memory structure for queuing InfiniBand traffic received via multiple virtual lanes and multiple queue pairs, said single memory structure comprising said queue.
6. The method of claim 5 , wherein said queue comprises a linked list of memory buffers within said single memory structure.
7. The method of claim 1 , further comprising:
maintaining a head pointer configured to identify a head of said linked list; and
maintaining a tail pointer configured to identify a tail of said linked list.
8. The method of claim 1 , further comprising:
maintaining a head pointer configured to identify a head of said queue;
maintaining a tail pointer configured to identify a tail of said queue; and
maintaining a next traffic entry pointer configured to identify a next entry in said queue to be processed.
9. The method of claim 8 , wherein said tail pointer is configured to identify where in said queue a next traffic entry is to be queued.
10. The method of claim 1 , further comprising, if said selected traffic entry comprises an RDMA Read descriptor:
appending space to a head of said queue;
wherein said described communication is assembled in said appended space.
11. The method of claim 1 , further comprising, if said selected traffic entry comprises an RDMA Read response to said first RDMA Read request:
dropping an RDMA Read response received out of order; and
requesting a retry of said first RDMA Read request.
12. A computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method of processing traffic received from an InfiniBand node via a first queue pair, the method comprising:
selecting a traffic entry in an InfiniBand receive queue, wherein said traffic entry comprises one of:
a Send command comprising an encapsulated communication;
a Send command comprising an RDMA Read descriptor; and
an RDMA Read response comprising a response to an RDMA Read request;
if said selected traffic entry comprises a Send command comprising an RDMA Read descriptor:
issuing a first RDMA Read request to retrieve one or more portions of a communication described by said RDMA Read descriptor;
in a linked list corresponding to the first queue pair, adding an entry corresponding to said first RDMA Read request, said entry comprising a first sequence number corresponding to a first expected response to said first RDMA Read request and a last sequence number corresponding to a last expected response to said first RDMA Read request, and optionally comprising a sequence number of the most recently received response or a link to the next entry in the linked list; and
in a retry queue, adding an entry corresponding to said first RDMA Read request; and
if said selected traffic entry comprises an RDMA Read response to said first RDMA Read request:
identifying a sequence number associated with said RDMA Read response;
comparing said sequence number to said range of sequence numbers;
storing said one or more portions of said described communication to facilitate assembly of said described communication in said queue; and
if said sequence number matches a final sequence number in said range, retiring in said retry queue said entry corresponding to said first RDMA Read request.