System and method for one-sided read RMA using linked queues
A system for one-sided read remote memory access is disclosed. In some embodiments, the system is configured to receive, at a responder SFA, a first packet comprising a read request to read a remote memory of a second host from a first host, wherein a payload of the first packet is mapped to be a transmit header queue (TxHQ) entry (TxHQE), and the TxHQE includes a pointer to a memory map; separate, the received packet into portions including a upper level protocol (ULP) portion, the ULP portion being the TxHQE; create a ULP header queue for the TxHQE; generate a read response based on mapping the ULP header queue into hardware as the TxHQ, wherein the TxHQE includes a pointer to data from a valid memory region of the second host identified by the memory mapping; and transmit a read response packet with the data identified by the pointer using the TxHQ to the first host.
1 . A server fabric adapter (SFA) communication system comprising:
a requester SFA communicatively coupled to a first host, the first host comprising a local memory; and
a responder SFA communicatively coupled to a second host, the second host comprising a remote memory;
wherein the requester SFA is configured to:
receive, from the first host, a first packet comprising a read request to read the remote memory of the second host, the first packet including a pointer to a memory mapping at the second host;
receive, from the responder SFA, a second packet comprising a read response and data retrieved from the second host using the memory mapping, wherein a completion queue (CQ) counter associated with a responder receive header queue (RxHQ) of the second host is mapped to a doorbell address of a responder transmit header queue (TxHQ), and the second packet is sent in response to a ringing of a doorbell identified by the doorbell address; and
separate the second packet into the completion queue counter and a receive header queue entry (RxHQE) in a requester RxHQ of the first host, wherein the RxHQE of the first host includes the data retrieved using the memory mapping.
2 . The SFA communication system of claim 1 , wherein a CQ corresponds to the requester RxHQ and indicates the RxHQE where the second packet is stored.
3 . The SFA communication system of claim 1 , prior to receiving the first packet comprising the read request, the requester SFA is further configured to:
receive, from the first host, a communication message for registering the remote memory of the second host;
receive memory mappings of the second host as a response to the communication message; and
transmit the memory mappings of the second host to the first host, wherein the first host mirrors the memory mappings of the second host as its local memory mappings in the first host.
4 . The SFA communication system of claim 1 , wherein the requester SFA is further configured to:
map a payload of the first packet as a transmit header queue entry (TxHQE) in a requester transmit header queue (TxHQ), the TxHQE including the pointer to a scatter gather list (SGL).
5 . The SFA communication system of claim 1 , wherein the requester SFA is further configured to:
receive an interrupt triggered by a CQ associated with the requester RxHQ to examine an entry in the requester RxHQ.
6 . The SFA communication system of claim 1 , wherein the first and second packets are transmitted using a network protocol, and the network protocol includes user datagram protocol (UDP).
7 . The SFA communication system of claim 1 , wherein the responder SFA is configured to:
receive the first packet;
generate the second packet; and
send the second packet to the requester SFA.
8 . The SFA communication system of claim 7 , wherein the responder SFA is further configured to:
create the responder receive header queue (RxHQ) for storing packet headers and the read request included in the first packet; and
create a CQ associated with the responder RxHQ for storing the CQ counter associated with the responder RxHQ.
9 . The SFA communication system of claim 8 , wherein a payload of the first packet includes a requester transmit header queue entry (TxHQE) in a requester TxHQ, and the requester TxHQE includes the pointer, and wherein the notification counter indicates a number of responder RxHQ entries that have been received at the second host and also indicates a number of valid responder TxHQEs.
10 . The SFA communication system of claim 9 , wherein the responder SFA is further configured to:
map the notification counter to the doorbell address, wherein the doorbell identified by the doorbell address indicates that the responder TxHQ is ready to be consumed.
11 . The SFA communication system of claim 7 , wherein the responder SFA is further configured to:
create a CQ associated with the responder TxHQ, wherein the CQ associated with the responder TxHQ indicates that a transmission is completed; and
in response to a transmission being completed, release the memory mapping and corresponding buffers.
12 . The SFA communication system of claim 11 , wherein the responder SFA is further configured to:
communicate with the first host to determine if data retransmission is needed; and
mark a buffer of the second host as invalid before releasing the buffer.
13 . The SFA communication system of claim 11 , wherein the responder SFA is further configured to fire an interrupt when the CQ associated with the responder TxHQ has activity.
14 . The SFA communication system of claim 7 , wherein the responder SFA is further configured to:
control a mapped memory of the second host, wherein the control includes canceling a requested read operation.
15 . A method for one-sided read remote memory access, the method comprising:
receiving, at a requester server fabric adapter (SFA) and from a first host, a first packet comprising a read request to read a remote memory of a second host, the first packet including a pointer to a memory mapping at the second host;
receiving, from a responder SFA associated with the second host, a second packet comprising a read response and data retrieved from the second host using the memory mapping, wherein a completion queue (CQ) counter associated with a responder receive header queue (RxHQ) of the second host is mapped to a doorbell address of a responder transmit header queue (TxHQ), and the second packet is sent in response to a ringing of a doorbell identified by the doorbell address; and
separating, by the requester SFA, the second packet into a completion queue counter and a receive header queue entry (RxHQE) in a requester RxHQ of the first host, wherein the RxHQE of the first host includes the data retrieved using the memory mapping.
16 . The method of claim 15 , further comprising:
receiving, from the first host, a communication message for registering the remote memory of the second host;
receiving memory mappings of the second host as a response to the communication message; and
transmitting the memory mappings of the second host to the first host, wherein the first host mirrors the memory mappings of the second host as its local memory mappings in the first host.
17 . The method of claim 15 , further comprising:
receiving the first packet at the responder SFA, the first packet comprising the read request;
generating the second packet at the responder SFA; and
sending the second packet from the responder SFA to the requester SFA.
18 . The method of claim 17 , further comprising:
creating, by the responder SFA, the responder RxHQ for storing packet headers and the read request included in the first packet; and
creating, by the responder SFA, a CQ associated with the responder RxHQ for storing a notification counter associated with the responder RxHQ.
19 . The method of claim 18 , wherein a payload of the first packet includes a requester transmit header queue entry (TxHQE) in a requester TxHQ, and the requester TxHQE includes the pointer, and wherein the notification counter indicates a number of responder RxHQ entries that have been received at the second host and also indicates a number of valid responder TxHQEs.
20 . The method of claim 19 , further comprising:
mapping the notification counter to the responder TxHQ doorbell address, wherein the doorbell indicates that the responder TxHQ is ready to be consumed.