RDMA transport with hardware integration
Techniques are disclosed for implementing direct memory access in a virtualized computing environment. A new mapping of interfaces between RNIC Consumer and RDMA Transport is defined, which enables efficient retry, a zombie detection mechanism, and identification and handling of invalid requests without bringing down the RDMA connection. Techniques are disclosed for out of order placement and delivery of ULP Requests without constraining the RNIC Consumer to the ordered networking behavior, if it is not required for the ULP (e.g., storage). This allows efficient deployment of an RDMA accelerated storage workload in a lossy network configuration, and reduction in latency jitter.
1. A remote direct memory access (RDMA) capable network interface card (RNIC) consumer device configured to provide interface functionality between an RNIC and an RNIC consumer, the RNIC consumer device comprising a hardware-based interface configured to interact with an RDMA transport mechanism of the RNIC, the RNIC consumer device comprising hardware-based logic that configures the RNIC consumer device to perform operations comprising:
send, via the hardware-based interface to the RNIC, RDMA requests on behalf of the RNIC consumer;
receive, by the hardware-based logic from the RNIC, RDMA responses sent to the RNIC consumer;
detect invalid RMDA responses and requests;
maintain an RDMA connection between the RNIC and the RNIC consumer when an invalid RDMA response or request is detected; and
communicate with a transaction interface configured to transmit inbound upper layer protocol (ULP) responses without reading receive queue (RQ) descriptors or implementing RQ semantics.
2. The RNIC consumer device of claim 1 , wherein the RNIC is implemented as an FPGA or ASIC.
3. The RNIC consumer device of claim 1 , wherein the RNIC consumer device is configured to generate, allocate, and manage remote access identifiers.
4. The RNIC consumer device of claim 3 , wherein the remote access identifiers are a Remote Key (RKey) and virtual address (VA).
5. The RNIC consumer device of claim 4 , wherein the RNIC consumer device is configured to pass the RKey and VA in lieu of implementing memory registration and remote access semantics.
6. The RNIC consumer device of claim 4 , wherein:
the RKey and VA are associated with an RNIC consumer request; and
the RNIC consumer device is configured to use the RKey to identify an RNIC consumer request context and use the VA to specify an offset within the RNIC consumer request.
7. The RNIC consumer device of claim 6 , wherein the RNIC consumer request context is used to identify data buffers associated with the RNIC consumer request.
8. The RNIC consumer device of claim 4 , wherein the RNIC consumer device is configured to use the RKey and VA to communicate a sequence number usable to identify invalid RDMA accesses.
9. The RNIC consumer device of claim 8 , wherein the sequence number is matched against a sequence number within a request context to identify an invalid RDMA access.
10. A device configured to provide interface functionality between an RNIC and an RNIC consumer, the device comprising a hardware-based interface configured to interact with an RDMA transport mechanism of the RNIC, the RNIC consumer device comprising hardware-based logic that configures the device to perform operations comprising:
sending, via the hardware-based interface to the RNIC, RDMA requests on behalf of the RNIC consumer;
receiving, by the hardware-based logic from the RNIC, RDMA responses sent to the RNIC consumer;
detecting invalid RMDA responses and requests
maintaining an RDMA connection between the RNIC and the RNIC consumer when an invalid RDMA request is identified; and
communicating with a transaction interface configured to transmit inbound ULP responses without reading receive queue (RQ) descriptors or implementing RQ semantics.
11. The device of claim 10 , wherein:
an upper layer protocol (ULP) response includes a sequence number and an RNIC consumer request context identifier; and
a sequence number carried by the ULP response is matched against a sequence number within a request context to identify an invalid ULP Response message.
12. The device of claim 11 , further configured to perform operations comprising:
dropping invalid RDMA writes identified by the sequence number without changing a state of an RDMA connection used for RDMA writes.
13. The device of claim 10 , further configured to perform operations comprising:
generating padding data for RDMA read responses without changing a state of an RDMA connection used for RDMA writes.
14. The device of claim 13 , further configured to perform operations comprising:
inverting cyclic redundancy checks (CRCs) for the generated padding data; and
detecting invalid data transfers based on the inverted CRCs.
15. A method for implementing remote direct memory access (RDMA) in a networked computing environment comprising one or more computing devices, the method performed by an RDMA capable network interface card (RNIC) consumer device configured to provide interface functionality between an RNIC and an RNIC consumer, the RNIC consumer device comprising a hardware-based interface configured to interact with an RDMA transport mechanism of the RNIC, the RNIC consumer device comprising hardware-based logic, the method comprising:
sending, via the hardware-based interface to the RNIC, RDMA requests on behalf of the RNIC consumer;
receiving, by the hardware-based logic from the RNIC, RDMA responses sent to the RNIC consumer;
detecting invalid RMDA responses and requests
maintaining a current RDMA connection between the RNIC and the RNIC consumer when a zombie response or request is detected; and
communicating with a transaction interface configured to transmit inbound ULP responses without reading receive queue (RQ) descriptors or implementing RQ semantics.
16. The method of claim 15 , further comprising:
generating and allocating remote access identifiers; and
communicating the remote access identifiers to a requesting application in lieu of enabling memory registration and remote access semantics.
17. The method of claim 16 , further comprising:
based on the remote access identifiers, identifying an RNIC consumer request context and specifying an offset within an RNIC consumer request; and
based on the RNIC consumer request context, identifying data buffers associated with the RNIC consumer request.
18. The method of claim 16 , further comprising using the remote access identifiers to communicate a sequence number usable to identify invalid RDMA accesses, wherein the sequence number is matched against a sequence number within a request context to identify an invalid RDMA access.
19. The method of claim 16 , further comprising:
matching a sequence number carried by an upper layer protocol (ULP) response against a sequence number within a request context to identify an invalid ULP Response message.