Reducing number of write operations relative to delivery of out-of-order RDMA send messages by managing reference counter
View Patent ↗An RNIC implementation that performs direct data placement to memory where all segments of a particular connection are aligned, or moves data through reassembly buffers where all segments of a particular connection are non-aligned. The type of connection that cuts-through without accessing the reassembly buffers is referred to as a “Fast” connection because it is highly likely to be aligned, while the other type is referred to as a “Slow” connection. When a consumer establishes a connection, it specifies a connection type. The connection type can change from Fast to Slow and back. The invention reduces memory bandwidth, latency, error recovery using TCP retransmit and provides for a “graceful recovery” from an empty receive queue. The implementation also may conduct CRC validation for a majority of inbound DDP segments in the Fast connection before sending a TCP acknowledgement (Ack) confirming segment reception.
1. A method of reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the method comprising the steps of:
providing a reference counter with a completion queue element;
setting the reference counter to a number of RDMA Send messages completed for a selected TCP hole;
reducing the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and
removing the CQE from a respective completion queue where the reference counter becomes zero.
2. The method of claim 1 , further comprising the step of having the completion queue element include at least part of completion data.
3. The method of claim 2 , wherein a remainder of the completion data is included in a work queue element associated with one or more RDMA Send messages.
4. The method of claim 2 , further comprising the step of indicating one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages.
5. The method of claim 1 , further comprising the step of updating a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages.
6. The method of claim 5 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages.
7. The method of claim 1 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing.
8. A computer system for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the computer system comprising:
at least one processing unit;
a memory operably associated with the at least one processing unit; and
at least one RDMA enabled network interface controller storable in the memory and executable by the at least one processing unit for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the tool including:
means for setting a reference counter of a completion queue element to a number of RDMA Send messages completed for a selected TCP hole;
means for reducing the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and
means for removing the completion queue element from a respective completion queue in a case that the reference counter becomes zero.
9. The computer system of claim 8 , further comprising means for updating a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages.
10. The computer system of claim 9 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages.
11. The computer system of claim 8 , further comprising means for storing at least part of completion data in the completion queue element and a remainder of the completion data in a work queue element associated with one or more RDMA Send messages.
12. The computer system of claim 11 , further comprising means for indicating one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages.
13. The computer system of claim 8 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing.
14. A computer program product comprising a computer useable storage medium having computer readable program code embodied therein for reducing a number of write operations relative to delivery of out-of-order RDMA Send messages, the program product comprising:
program code configured to set a reference counter associated with a completion queue element to a number of RDMA Send messages completed for a selected TCP hole;
program code configured to reduce the reference counter by one for each Poll-for-Completion conducted by an RDMA verb interface; and
program code configured to remove the completion queue element from a respective completion queue in a case that the reference counter becomes zero.
15. The program product of claim 14 , further comprising program code configured to store at least part of completion data in the completion queue element.
16. The program product of claim 15 , wherein the storing program code stores a remainder of the completion data in a work queue element associated with one or more RDMA Send messages.
17. The program product of claim 15 , further comprising an indicator of one of: 1) the completion queue element includes all completion data, and 2) completion queue element completion data is at least partially included in a work queue element associated with one or more RDMA Send messages.
18. The program product of claim 15 , wherein a number of write operations equals N+1, where N is a number of completed RDMA Send messages that were pending prior to the selected TCP hole closing.
19. The program product of claim 14 , further comprising program code configured to update a work queue element of a send queue only if it has greater than a threshold number of pending RDMA Send messages.
20. The program product of claim 14 , wherein the threshold is commensurate with resources allocated for storing information for pending RDMA Send messages.