Throughput in openfabrics environments
Disclosed herein are systems, methods, and processes to improve throughput in OpenFabrics and Remote Direct Memory Access (RDMA) computing environments. Data and a header is received. Buffers in which the data and the header are to be written are identified. Placement information for the data and the header is determined based on a size of each buffer, a page-boundary-alignment of the data, and a header alignment of the header. The data and the header are written to the buffer(s) using the placement information. In such computing environments, throughout can be improved by writing data on page boundaries and the header on a header boundary in a second to last buffer.
1. A computer-implemented method comprising:
receiving data and a header;
determining a minimum number of buffers that are configured to collectively store the data and the header, wherein
the minimum number is an integer greater than or equal to two;
determining placement information for the data and the header within the buffers,
wherein
the placement information is determined based, at least in part, on
a size of each of the buffers,
a page-boundary-alignment of the data,
a header alignment of the header, and
the placement information identifies a second-to-last buffer; and
writing the data and the header to the buffers, wherein
the writing of the data uses the placement information,
the data is written on page boundaries, and
the header is written on a header boundary of the second-to-last buffer.
2. The computer-implemented method of claim 1 , wherein
using the placement information results in
utilizing the minimum number of buffers,
the data being page-boundary-aligned when written to the minimum number of buffers, and
minimizing on-wire wastage.
3. The computer-implemented method of claim 1 , further comprising:
coalescing the header and the data into a Remote Direct Memory Access (RDMA) write by mapping the header and the data comprised in a plurality of source buffers to the buffers based on the placement information.
4. The computer-implemented method of claim 3 , wherein
the RDMA write comprises a 32-bit data space.
5. The computer-implemented method of claim 4 , further comprising:
including an offset of the header in the 32-bit data space.
6. The computer-implemented method of claim 1 , further comprising:
selecting one or more additional buffers if the data cannot be page-boundary aligned in the minimum number of buffers.
7. The computer-implemented method of claim 1 , wherein
the minimum number of buffers comprise one or more destination buffers.
8. A non-transitory computer-readable storage medium storing program instructions executable to:
receive data and a header;
determine a minimum number of buffers that are configured to collectively store the data and the header, wherein
the minimum number is an integer greater than or equal to two;
determine placement information for the data and the header within the buffers, wherein the placement information is determined based, at least in part, on
a size of each of the buffers,
a page-boundary-alignment of the data,
a header alignment of the header, and
the placement information identifies a second-to-last buffer; and
write the data and the header to the buffers, wherein
writing the data uses the placement information,
the data is written on page boundaries, and
the header is written on a header boundary of the second-to-last buffer.
9. The non-transitory computer-readable storage medium of claim 8 , wherein
using the placement information results in
utilizing the minimum number of buffers,
the data being page-boundary-aligned when written to the minimum number of buffers, and
minimizing on-wire wastage.
10. The non-transitory computer-readable storage medium of claim 8 , further comprising:
coalescing the header and the data into a Remote Direct Memory Access (RDMA) write by mapping the header and the data comprised in a plurality of source buffers to the buffers based on the placement information, wherein
the RDMA write comprises a 32-bit data space; and
including an offset of the header in the 32-bit data space.
11. The non-transitory computer-readable storage medium of claim 8 , further comprising:
selecting one or more additional buffers if the data cannot be page-boundary aligned in the minimum number of buffers.
12. The non-transitory computer-readable storage medium of claim 8 , wherein
the minimum number of buffers comprise one or more destination buffers.
13. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to:
receive data and a header;
determine a minimum number of buffers that are configured to collectively store the data and the header, wherein
the minimum number is an integer greater than or equal to two;
determine placement information for the data and the header within the buffers, wherein
the placement information is determined based, at least in part, on
a size of each of the buffers,
a page-boundary-alignment of the data, and
a header alignment of the header, and
the placement information identifies a second-to-last buffer; and
write the data and the header to the buffers, wherein
writing the data uses the placement information,
the data is written on page boundaries, and
the header is written on a header boundary of the second-to-last buffer.
14. The system of claim 13 , wherein
using the placement information results in
utilizing the minimum number of buffers,
the data being page-boundary-aligned when written to the minimum number of buffers, and
minimizing on-wire wastage.
15. The system of claim 13 , further comprising:
coalescing the header and the data into a Remote Direct Memory Access (RDMA) write by mapping the header and the data comprised in a plurality of source buffers to the buffers based on the placement information, wherein
the RDMA write comprises a 32-bit data space; and
including an offset of the header in the 32-bit data space.
16. The system of claim 13 , further comprising:
selecting one or more additional buffers if the data cannot be page-boundary aligned in the minimum number of buffers.
17. The system of claim 13 , wherein
the minimum number of buffers comprise one or more destination buffers.