Asymmetric fulfillment of remote procedure calls by multi-core systems
A method of performing a remotely-initiated procedure on a computing device is provided. The method includes (a) receiving, by memory of the computing device, a request from a remote device via remote direct memory access (RDMA); (b) in response to receiving the request, assigning processing of the request to one core of a plurality of processing cores of the computing device, wherein assigning includes the one core receiving a completion signal from a shared completion queue (Shared CQ) of the computing device, the Shared CQ being shared between the plurality of cores; and (c) in response to assigning, performing, by the one core, a procedure described by the request. An apparatus, system, and computer program product for performing a similar method are also provided.
1. A method of performing a remotely-initiated procedure on a computing device, the method comprising:
receiving, by memory of the computing device, a request from a remote device via remote direct memory access (RDMA), wherein receiving the request includes receiving, from a remote core specific send queue of the remote device, a request descriptor that describes the request at a shared receive queue (SRQ) of the computing device, the SRQ being shared between a plurality of processing cores of the computing device, and wherein the remote core specific send queue is one of a plurality of core-specific send queues in the remote device that are each paired to a respective remote processing core of the remote device, each remote core specific send queue being configured to synchronize with the SRQ, the remote core specific send queue corresponding to a remote processing core of the remote device having a different local core number than a local core number of one processing core of the plurality of processing cores of the computing device;
in response to receiving the request, assigning processing of the request to the one processing core of the plurality of processing cores of the computing device, wherein assigning includes the one processing core receiving a completion signal from the shared completion queue (Shared CQ) of the computing device, the Shared CQ being shared between the plurality of processing cores of the computing device, wherein the completion signal from the Shared CQ reports completed receipt of a request descriptor that describes the request, wherein the assigning further includes, prior to the one processing core receiving the completion signal from the Shared CQ, polling the Shared CQ by the one processing core for a completion signal, the one processing core being the first processing core of the plurality of processing cores to poll the Shared CQ after receipt of the request, each of the plurality of processing cores being configured to poll the Shared CQ when that processing core has available processing cycles;
in response to assigning, performing, by the one processing core, a procedure described by the request; and
sending a reply to the remote device using a core-specific send queue, each of the plurality of processing cores having a respective core-specific send queue.
2. A method of performing a remotely-initiated procedure on a computing device, the method comprising:
allocating, by a first processing core of a plurality of processing cores of the computing device, a memory buffer in which to store a request from a remote device;
receiving, by memory of the computing device, the request from the remote device via remote direct memory access (RDMA), wherein receiving the request includes receiving, from a remote core specific send queue of the remote device, a request descriptor that describes the request at a shared receive queue (SRQ) of the computing device, the SRQ being shared between a plurality of processing cores of the computing device, and wherein the remote core specific send queue is one of a plurality of core-specific send queues in the remote device that are each paired to a respective remote processing core of the remote device, each remote core specific send queue being configured to synchronize with the SRQ, the remote core specific send queue corresponding to a remote processing core of the remote device having a different local core number than a local core number of a second processing core of the plurality of processing cores of the computing device different from the first processing core;
in response to receiving the request, assigning processing of the request to the second processing core of the plurality of processing cores of the computing device, wherein assigning includes the second core receiving a completion signal from a shared completion queue (Shared CQ) of the computing device, the Shared CQ being shared between the plurality of processing cores of the computing device, wherein the completion signal from the Shared CQ reports completed receipt of a request descriptor that describes the request, wherein the assigning further includes, prior to the second processing core receiving the completion signal from the Shared CQ, polling the Shared CQ by the second processing core for a completion signal, the second processing core being the first processing core of the plurality of processing cores to poll the Shared CQ after receipt of the request, each of the plurality of processing cores being configured to poll the Shared CQ when that processing core has available processing cycles; and
in response to assigning, performing, by the second processing core, a procedure described by the request.
3. The method of claim 2 wherein allocating the memory buffer in which to store the request includes allocating the memory buffer from a shared pool of memory of the computing device, the shared pool of memory being shared by the plurality of processing cores.
4. A system for performing a remotely-initiated procedure, the system comprising:
a first computing device, the first computing device including a first plurality of processing cores;
a network connection; and
a second computing device, the second computing device including a second plurality of processing cores, the second computing device being communicatively coupled to the first computing device via the network connection, the second computing device being configured to:
receive, by memory of the second computing device, a request from the first computing device via remote direct memory access (RDMA) over the network connection, wherein receipt of the request includes receipt, from a remote core specific send queue of the first computing device, a request descriptor that describes the request at a shared receive queue (SRQ) of the second computing device, the SRQ being shared between the second plurality of processing cores of the second computing device, and wherein the remote core specific send queue is one of a plurality of core-specific send queues in the first computing device that are each paired to a respective remote processing core of the first computing device, each remote core specific send queue being configured to synchronize with the SRQ, the remote core specific send queue corresponding to a remote processing core of the first computing device having a different local core number than a local core number of one processing core of the second plurality of processing cores of the second computing device;
in response to receiving the request, assign processing of the request to the one processing core of the second plurality of processing cores of the second computing device, wherein assigning includes the one processing core receiving a completion signal from a shared completion queue (Shared CQ) of the second computing device, the Shared CQ being shared between the second plurality of processing cores of the second computing device, wherein the completion signal from the Shared CQ reports completed receipt of a request descriptor that describes the request, wherein the assigning further includes, prior to the one processing core receiving the completion signal from the Shared CQ, polling the Shared CQ by the one processing core for a completion signal, the one processing core being the first processing core of the second plurality of processing cores to poll the Shared CQ after receipt of receiving the request, each of the second plurality of processing cores being configured to poll the Shared CQ when that processing core has available processing cycles; and
in response to assigning, perform, by the one processing core, a procedure described by the request.
5. The system of claim 4 wherein the second computing device is further configured to send a reply to the first computing device using a core-specific send queue, each of the second plurality of processing cores having a respective core-specific send queue.
6. The system of claim 4 wherein the second computing device is further configured to, prior to receiving the request, allocate, by a different processing core of the second plurality of cores, a memory buffer in which to store the request.
7. The system of claim 6 wherein allocating the memory buffer in which to store the request includes allocating the memory buffer from a shared pool of memory of the second computing device, the shared pool of memory being shared by the second plurality of processing cores.
8. The system of claim 4 wherein the first computing device is configured to:
prior to sending the request to the second computing device, allocating, by a processing core of the first plurality of processing cores, from a shared pool of memory of the first computing device, a memory buffer in which to store a reply from the second computing device, the shared pool of memory being shared by the first plurality of processing cores; and
in response to the second computing device performing the procedure described by the request, receiving the reply from the second computing device via RDMA over the network connection and storing the reply in the allocated memory buffer.
9. The system of claim 8 wherein the first computing device is further configured to:
in response to the first computing device receiving the reply from the second computing device, assign processing the reply to another processing core of the first plurality of processing cores; and
process the reply by the other processing core of the first plurality of processing cores.
10. The system of claim 8 wherein the first computing device is further configured to: at the other processing core of the first plurality of processing cores that the reply was in response to the request initiated by the processing core of the plurality of cores;
in response to determining, reassign processing the reply to the
processing core of the first plurality of cores; and
process the reply by the processing core of the first plurality of cores.
11. A computer program product comprising a non-transitory computer-readable storage medium storing instructions, which, when executed by processing circuitry of a computing device, cause the computing device to perform a remotely-initiated procedure by:
receiving, by memory of the computing device, a request from a remote device via remote direct memory access (RDMA), including receiving at a shared receive queue (SRQ) of the computing device, a request descriptor that describes the request, the SRQ being shared between a plurality of processing cores of the processing circuitry, wherein receiving the request descriptor includes receiving the request descriptor from a remote core-specific send queue of the remote device, the remote device including a plurality of core-specific send queues that are each paired to a respective remote processing core of the remote device, each remote core-specific send queue being configured to synchronize with the SRQ, the remote core specific send queue corresponding to a remote processing core of the remote device having a different local core number than a local core number of one processing core of the plurality of processing cores of the computing device;
in response to receiving the request, assigning processing of the request to the one processing core of the plurality of processing cores, wherein assigning includes the one processing core receiving a completion signal from a shared completion queue (Shared CQ) of the computing device, the Shared CQ being shared between the plurality of processing cores of the computing device, wherein the completion signal from the Shared CQ reports completed receipt of the request descriptor that describes the request, wherein the assigning further includes, prior to the one processing core receiving the completion signal from the Shared CQ, polling the Shared CQ by the one processing core for a completion signal, the one processing core being the first processing core of the plurality of processing cores to poll the Shared CQ after receipt of the request, each of the plurality of processing cores being configured to poll the Shared CQ when that processing core has available processing cycles; and
in response to assigning, performing, by the one processing core, a procedure described by the request.
12. The method of claim 11 , wherein the completion signal from the Shared CQ reports completed receipt of the request descriptor by the associated SRQ;
wherein the request descriptor points to a memory address within the memory of the computing device of a request buffer allocated from a shared request buffer pool that is shared by the plurality of processing cores and that is associated with the Shared CQ and the SRQ; and
wherein the request buffer stores the request.