System and method for explicit communication of messages between processes running on different nodes in a clustered multiprocessor system
View Patent ↗Embodiments of the invention include a mechanism for explicit communication in a clustered multiprocessor system that supports low-latency, protected, user-mode, communication across the machine boundaries of a clustered multiprocessor. Data transport may be accomplished over persistent, unidirectional, point-to point connections, each of which may be embodied in a small amount of state at each end, along with a statically allocated per-connection memory buffer, which may be directly accessible to the transport mechanism at both ends of each notional link. System Memory protection may be afforded by operating system (“OS”) facilitated allocation of both restricted control of the network interface, and responsibility for data transport, to an application thread that may execute in the context of the processor-managed virtual address space. Connection buffer protection may be afforded by restricting access to connection state to those entries associated with the currently controlling thread.
1. An OS-bypass message transport mechanism to communicate between processor nodes that execute a process on a clustered multiprocessor system, said transport mechanism comprising:
a connection that includes a send side coupled to a source processor node and a receive side coupled to a destination processor node;
a single per-connection memory buffer comprising an output data buffer and a input data buffer coupled respectively to the send side and the receive side of the connection by a respective switch/network adaptor interface; and
dynamic common memory read and write pointers associated with the single per-connection memory buffer and each of the send side switch/network adaptor interface and the receive side switch/network adaptor interface wherein the send side and the receive side of the connection are decoupled from each other, and
wherein the output data buffer is coupled between the source processor node and the per-connection memory buffer and the input data buffer is coupled between the per-connection memory buffer and the destination processor node, and
wherein the per-connection memory buffer is dedicated to the process.
2. The OS-bypass message transport mechanism of claim 1 , wherein the per-connection memory buffer comprises a portion of a common memory of the clustered multiprocessor system.
3. The OS-bypass message transport mechanism of claim 1 , wherein said send side includes outgoing direct memory access (ODMA) engine that controls the transfer of data from the output data register to the per-connection memory buffer.
4. The OS-bypass message transport mechanism of claim 1 , comprising an IPI generator coupled to the send side and the receive side, wherein the IPI generator generates control data shared between the send side and receive side.
5. The OS-bypass message transport mechanism of claim 4 , comprising an IPI handler coupled to the send side and the receive side for processing the control data generated by the IPI generator.
6. The OS-bypass message transport mechanism of claim 1 , wherein said output data buffer and input data buffer comprise first-in-first-out (FIFO) buffers.
7. The OS-bypass message transport mechanism of claim 1 , wherein each said switch/network adaptor interface comprises a switch/network adaptor port (SNAP).
8. The OS-bypass message transport mechanism of claim 1 , wherein the send side includes an outgoing local connection table (OLCT) and the receive side includes an incoming local connection table (ILCT), and
wherein a connection is established between the send side and the receive side by entries in the OLCT and ILCT.
9. The OS-bypass message transport mechanism of claim 8 , wherein the connection is a persistent, unidirectional connection.
10. The OS-bypass message transport mechanism of claim 1 , wherein said processor nodes are coupled to a multi-adaptive processor (MAP) element.
11. The OS-bypass message transport mechanism of claim 10 , wherein the processor nodes coupled to the MAP element are configured for a dual in-line memory module (DIMM) socket.
12. The OS-bypass message transport mechanism of claim 1 , wherein the per-connection memory buffer is statically allocated.
13. A clustered multiprocessor system comprising:
an OS-bypass message transport mechanism to communicate between processor nodes that execute a process on the system, wherein said mechanism includes
a connection that includes a send side coupled to a source processor node and a receive side coupled to a destination processor node;
a single per-connection memory buffer comprising an update buffer and an input buffer coupled respectively to the send side and the receive side of the connection by a respective switch/network adaptor interface; and
dynamic common memory read and write pointers associated with the single per-connection memory buffer and each of the send side switch/network adaptor interface and the receive side switch/network adaptor interface wherein the send side and the receive side of the connection are decoupled from each other, and
wherein the output data buffer is coupled between the source processor node and the per-connection memory buffer and the input data buffer is coupled between the per-connection memory buffer and the destination processor node, and
wherein the per-connection memory buffer is dedicated to the process.
14. A method of transferring process data from a source processor node to a destination processor node that bypasses an operating system in a clustered multiprocessor system, said method comprising the steps of:
allocating a single per-connection memory buffer comprising an input buffer and an output buffer accessible by the source processor node and the destination processor node via a respective switch/network adaptor interface;
providing dynamic common memory read and write pointers associated with the single per-connection memory buffer and each of the source processor node switch/network adaptor interface and the destination processor node switch/network adaptor interface wherein the source processor node and the destination processor node of the connection are decoupled from each other;
transferring the process data from the source processor node to the output data buffer on a send side;
transferring the process data from the output data buffer to the input data buffer on a receive side; and
processing the process data from the input data buffer using the destination processor node.
15. The method of claim 14 , wherein the per-connection memory buffer is statically allocated.
16. The method of claim 14 , further comprising the step of restricting the per-connection memory buffer to the process data while transferring the process data from the source processor node to the destination processor node.
17. The method of claim 14 , wherein the source processor node and the destination processor node are coupled in a duel in-line memory module (DIMM) format.
18. The method of claim 14 , wherein each switch/network adaptor interface includes a switch/network adaptor port.