Low latency communication via memory windows
View Patent ↗A high performance computer system has compute nodes interconnected by an inter-node communication network. The compute nodes include local packetized interconnects. Shared memory between compute nodes is emulated by tunneling packetized interconnect packets between the compute nodes. A memory region on one of the compute nodes is used as a window into a memory region in another compute node. Special registers may be provided to expedite remote operations on small amounts of data.
1. A method for exchanging data between compute nodes of a computer system having a plurality of compute nodes interconnected by an inter-node communication network, each of the compute nodes having a distinct address space and a network interface interconnecting the inter-node communication network to a local packetized interconnect, the method comprising:
providing a register accessible to the network interface of a sending one of the compute nodes, the register associated with a receiving one of the compute nodes;
under the control of independent control logic in the network interface of the sending compute node detecting that the register has been modified;
in response to modification of the register automatically generating a local packetized interconnect packet addressed to an address specified by a value from the register in an address space of the receiving compute node corresponding to the register; encapsulating the local packetized interconnect packet in an inter-node communication network packet; and forwarding the inter-node communication network packet to the receiving compute node,
wherein the register comprises an atomic read-modify-write register comprising a comparison field and a replacement value field; and generating the local packetized interconnect packet comprises generating a local packetized interconnect atomic read-modify-write packet containing a comparison from the comparison field and a replacement value from the replacement value field.
2. A method according to claim 1 comprising providing a plurality of registers at the sending compute node, each of the plurality of registers associated with a different receiving compute node.
3. A method according to claim 1 wherein the register comprises a remote memory access register comprising a data field and the method comprises including data from the data field in the local packetized interconnect packet.
4. A method according to claim 2 comprising, at the receiving compute node performing a memory access operation in response to the local packetized interconnect packet.
5. A method according to claim 4 wherein the memory access operation comprises a memory write operation.
6. A method according to claim 5 comprising forwarding a write confirmation message from the receiving compute node to the sending compute node upon completion of the memory write operation.
7. A method according to claim 1 comprising, at the receiving compute node, performing an atomic read-modify-write operation in response to the local packetized interconnect atomic read-modify-write packet.
8. A method according to claim 1 wherein the register comprises a remote read register and generating the local packetized interconnect packet comprises generating a local packetized interconnect read request packet.
9. A method according to claim 8 comprising, at the receiving compute node, performing a memory read operation in response to the local packetized interconnect read request packet.
10. A method according to claim 9 comprising forwarding one or more local packetized read response packets generated at the receiving compute node in response to the memory read operation to the sending compute node.
11. A computer system comprising
a plurality of compute nodes interconnected by an inter-node communication network, each of the compute nodes having an independent address space and comprising:
a local packetized interconnect,
a network interface coupled to the local packetized interconnect and the inter-node communication network,
at least one data processor coupled to the local packetized interconnect; and,
a memory system coupled to the local packetized interconnect;
wherein a sending one of the compute nodes comprises a register accessible to the network interface of the sending compute node, the register associated with a receiving one of the compute nodes; and wherein, the network interface of the sending compute node comprises independent control logic configured to:
detect modification of the register;
in response to a detected modification of the register automatically generate a local packetized interconnect packet addressed to an address specified by a value from the register in an address space of the receiving compute node corresponding to the register;
encapsulate the local packetized interconnect packet in an inter-node communication network packet; and,
forward the inter-node communication network packet to the receiving compute node,
wherein the register comprises an atomic read-modify-write register comprising a comparison field and a replacement value field; and
the network interface of the sending compute node is configured to generate a local packetized interconnect atomic read-modify-write packet containing a comparison from the comparison field and a replacement value from the replacement value field in response to the detected modification of the register.
12. A computer system according to claim 11 wherein the sending node comprises a plurality of registers each associated with a different one of a corresponding plurality of receiving ones of the compute nodes.
13. A computer system according to claim 12 wherein each of the compute nodes constitutes a sending one of the compute nodes and each of the compute nodes comprises:
a plurality of registers each associated with a different one of a corresponding plurality of receiving ones of the compute nodes; and
wherein, the network interface of each of the compute nodes comprises independent control logic configured to:
detect modification of the registers;
in response to a detected modification of one of the registers, automatically generate a local packetized interconnect packet addressed to an address specified by a value from the register in an address space of the receiving compute node corresponding to the register;
encapsulate the local packetized interconnect packet in an inter-node communication network packet; and,
forward the inter-node communication network packet to the receiving compute node.
14. A computer system according to claim 11 wherein the register comprises a remote memory access register comprising a data field and the network interface is configured to include data from the data field in the local packetized interconnect packet.
15. A computer system according to claim 11 wherein the register comprises a remote read register and the network interface of the sending compute node is configured to generate a local packetized interconnect read request packet in response to the detected modification of the register.
16. A computer system according to claim 11 wherein the sending compute node comprises a register set comprising a plurality of registers, each associated with a memory operation and the network interface of the sending compute node is configured to, in response to detecting modification of one of the registers, automatically generate a local packetized interconnect packet that includes a request for performance of the memory operation associated with the modified one of the registers.
17. A computer system according to claim 16 wherein the sending node comprises a plurality of the register sets, each of the register sets associated with a different one of a corresponding plurality of receiving ones of the compute nodes.
18. A computer system according to claim 16 wherein the sending node comprises a plurality of the register sets, each of the register sets associated with a different region of memory in the receiving compute node.