System and method for low overhead message passing between domains in a partitioned server
View Patent ↗A system and method for passing messages between domains with low overhead in a multi-node computer system. A CPU node in a sending domain issues a request to a memory node in a receiving domain using memory-mapped input/output window. This causes the message to be transmitted to a coherent space of the receiving domain. All messages are cache-line in size. A small portion of each cache line, cyclic counter field, is overwritten before the cache line is written in the coherent address space of the receiving domain. A massaging driver polls the cyclic count field of the cache line in the processor cache to determine when the next message is written in the coherent address space of the receiving domain. This allows the CPU to detect when the last received message is written into the coherent address space of the receiving domain without generating transactions on CPU interface.
1. In a multi-node computer system partitioned into a sending domain and a receiving domain, the sending domain and the receiving domain each having a processor node, an I/O node, and a memory node communicatively coupled through an interconnect, a method for message passing between the sending domain and the receiving domain, comprising:
receiving from a processor node, a write request to a memory-mapped input/output (MMIO) window in an address space of the sending domain, the request including an address;
comparing a portion of the address with a value stored in a base register, wherein the value indicates where the MMIO window starts in the address space of the sending domain; and
responsive to the portion of the address matching the value in the base register:
decoding the receiving domain;
combining a plurality of write requests; and
responsive to the plurality of the write requests having a cache-line size,
transmitting the cache-line size write request from the sending domain to the receiving domain.
2. The method of claim 1 , further comprising combining internally a plurality of processor write requests in the cache-line size write request using a write combining mechanism.
3. In a computer system comprising a sending domain and a receiving domain, the sending domain and the receiving domain each includes a plurality of CPU nodes, each CPU node communicatively coupled to at least one processor cache memory, a method for providing an in-memory notification when a cache-line sized message arrives in the receiving domain, the method comprising repeatedly performed steps of:
receiving the cache-line size message in the receiving domain;
acquiring exclusive ownership of a next cache line entry in a message receive buffer residing in the receiving domain;
modifying a cyclic count field in the received cache-line sized message;
writing the received cache-line size message to the exclusively owned cache line entry in the message receive buffer;
polling the cyclic count field in the at least one processor cache memory to detect a continuation of the cyclic sequence;
issuing a miss in the at least one processor cache memory;
installing the received cache-line sized message in the at least one processor cache memory; and
polling the cyclic count field of the cache-line sized message in the at least one processor cache memory to notify when the last message is written to the message receive buffer.
4. The method of claim 3 , wherein the last message written in the message receive buffer is detected when a cyclic sequence of the cyclic count field in sequentially addressed cache lines is broken.
5. The method of claim 3 , wherein the step of inserting a cyclic counter further comprises overwriting the cyclic counter field with a new value when a new cache line size message is written in the message receive buffer.
6. The method of claim 3 , wherein the cyclic counter is a modulo N counter, wherein N is prime with respect to a number of cache lines in the message receive buffer.
7. The method of claim 3 , further comprising:
checking the cache-line sized message for error using error correction code (ECC);
writing the cache-line sized message to a holding FIFO unit; and
regenerating the ECC code.
8. A multi-node computer system for sending a plurality of messages between a sending domain and a receiving domain connected by an interconnect, the system comprising:
a processor node adapted to issue a write request to a memory-mapped input/output (MMIO) window in an address space of the sending domain, the request including an address;
a sending engine residing in the sending domain adapted to:
compare a portion of the address with a value stored in a base register, wherein the value indicates where the MMIO window starts in the address space of the sending domain,
responsive to the portion of the address matching the value in the base register:
decode the receiving domain,
combine a plurality of write requests; and
responsive to the plurality of the write requests having a cache-line size, transmit a cache-line size write request from the sending domain to the receiving domain.
9. The system of claim 8 , further comprising a receiving engine residing in the receiving domain, the receiving engine adapted to write the cache-line size request to a message receive buffer in the receiving domain.
10. The system of claim 8 , wherein the address comprises:
a domain field for indicating a receiving domain.
11. The system of claim 8 , wherein the address comprises:
a size field indicating a number of cache lines that can be written into the MMIO window.
12. The system of claim 9 , wherein the message receive buffer stores 2 W cache lines, and wherein the number of stored cache lines is prime with respect to the modulus of a cyclic counter.