Mechanisms for synchronizing data transfers between non-uniform memory architecture computers

1. A method for synchronizing memories on a first node and a second node, the first node comprising a direct memory access (DMA) engine for transferring data from the first node to the second node over a communication link between the first and the second nodes, the second node comprising a processor complex and independent memory controllers, the method comprising:

the DMA engine reading a control block for a data transfer from a sequence of control blocks;

in response to the control block, the DMA engine sending data specified by the control block over the communication link to the second node;

when the control block does not specify synchronization between the memories on the first and the second nodes:

the DMA engine sending an end of transfer (EOT) message after a last datum of the control block and incrementing a counter for EOT messages to the second node; and

after sending the EOT message, the DMA engine processing any subsequent control block; and

when the control block specifies synchronization between the memories on the first and the second nodes:

the DMA engine determining if the counter is at zero so there is no unacknowledged EOT message;

when the counter is at zero, the DMA engine sending an EOT message with a synchronization flag after the last datum and incrementing the counter; and

after sending the EOT message with the synchronization flag, the DMA engine determining if the counter is again at zero so the EOT message with the synchronization flag has been acknowledged;

when the counter is again at zero, the DMA engine processing any subsequent control block.

2. The method of claim 1 , further comprising:

one of the memory controllers receiving the data specified by the control block and the EOT message with or without the synchronization flag;

the memory controller pushing the data to a memory queue for a memory in the second node;

the memory controller extracting the EOT message and determining if it includes the synchronization flag; and

when the EOT message does not include the synchronization flag, the memory controller sending an EOT acknowledgment without a synchronization flag over the communication link to the first node; and

when the EOT message includes the synchronization flag, the memory controller sending an EOT acknowledgment with a synchronization flag over the communication link to the first node.

3. The method of claim 2 , wherein the second node further comprises a link interface coupled to the communication link, the method further comprising:

the link interface receiving the EOT acknowledgement with or without the synchronization flag;

the link interface forwarding the EOT acknowledgment over the communication link to the first node;

the link interface determining if the EOT acknowledgment includes the synchronization flag; and

when the EOT acknowledgement includes the synchronization flag, the link interface sending an interrupt to the processor complex of the second node.

4. The method of claim 3 , further comprising:

the DMA engine receiving the EOT acknowledgement with or without the synchronization flag from the second node; and

the DMA engine decrementing the counter in response to the EOT acknowledgement.

5. The method of claim 4 , wherein the control block comprises an operation, a length of the data, a destination interrupt, a starting destination address, and one or more starting source addresses.

6. A system, comprising:

a first node, comprising:

a first data cache memory; and

a first application specific integrated circuit (ASIC), comprising:

a first memory controller coupled to the first data cache memory;

a first link interface; and

a direct memory access (DMA) engine coupled to the first memory controller and the first link interface, the DMA engine comprising:

a counter; and

a DMA control logic;

a second node, comprising:

a processor complex;

a control cache memory;

a chipset coupled to the processor complex and the control cache memory;

switches coupled to the chipset;

a second data cache memory; and

a second ASIC, comprising:

local interfaces to the second switches;

a second link interface; and

a second memory controller coupled to the second cache memory; and

a communication link coupled to the first and the second link interfaces;

when the DMA control logic is configured to process a sequence of control blocks, comprising:

when a control block does not specify synchronization between the first and the second data cache memories on the first and the second nodes:

sending an end of transfer (EOT) message after a last datum of the control block and incrementing a counter for EOT acknowledgments; and

after sending the EOT message, processing any subsequent control block; and

when a control block specifies synchronization between the first and the second data cache memories on the first and the second nodes:

when the counter is at zero, sending an EOT message with a synchronization flag after the last datum and incrementing the counter;

after sending the EOT message with the synchronization flag, determining if the counter is again at zero; and

when the counter is again at zero, processing any subsequent control block.

7. The system of claim 6 , wherein the second memory controller is configured to process EOT messages from the first node, comprising:

extracting the EOT message with or without the synchronization flag;

determining if the EOT message includes the synchronization flag;

when the EOT message does not include the synchronization flag, sending an EOT acknowledgment over the communication link to the first node; and

when the EOT message includes the synchronization flag, sending the EOT acknowledgment with a synchronization flag over the communication link to the first node.

8. The system of claim 7 , wherein the second link interface is configured to detect EOT acknowledgments from the second node to the first node;

detecting the EOT acknowledgement with or without the synchronization flag;

when the EOT acknowledgement includes the synchronization flag, sending an interrupt to the processor complex.