Supporting a weak ordering memory model for a virtual physical address space that spans multiple nodes
In one embodiment, a virtual NUMA system may be formed from multiple computer systems coupled to a network such as InfiniBand, Ethernet, etc. Each computer includes one or more software modules which present the resources of the computers as a virtual NUMA machine. The virtual machine is a non-uniform memory access (NUMA) machine comprising a plurality of nodes, each node having memory that is part of a distributed shared memory. Additionally, the virtual machine is coherent with a weakly ordered memory model. When executed in a current owner node of a first block in response to an ownership transfer request from a requesting node of the plurality of nodes for the first block, the software modules perform a synchronization operation if the first block has been modified in the current owner node.
1. A system comprising:
a plurality of computers coupled to a network, wherein each of the plurality of computers comprises one or more software modules which, when executed on the computer, present resources of the plurality of computers as a virtual machine;
wherein the virtual machine is a non-uniform memory access (NUMA) machine comprising a plurality of nodes, each node having memory that is part of a distributed shared memory, and wherein the virtual machine is coherent with a weakly ordered memory model; and
wherein the one or more software modules, are executable in a current owner node of a first block in response to an ownership transfer request from a requesting node of the plurality of nodes for the first block, wherein the ownership transfer request is a request to transfer ownership of the first block from the current owner node to the requesting node, wherein the one or more software modules, when executed:
determine that the first block has been modified in the current owner node; and
perform a synchronization operation in response to determining that the first block is modified in the current owner node, wherein the synchronization operation comprises determining each other modified block in the memory of the current owner node and wherein the synchronization operation further comprises writing the first block to a home node of the first block and writing each other modified block in the current owner node to the home node of that modified block in response to detecting that the first block is modified in the current owner node and wherein the first block and each modified block is stored in a separate memory location in the memory of the respective home node of that block.
2. The system as recited in claim 1 wherein the synchronization operation further comprises transmitting an invalidate request to at least one other node of the plurality of nodes that has a shared copy of a modified block.
3. The system as recited in claim 2 wherein the synchronization operation comprises transmitting invalidate requests to each other node.
4. The system as recited in claim 2 wherein the one or more software modules do not check a read share flag corresponding to the modified block prior to transmitting the invalidate requests for the modified block, wherein the read share flag indicates whether or not at least one shared copy of the modified block was provided to another node.
5. The system as recited in claim 2 wherein the other node is configured to return an acknowledgement of the invalidate request, and wherein the one or more software modules, when executed, complete the synchronization operation without waiting for the acknowledgements.
6. The system as recited in claim 1 wherein the block is a page.
7. A computer accessible storage medium storing a plurality of instructions which, when executed on a plurality of computers coupled to a network, present resources of the plurality of computers as a virtual machine, wherein the virtual machine is a non-uniform memory access (NUMA) machine comprising a plurality of nodes, each node having memory that is part of a distributed shared memory, and wherein the virtual machine is coherent with a weakly ordered memory model, and wherein the plurality of instructions, when executed on a first computer of the plurality of computers and the first computer includes a current owner node of a first block, wherein the plurality of instructions are executed in response to an ownership transfer request for the first block received from a requesting node of the plurality of nodes, wherein the ownership transfer request is a request to transfer ownership of the first block from the current owner node to the requesting node, and wherein the plurality of instructions, when executed:
detect that the first block has been modified in the current owner node; and
perform a synchronization operation responsive to detecting that the first block has been modified in the current owner node, wherein the synchronization operation comprises determining each other modified block in the memory of the current owner node and wherein the synchronization operation further comprises writing the first block to a home node of the first block and writing each other modified block in the current owner node to the home node of that modified block in response to detecting that the first block is modified in the current owner node, and wherein the first block and each modified block is stored in a separate memory location in the memory of the respective home node of that block.
8. The computer accessible storage medium as recited in claim 7 wherein the synchronization operation further comprises transmitting an invalidate request to at least one other node of the plurality of nodes that has a shared copy of a modified block.
9. The computer accessible storage medium as recited in claim 8 wherein the synchronization operation comprises transmitting invalidate requests to each other node.
10. The computer accessible storage medium as recited in claim 8 wherein the plurality of instructions, when executed to perform the synchronization operation, do not check a read share flag corresponding to the modified block prior to transmitting the invalidate requests for the modified block, wherein the read share flag indicates whether or not at least one shared copy of the modified block was provided to another node.
11. The computer accessible storage medium as recited in claim 8 wherein the other node is configured to return an acknowledgement of the invalidate request, and wherein plurality of instructions, when executed to perform the synchronization operation, complete the synchronization operation without waiting for the acknowledgements.
12. The computer accessible storage medium as recited in claim 7 wherein the block is a page.
13. In a system comprising a plurality of computers coupled to a network, a method comprising:
presenting resources of the plurality of computers as a virtual machine, wherein the virtual machine is a non-uniform memory access (NUMA) machine comprising a plurality of nodes, each node having memory that is part of a distributed shared memory, wherein the virtual machine is coherent with a weakly ordered memory model;
receiving an ownership transfer request for a first block in a current owner node of the first block from a requesting node of the first block, wherein the ownership transfer request is a request to transfer ownership of the first block from the current owner node to the requesting node;
determining that the first block is modified in the current owner node; and
responsive to the ownership transfer request and determining that the first block is modified in the current owner node, performing a synchronization operation in the current owner node, wherein performing the synchronization operation comprises determining each other modified block in the memory of the current owner node, and wherein performing the synchronization operation further comprises writing the first block to a home node of the first block and writing each other modified block in the current owner node to the home node of that modified block in response to detecting that the first block is modified in the current owner node, and wherein the first block and each modified block is stored in a separate memory location in the memory of the respective home node of that block.
14. The method as recited in claim 13 wherein performing the synchronization operation further comprises transmitting an invalidate request to at least one other node of the plurality of nodes that has a shared copy of a modified block.
15. The method as recited in claim 14 wherein performing the synchronization operation further comprises transmitting invalidate requests to each other node.
16. The method as recited in claim 14 wherein performing the synchronization operation further comprises not checking a read share flag corresponding to the modified block prior to transmitting the invalidate requests for the modified block, wherein the read share flag indicates whether or not at least one shared copy of the modified block was provided to another node.
17. A computer comprising a memory, execution resources configured to execute instructions, during use, and hardware to couple to a network, wherein the execution resources, during use, execute one or more software modules stored in the memory, wherein the one or more software modules, when executed, configure the computer with other computers coupled to the network to form a virtual machine wherein the virtual machine is a non-uniform memory access (NUMA) machine comprising a plurality of nodes, each node having memory that is part of a distributed shared memory, and wherein the virtual machine is coherent with a weakly ordered memory model; and wherein the one or more software modules, when executed in a current owner node of a first block in response to an ownership transfer request from a requesting node of the plurality of nodes for the first block, wherein the ownership transfer request is a request to transfer ownership of the first block from the current owner node to the requesting node, and wherein the one or more software modules when executed:
detect that the first block has been modified in the current owner node; and
perform a synchronization operation in the current owner node responsive to detecting that the first block has been modified in the current owner node and responsive to the ownership transfer request, wherein performing the synchronization operation comprises determining each other modified block in the memory of the current owner node, and wherein performing the synchronization operation further comprises writing the first block to a home node of the first block and writing each other modified block in the current owner node to the home node of that modified block in response to detecting that the first block is modified in the current owner node, and wherein the first block and each modified block is stored in a separate memory location in the memory of the respective home node of that block.
18. The computer system as recited in claim 17 wherein the synchronization operation further comprises transmitting an invalidate request to at least one other node of the plurality of nodes that has a shared copy of a modified block.
19. The computer system as recited in claim 18 wherein the invalidate request comprises a list identifying the modified blocks that are to be invalidated in the at least one other node.
20. The computer system as recited in claim 18 wherein the synchronization operation comprises transmitting invalidate requests to each other node.
21. The computer system as recited in claim 18 wherein the one or more software modules do not check a read share flag corresponding to the modified block prior to transmitting the invalidate requests for the modified block, wherein the read share flag indicates whether or not at least one shared copy of the modified block was provided to another node.
22. The system as recited in claim 2 wherein the invalidate request comprises a list identifying the modified blocks that are to be invalidated in the at least one other node.
23. The computer accessible storage medium as recited in claim 8 wherein the invalidate request comprises a list identifying the modified blocks that are to be invalidated in the at least one other node.
24. The method as recited in claim 14 wherein the invalidate request comprises a list identifying the modified blocks that are to be invalidated in the at least one other node.