Memory coherency in application-level virtualization
A coherence protocol applied to memory pages maintains coherence between memory spaces on a plurality of nodes so that the threads of the runtime are operable on any of the nodes. The nodes operating according to the coherence protocol track a state and an epoch number for each memory page residing therein. The states include a modified state in which only one particular node has an up-to-date copy of the memory page, an exclusive state in which only one particular node owns the memory page, a shared state in which all nodes that have the memory page in the shared state have the same copy, and a lost state in which the memory page cannot be either read or written. The epoch number is a number that is incremented each time the page enters the modified state and is used to determine whether the page contains data that is stale.
1. A method of maintaining coherency among memory residing in each node of a set of nodes that are interconnected by a network and include a first node and a second node, the method comprising:
tracking a state and an epoch number of each page of the memory residing in each of the first node and the second node, wherein the state is one of a modified state, an exclusive state, a shared state, and a lost state in which the page cannot be either read or written, and an epoch number is a number that is incremented each time the page enters the modified state;
in connection with performing a write operation on a first page residing in the first node while the first page is in the shared state, sending a request to the second node that the first node needs ownership of the first page, setting the state of the first page residing in the first node to the modified state, and incrementing the epoch number of the first page residing in the first node; and
in connection with performing a read operation on a second page residing in the first node while the second page is in the lost state, sending a request to the second node that the first node needs to access the second page, and setting the state of the second page residing in the first node to the exclusive state.
2. The method of claim 1 , further comprising:
while the first page residing in the first node is in the modified state, in response to receiving a request for access to the first page from the second node, setting the state of the first page residing in the first node to the lost state, and sending a positive acknowledgment to the second node; and
while the second page residing in the first node is in the exclusive state, in response to receiving a request to access the second page from the second node, setting the state of the second page residing in the first node to the lost state, and sending a positive acknowledgment to the second node.
3. The method of claim 1 , further comprising:
while the first page residing in the first node is in the modified state, sending page data of the first page residing in the first node to the second node and then transitioning the state of the first page residing in the first node from the modified state to the exclusive state.
4. The method of claim 3 , further comprising:
determining that the number of pages residing in the first node that are in the modified state exceeds a maximum, wherein
the page data of the first page residing in the first node is sent to the second node and the state of the first page residing in the first node is transitioned from the modified state to the exclusive state responsive to determining that the number of pages residing in the first node that are in the modified state exceeds the maximum.
5. The method of claim 4 , wherein
the second node also receives an epoch number with the page data of the first page from the first node, and
the second node updates the first page residing in the second node with the page data of the first page received from the first node, and updates the epoch number of the second page residing in the second node with the epoch number received from the first node.
6. The method of claim 1 , further comprising:
while a third page residing in the first node is in the shared state, in response to receiving a request for ownership of the third page, setting the state of the third page residing in the first node to the lost state.
7. The method of claim 1 , further comprising:
while a third page residing in the first node is in the modified state or the exclusive state, in response to receiving a request for access to the third page, setting the state of the third page residing in the first node to the lost state, and sending a positive acknowledgment to the second node, wherein the positive acknowledgement includes page data of the third page residing in the first node.
8. The method of claim 1 , further comprising:
while the second page residing in the first node is in the exclusive state, in response to receiving the request to access the second page from the second node, setting the state of the second page residing in the first node to the lost state and incrementing a count value for the second page, and sending a positive acknowledgment to the second node,
wherein the positive acknowledgement includes the count value for the second page, and the second page transitions to the shared state when a node owning the second page receives a request to access the second page when the count value has reached a threshold.
9. A computing system including a set of nodes that are interconnected by a network and include a first node and a second node, wherein each of the nodes includes a processor that is programmed to carry out a method of maintaining coherency among memory residing in each node, said method comprising:
tracking a state and an epoch number of each page of the memory residing in each of the first node and the second node, wherein the state is one of a modified state, an exclusive state, a shared state, and a lost state in which the page cannot be either read or written, and an epoch number is a number that is incremented each time the page enters the modified state;
in connection with performing a write operation on a first page residing in the first node while the first page is in the shared state, sending a request to the second node that the first node needs ownership of the first page, setting the state of the first page residing in the first node to the modified state, and incrementing the epoch number of the first page residing in the first node; and
in connection with performing a read operation on a second page residing in the first node while the second page is in the lost state, sending a request to the second node that the first node needs to access the second page, and setting the state of the second page residing in the first node to the exclusive state.
10. The computing system of claim 9 , wherein the method further comprises:
while the first page residing in the first node is in the modified state, in response to receiving a request for access to the first page from the second node, setting the state of the first page residing in the first node to the lost state, and sending a positive acknowledgment to the second node; and
while the second page residing in the first node is in the exclusive state, in response to receiving a request to access the second page from the second node, setting the state of the second page residing in the first node to the lost state, and sending a positive acknowledgment to the second node.
11. The computing system of claim 9 , wherein the method further comprises:
while the first page residing in the first node is in the modified state, sending page data of the first page residing in the first node to the second node and then transitioning the state of the first page residing in the first node from the modified state to the exclusive state.
12. The computing system of claim 11 , wherein the method further comprises:
determining that the number of pages residing in the first node that are in the modified state exceeds a maximum, wherein
the page data of the first page residing in the first node is sent to the second node and the state of the first page residing in the first node is transitioned from the modified state to the exclusive state responsive to determining that the number of pages residing in the first node that are in the modified state exceeds the maximum.
13. The computing system of claim 12 , wherein
the second node also receives an epoch number with the page data of the first page from the first node, and
the second node updates the first page residing in the second node with the page data of the first page received from the first node, and updates the epoch number of the second page residing in the second node with the epoch number received from the first node.
14. The computing system of claim 9 , wherein the method further comprises:
while a third page residing in the first node is in the shared state, in response to receiving a request for ownership of the third page, setting the state of the third page residing in the first node to the lost state.
15. The computing system of claim 9 , wherein the method further comprises:
while a third page residing in the first node is in the modified state or the exclusive state, in response to receiving a request for access to the third page, setting the state of the third page residing in the first node to the lost state, and sending a positive acknowledgment to the second node, wherein the positive acknowledgement includes page data of the third page residing in the first node.
16. The computing system of claim 9 , wherein the method further comprises:
while the second page residing in the first node is in the exclusive state, in response to receiving the request to access the second page from the second node, setting the state of the second page residing in the first node to the lost state and incrementing a count value for the second page, and sending a positive acknowledgment to the second node,
wherein the positive acknowledgement includes the count value for the second page, and the second page transitions to the shared state when a node owning the second page receives a request to access the second page when the count value has reached a threshold.
17. A non-transitory computer readable medium comprising instructions to be executed in a processor of a computer system to carry out a method of maintaining coherency among memory residing in each node of a set of nodes that are interconnected by a network and include a first node and a second node, said method comprising:
tracking a state and an epoch number of each page of the memory residing in each of the first node and the second node, wherein the state is one of a modified state, an exclusive state, a shared state, and a lost state in which the page cannot be either read or written, and an epoch number is a number that is incremented each time the page enters the modified state;
in connection with performing a write operation on a first page residing in the first node while the first page is in the shared state, sending a request to the second node that the first node needs ownership of the first page, setting the state of the first page residing in the first node to the modified state, and incrementing the epoch number of the first page residing in the first node; and
in connection with performing a read operation on a second page residing in the first node while the second page is in the lost state, sending a request to the second node that the first node needs to access the second page, and setting the state of the second page residing in the first node to the exclusive state.
18. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:
while the first page residing in the first node is in the modified state, in response to receiving a request for access to the first page from the second node, setting the state of the first page residing in the first node to the lost state, and sending a positive acknowledgment to the second node; and
while the second page residing in the first node is in the exclusive state, in response to receiving a request to access the second page from the second node setting the state of the second page residing in the first node to the lost state, and sending a positive acknowledgment to the second node.
19. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:
while a third page residing in the first node is in the shared state, in response to receiving a request for ownership of the third page, setting the state of the third page residing in the first node to the lost state.
20. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:
while a third page residing in the first node is in the modified state or the exclusive state, in response to receiving a request for access to the third page, setting the state of the third page residing in the first node to the lost state, and sending a positive acknowledgment to the second node, wherein the positive acknowledgement includes page data of the third page residing in the first node.