Transparent checkpointing and process migration in a distributed system
A distributed system for creating a checkpoint for a plurality of processes running on the distributed system. The distributed system includes a plurality of compute nodes with an operating system executing on each compute node. A checkpoint library resides at the user level on each of the compute nodes, and the checkpoint library is transparent to the operating system residing on the same compute node and to the other compute nodes. Each checkpoint library uses a windowed messaging logging protocol for checkpointing of the distributed system. Processes participating in a distributed computation on the distributed system may be migrated from one compute node to another compute node in the distributed system by re-mapping of hardware addresses using the checkpoint library.
1. A node, comprising:
a processor configured to:
record a first set of memory locations to a buffer during a first checkpoint interval;
store contents of the first set of memory locations in a certain order, while recording a second set of memory locations to the buffer during a second checkpoint interval; and
store contents of a memory location from the first set of memory locations earlier than it would otherwise be stored when the processor needs to modify the memory location while the processor creates a checkpoint.
2. The node of claim 1 , wherein the processor is configured to:
write protect the first set of memory locations before the second set of memory locations is modified.
3. The node of claim 1 , wherein the processor is configured to:
suspend a process between the first checkpoint interval and the second checkpoint interval and to write protect the first set of memory locations while the process is suspended.
4. The node of claim 3 , wherein the processor is configured to:
execute a barrier operation following a completion of the write protect of the first set of memory locations and to end the barrier operation before the process is resumed during the second checkpoint interval.
5. The node of claim 1 , wherein the processor is configured to:
create the checkpoint when the contents of the first set of memory locations are stored; and
remove a write protection for a memory location from the first set of memory locations when the processor needs to modify the memory location upon the contents of the memory location being stored.
6. The node of claim 5 , wherein the processor is configured to:
remove a record of the memory location and the write protection for the memory location after the contents of the memory location have been stored.
7. The node of claim 1 , wherein the processor is configured to:
create the checkpoint when the contents of the first set of memory locations are stored to a checkpoint file.
8. The node of claim 1 , wherein the processor is configured to:
remove a record of a memory location from the first set of memory locations after the contents from the memory location is stored in checkpoint file.
9. The node of claim 1 , wherein the processor is configured to:
create the checkpoint when the contents of the first set of memory locations are stored in non-volatile storage.
10. A node, comprising:
a processor configured to:
store contents of memory locations modified by the processor during a checkpoint interval in a certain order; and
store contents of a memory location in the memory locations out-of-order when the processor needs to modify the memory location while the processor creates a checkpoint.
11. The node of claim 10 , wherein the processor is configured to:
store the contents of the memory location earlier than it would otherwise be stored when the processor needs to modify the memory location while the processor is creating the checkpoint.
12. A method, comprising:
recording, by a hardware-implemented node comprising a processor, a first set of memory locations to a buffer during a first checkpoint interval;
storing, by a hardware-implemented node, contents of the first set of memory locations in a certain order, while recording a second set of memory locations to the buffer during a second checkpoint interval; and
storing, by a hardware-implemented node, contents of a memory location from the first set of memory locations earlier than it would otherwise be stored when the processor needs to modify the memory location while the processor creates a checkpoint.
13. The method of claim 12 , further comprising:
write protecting the first set of memory locations before the second set of memory locations is modified.
14. The method of claim 12 , further comprising:
suspending a process between the first checkpoint interval and the second checkpoint interval; and
write protecting the first set of memory locations while the process is suspended.
15. The method of claim 14 , further comprising:
executing a barrier operation following a completion of the write protect of the first set of memory locations; and
ending the barrier operation before the process is resumed during the second checkpoint interval.
16. The method of claim 12 , further comprising:
creating the checkpoint when the contents of the first set of memory locations are stored; and
removing a write protection for a memory location from the first set of memory locations when modifying the memory location upon storing the contents of the memory location.
17. The method of claim 16 , further comprising:
removing a record of the memory location and the write protection for the memory location after the contents of the memory location have been stored.
18. The method of claim 12 , further comprising:
creating the checkpoint when the contents of the first set of memory locations are stored to a checkpoint file.
19. The method of claim 12 , further comprising:
removing a record of a memory location from the first set of memory locations after the contents from the memory location is stored in checkpoint file.
20. The method of claim 12 , further comprising:
creating the checkpoint when the contents of the first set of memory locations are stored in non-volatile storage.