Systems and methods for checkpointing in a fault tolerant system
A method and system of checkpointing in a computing system having a primary node and a secondary node is disclosed. In one embodiment the method includes the steps of determining by the primary node to initiate a checkpoint process; sending a notification to the secondary node, by the primary node, of an impending checkpoint process; blocking, by the primary node, I/O requests from the Operating System (OS) that arrive at the primary node after the determination to initiate the checkpoint process; completing, by the primary node, active I/O requests for data received from the OS prior to the determination to initiate the checkpoint process, by accessing the primary node data storage; and upon receiving, by the primary node, a notice of checkpoint readiness from the secondary node, initiating a checkpoint process to move state and data from the primary node to the secondary node.
1. A method of checkpointing in a computing system having a primary node and a secondary node wherein each of the primary node and the secondary node includes an operating system (OS), a respective cache memory and a respective storage system wherein each storage system includes a storage driver, the method comprising the steps of:
determining, by the primary node, when to initiate a checkpoint process;
sending a notification to the secondary node, by the primary node, of an impending checkpoint process;
blocking, by the storage driver in the primary node, I/O requests from the OS on the primary node after the determination to initiate the checkpoint process;
completing, by the primary node, active I/O requests for data, received from the OS of the primary node prior to the determination to initiate the checkpoint process, by accessing the primary node data storage system; and
upon receiving, by the primary node, a notice of checkpoint readiness from the secondary node, initiating the checkpoint process from the primary node to the secondary node.
2. The method of claim 1 , further comprising:
receiving, at the secondary node, the notification of the impending checkpoint process;
blocking, by the secondary node, new I/O requests from the OS of the secondary node that arrive at the secondary node after receipt, by the secondary node, of a notification of the determination to initiate the checkpoint process;
completing active I/O requests for data at the secondary node; and
initializing a checkpoint process on the secondary node.
3. The method of claim 1 , wherein the checkpoint process comprises the steps of:
determining what data from the primary node to include in checkpoint data; and
copying the checkpoint data from the primary node to the secondary node.
4. The method of claim 3 , wherein data in the cache memory of the secondary node is overwritten when the checkpoint process is complete.
5. The method of claim 3 , wherein both the primary node and the secondary node apply data from the checkpoint to each respective storage system in the background after the checkpoint process is complete.
6. The method of claim 1 wherein a request from a client to write data to the primary node writes data to the respective cache memory of the primary node and wherein the primary node sends the request to the secondary node which writes the data to the respective cache memory of the secondary node.
7. The method of claim 1 wherein a read request by the respective OS of the primary node and the secondary node for data from the primary node and the secondary node reads data from the respective cache memory of the primary node and the secondary node if data requested resides in the respective cache memory of the primary node and the secondary node, and reads from the respective storage system of the primary node and the secondary node if data requested does not reside in the respective cache memory of the primary node and the secondary node.
8. The method of claim 1 , wherein during the checkpoint process, data from the cache memory of the primary node is written to the cache memory of the secondary node.
9. The method of claim 8 , wherein one portion of the cache memory of the primary and secondary node is reserved for data modified during execution since a previous checkpoint process, and wherein a second portion of the cache memory of the primary and secondary node is reserved for data modified prior to the previous checkpoint process.
10. A computing system comprising:
a primary node and a secondary node wherein each of the primary node and the secondary node includes an operating system (OS), a respective cache memory and a respective storage system wherein each storage system includes a storage driver; and
computer-executable logic, encoded in memory of the computing system, for executing checkpointing in the computing system, wherein the computer-executable program logic is configured for the execution of:
determining, by the primary node, when to initiate a checkpoint process;
sending a notification to the secondary node, by the primary node, of an impending checkpoint process;
blocking, by the storage driver in the primary node, I/O requests received from the OS on the primary node after the determination to initiate the checkpoint process;
completing, by the primary node, active I/O requests for data, received from the OS of the primary node prior to the determination to initiate the checkpoint process, by accessing the primary node data storage system; and
upon receiving, by the primary node, a notice of checkpoint readiness from the secondary node, initiating the checkpoint process from the primary node to the secondary node.
11. The system of claim 10 , wherein the computer-executable program logic is further configured for the execution of:
receiving, at the secondary node, the notification of the impending checkpoint process;
blocking, by the secondary node, new I/O requests from the OS of the secondary node, that arrive at the secondary node after receipt, by the secondary node, of a notification of the determination to initiate the checkpoint process;
completing active I/O requests for data at the secondary node; and
initializing a checkpoint process on the secondary node.
12. The system of claim 10 , wherein the checkpoint process comprises the steps of:
determining what data from the primary node to include in checkpoint data; and
copying the checkpoint data from the primary node to the secondary node.
13. The system of claim 12 , wherein data in the cache memory of the secondary node is overwritten when the checkpoint process is complete.
14. The system of claim 12 , wherein both the primary node and the secondary node apply data from the checkpoint to each respective storage system in the background after the checkpoint process is complete.
15. The system of claim 10 , wherein a request from a client to write data to the primary node writes data to the respective cache memory of the primary node and wherein the primary node sends the request to the secondary node which writes the data to the respective cache memory of the secondary node.
16. The system of claim 10 , wherein a read request by the respective OS of the primary node and the secondary node for data from the primary node and the secondary node reads data from the respective cache memory of the primary node and the secondary node if data requested resides in the respective cache memory of the primary node and the secondary node, and reads from the respective storage system of the primary node and the secondary node if data requested does not reside in the respective cache memory of the primary node and the secondary node.
17. The system of claim 10 , wherein during the checkpointing process, data from the cache memory of the primary node is written to the cache memory of the secondary node.
18. The system of claim 17 , wherein one portion of the cache memory of the primary and secondary node is reserved for data modified during execution since a previous checkpoint process, and wherein a second portion of the cache memory of the primary and secondary node is reserved for data modified prior to the previous checkpoint process.
19. The system of claim 10 wherein the respective storage system comprises a non-volatile memory express (NVMe) data storage device or a bus.
20. A computer program product for checkpointing in a computing system having a primary node and a secondary node wherein each of the primary node and the secondary node includes an operating system (OS), a respective cache memory and a respective storage system wherein each storage system includes a storage driver, the computer program product comprising:
a non-transitory computer readable medium encoded with computer executable program code for executing checkpointing in the computing system, the code configured to enable the execution of:
determining, by the primary node, when to initiate a checkpoint process;
sending a notification to the secondary node, by the primary node, of an impending checkpoint process;
blocking, by the storage driver in the primary node, I/O requests received from the OS on the primary node after the determination to initiate the checkpoint process;
completing, by the primary node, active I/O requests from the OS of the primary node prior to the determination to initiate the checkpoint process, by accessing the primary node data storage system; and
upon receiving, by the primary node, a notice of checkpoint readiness from the secondary node, initiating the checkpoint process from the primary node to the secondary node.
21. The computer program product of claim 20 , wherein the code is further configured to enable the execution of:
receiving, at the secondary node, the notification of the impending checkpoint process;
blocking, by the secondary node, new I/O requests from the OS of the secondary node, that arrive at the secondary node after receipt, by the secondary node, of a notification of the determination to initiate the checkpoint process;
completing active I/O requests for data on the secondary node; and
initializing a checkpoint process on the secondary node.
22. The computer program product of claim 20 , wherein the checkpoint process comprises the steps of:
determining what data from the primary node to include in checkpoint data; and copying the checkpoint data from the primary node to the secondary node.
23. The computer program product of claim 22 , wherein data in the cache memory of the secondary node is overwritten when the checkpoint process is complete.
24. The computer program product of claim 22 , wherein both the primary node and the secondary node apply data from the checkpoint to each respective storage system in the background after the checkpoint process is complete.
25. The computer program product of claim 20 , wherein a request from a client to write data to the primary node writes data to the respective cache memory of the primary node and wherein the primary node sends the request to the secondary node which writes the data to the respective cache memory of the secondary node.
26. The computer program product of claim 20 , wherein a read request by the respective OS of the primary node and the secondary node for data from the primary node and the secondary node reads data from the respective cache memory of the primary node and the secondary node if data requested resides in the respective cache memory of the primary node and the secondary node, and reads from the respective storage system of the primary node and the secondary node if data requested does not reside in the respective cache memory of the primary node and the secondary node.
27. The computer program product of claim 20 , wherein during the checkpoint process, data from the cache memory of the primary node is written to the cache memory of the secondary node.
28. The computer program product of claim 27 , wherein one portion of the cache memory of the primary and secondary node is reserved for data modified during execution since a previous checkpoint process, and wherein a second portion of the cache memory of the primary and secondary node is reserved for data modified prior to the previous checkpoint process.