Persistent images of distributed shared memory segments and in-memory checkpoints
A method, system, computer system, and computer-readable medium that enable quick recovery from failure of one or more nodes, applications, and/or communication links in a distributed computing environment, such as a cluster. Recovery is facilitated by regularly saving persistent images of the in-memory checkpoint data and/or of distributed shared memory segments. The persistent checkpoint images are written asynchronously so that applications can continue to write data even during creation and/or updating the persistent image and with minimal effect on application performance. Furthermore, multiple updater nodes can simultaneously update the persistent checkpoint image using normal synchronization operations. When one or more nodes fail, the persistent checkpoint image can be read and used to restart the application in the most recently-saved state prior to the failure. The persistent checkpoint image can also be used to initialize the state of the application in a new node joining the distributed computing environment.
1. A method implemented in a system comprising an updater node and a shared persistent storage, wherein the updater node comprises a local committed checkpoint image, wherein the persistent storage comprises persistent first and second copies of a checkpoint image, first metadata designating the persistent first copy of the checkpoint image as a shadow checkpoint image, and second metadata designating the persistent second copy of the checkpoint image as a recovery checkpoint image, the method comprising:
updating the local committed checkpoint image;
locking the local committed checkpoint image after it is updated;
creating a copy of the locked local committed checkpoint image;
unlocking the local committed checkpoint image after the creating the copy;
updating the persistent first copy of the checkpoint image using the copy, wherein the checkpoint image comprises a prior state of an application,
updating the first metadata to designate the updated, first persistent copy of the checkpoint image as the recovery checkpoint image;
updating the persistent second copy of the checkpoint image using the copy after the updating the first metadata;
updating the second metadata to designate the updated, persistent second copy of the checkpoint image as the shadow checkpoint image;
wherein the updated, persistent first and second copies of the checkpoint image are identical to each other.
2. The method of claim 1 wherein the persistent first copy of the checkpoint image is updated in response to an event.
3. The method of claim 2 wherein
the event comprises one of
completion of a number of transactions,
a command to update a persistent copy of the checkpoint image, or
expiration of a timer.
4. The method of claim 1 further comprising:
designating a second node as a first updater node, wherein
the first updater node updates the persistent first copy of the checkpoint image.
5. The method of claim 4 further comprising:
in response to failure of the first updater node, designating another node to update the persistent first copy of the checkpoint image.
6. The method of claim 4 further comprising:
designating another node as a second updater node; and
allowing the first updater node and the second updater node to update the persistent first copy of the checkpoint image simultaneously.
7. The method of claim 6 wherein
the allowing the first and second updater nodes to update the persistent first copy of the checkpoint image simultaneously comprises
locking a first respective portion of the persistent first copy to be updated by the first updater node and
locking a second respective portion of the first persistent copy to be updated by the second updater node, wherein
the first and second respective portions of the persistent first copy do not overlap.
8. The method of claim 1 further comprising:
initializing an instance of the application on a third node, wherein the initializing comprises making a copy of the updated persistent first copy of the checkpoint image in a memory of the third node.
9. The method of claim 8 wherein the initializing is performed in response to a failure of a cluster.
10. A computer readable memory comprising executable instructions, wherein a method is implemented in a system in response to executing the instructions, the system comprising an updater node and a shared persistent storage, wherein the updater node comprises a local committed checkpoint image, wherein the persistent storage comprises persistent first and second copies of a checkpoint image, first metadata designating the persistent first copy of the checkpoint image as a shadow checkpoint image, and second metadata designating the persistent second copy of the checkpoint image as a recovery checkpoint image, the method comprising:
updating the local committed checkpoint image;
locking the local committed checkpoint image after it is updated;
creating a copy of the locked local committed checkpoint image;
unlocking the local committed checkpoint image after the creating the copy;
updating the persistent first copy of the checkpoint image using the copy, wherein the checkpoint image comprises a prior state of an application,
updating the first metadata to designate the updated, first persistent copy of the checkpoint image as the recovery checkpoint image;
updating the persistent second copy of the checkpoint image using the copy after the updating the first metadata;
updating the second metadata to designate the updated, persistent second copy of the checkpoint image as the shadow checkpoint image;
wherein the updated, persistent first and second copies of the checkpoint image are identical to each other.
11. The computer readable memory of claim 10 wherein the persistent first copy of the checkpoint image is updated in response to an event.
12. The computer readable memory of claim 11 wherein
the event comprises one of
completion of a number of transactions,
a command to update a persistent copy of the checkpoint image, or
expiration of a timer.
13. The computer readable memory of claim 10 wherein the method further comprises:
designating a second node as a first updater node, wherein
the first updater node updates the persistent first copy of the checkpoint image.
14. The computer readable memory of claim 13 wherein the method further comprises:
in response to failure of the first updater node, designating another node to update the persistent first copy of the checkpoint image.
15. The computer readable memory of claim 13 wherein the method further comprises:
designating another node as a second updater node; and
allowing the first updater node and the second updater node to update the persistent first copy of the checkpoint image simultaneously.
16. The computer readable memory of claim 15 wherein the allowing the first and second updater nodes to update the persistent first copy of the checkpoint image simultaneously comprises
locking a first respective portion of the persistent first copy to be updated by the first updater node and
locking a second respective portion of the first persistent copy to be updated by the second updater node, wherein
the first and second respective portions of the persistent first copy do not overlap.