Method and system for rollback-free failure recovery of multi-step procedures
View Patent ↗Disclosed is a rollback-free method for performing multi-step procedures in the presence of possible failure. As the procedure proceeds from its initial state through transitions to its final state, its constituent transitions are monitored for failure. If a failure is detected, then the procedure is “sidetracked” into a recovery coordination state. From the recovery coordination state, the procedure is logically taken back to its initial state and retried. In this manner, the procedure is shepherded, without rollbacks, through its transitions until it successfully reaches its final state. In a particular embodiment, a multi-step procedure is developed for moving a resource from one resource server to another. The steps of the procedure are designed so that all throughout the procedure, both of the resource servers and a directory server are kept synchronized. This allows client requests to proceed without interruption even during the resource movement.
1. In a computing system, a method for performing a procedure in the presence of possible failure, the procedure specifying a plurality of states including an initial state and a final state, the procedure further specifying a plurality of transitions between the states, the procedure further specifying an order of the transitions and of the states, the method comprising:
monitoring for failures;
initializing the procedure at the initial state; and
performing the procedure by executing transitions from one state to another state according to the specified order until either the final state is reached or until a failure is detected in a transition;
wherein if a failure is detected in a transition, then:
transitioning to a recovery coordination state;
transitioning from the recovery coordination state to the initial state;
re-visiting transitions from one state to another state according to the specified order until either the failed transition is reached or until a further failure is detected in a transition;
if the failed transition is reached without further failure, then executing transitions from one state to another state according to the specified order from the failed transition until either the final state is reached or until a further failure is detected in a transition wherein the computing system comprises a plurality of computing devices;
wherein at least one transition specified by the procedure is executed on a first computing device;
wherein at least one other transition specified by the procedure is executed on a second computing device;
wherein the computing system comprises a third computing device;
wherein the third computing device comprises a target resource server flag and a current resource server flag; and
wherein at least one transition specified by the procedure atomically changes values of both the target resource server flag and the current resource server flag.
2. The method of claim 1 wherein the first computing device is a resource server;
wherein the second computing device is a resource server; and
wherein the third computing device is a directory server.
3. In a computing system comprising a directory server, a first resource server, and a second resource server, the directory server comprising a current resource server flag and a target resource server flag, the first and second resource servers each comprising a moveaway flag, a method for moving a resource from the first resource server to the second resource server, the method comprising the following elements:
setting the directory server's current resource server flag to indicate the first resource server;
setting the directory server's target resource server flag to not indicate any resource server;
setting the first resource server's moveaway flag to FALSE;
setting the directory server's target resource server flag to indicate the second resource server;
setting the first resource server's moveaway flag to TRUE;
copying the resource from the first resource server to the second resource server;
setting the second resource server's moveaway flag to FALSE;
setting the directory server's current resource server flag to indicate the second resource server; and
setting the directory server's target resource server flag to not indicate any resource server.
4. The method of claim 3 wherein the directory server and the first resource server are the same server.
5. The method of claim 3 wherein the elements of the method are idempotent operations.
6. The method of claim 3 further comprising:
deleting the resource from the first resource server.
7. The method of claim 3 further comprising:
monitoring for failures;
wherein if a failure is detected in an element of the method, then:
coordinating a failure recovery;
re-visiting elements of the procedure until either the failed element is reached or until a further failure is detected; and
if the failed element is reached without further failure, then performing elements from the failed element until either the final element is successfully performed or until a further failure is detected.
8. The method of claim 7 wherein coordinating a failure recovery comprises generating an error message.
9. The method of claim 7 wherein, for at least one element before the failed element, re-visiting the element comprises skipping the element.
10. The method of claim 7 wherein, for at least one element before the failed element, re-visiting the element comprises re-executing the element.
11. The method of claim 10 wherein re-executing the element comprises receiving an error message that the element has already been performed.
12. A computer-readable storage medium containing computer-executable instructions for performing a method for moving a resource from a first resource server to a second resource server, the first and second resource servers each comprising a moveaway flag, the method comprising the following elements:
setting the directory server's current resource server flag to indicate the first resource server;
setting the directory server's target resource server flag to not indicate any resource server;
setting the first resource server's moveaway flag to FALSE;
setting the directory server's target resource server flag to indicate the second resource server;
setting the first resource server's moveaway flag to TRUE;
copying the resource from the first resource server to the second resource server;
setting the second resource servers moveaway flag to FALSE;
setting the directory server's current resource server flag to indicate the second resource server; and
setting the directory server's target resource server flag to not indicate any resource server.
13. In a computing system comprising a directory server computing device, a first resource server computing device, and a second resource server computing device, the directory server comprising a current resource server flag and a target resource server flag, the first and second resource servers each comprising a moveaway flag, a method for the directory server to move a resource from the first resource server to the second resource server, the method comprising the following elements:
setting the directory server's current resource server flag to indicate the first resource server;
setting the directory server's target resource server flag to not indicate any resource server;
setting the directory server's target resource server flag to indicate the second resource server;
requesting that the first resource server set its moveaway flag to TRUE;
copying the resource from the first resource server to the second resource server;
requesting that the second resource server set its moveaway flag to FALSE;
setting the directory server's current resource server flag to indicate the second resource server; and
setting the directory server's target resource server flag to not indicate any resource server.
14. The method of claim 13 wherein the directory server and the first resource server are the same server.
15. The method of claim 13 wherein the elements of the method are idempotent operations.
16. The method of claim 13 wherein copying the resource comprises requesting that the first resource server send a copy of the resource to the second resource server.
17. The method of claim 13 further comprising:
requesting that the first resource server delete the resource.
18. The method of claim 13 further comprising:
monitoring for failures;
wherein if a failure is detected in an element of the method, then:
coordinating a failure recovery;
re-visiting elements of the procedure until either the failed element is reached or until a further failure is detected; and
if the failed element is reached without further failure, then performing elements from the failed element until either the final element is successfully performed or until a further failure is detected.
19. The method of claim 18 wherein coordinating a failure recovery comprises generating an error message.
20. The method of claim 18 wherein, for at least one element before the failed element, re-visiting the element comprises skipping the element.
21. The method of claim 18 wherein, for at least one element before the failed element, re-visiting the element comprises re-executing the element.
22. The method of claim 21 wherein re-executing the element comprises receiving an error message that the element has already been performed.
23. A computer-readable storage medium containing computer-executable instructions for performing a method for a directory server to move a resource from a first resource server to a second resource server, the directory server comprising a current resource server flag and a target resource server flag, the first and second resource servers each comprising a moveaway flag, the method comprising the following elements:
setting the directory server's current resource server flag to indicate the first resource server;
setting the directory server's target resource server flag to not indicate any resource server;
setting the directory server's target resource server flag to indicate the second resource server;
requesting that the first resource server set its moveaway flag to TRUE;
copying the resource from the first resource server to the second resource server;
requesting that the second resource server set its moveaway flag to FALSE;
setting the directory server's current resource server flag to indicate the second resource server; and
setting the directory server's target resource server flag to not indicate any resource server.