Methods and apparatuses for improving failure recovery in a distributed system
Embodiments are disclosed for improving failure recovery in a distributed system during execution of a processing chain. In the context of a method, an example embodiment includes implementing, by the distributed system, a task coordination state machine corresponding to the processing chain. This example embodiment of the method further includes discovering, using the task coordination state machine, an error occurring during execution of the processing chain. Finally, the example embodiment of the method further includes reporting the error by a failure report generator service and recovering, by a failure recovery service, from the error. Corresponding apparatuses and computer program products are also provided.
1. A method for providing for failure recovery in a distributed system during execution of a processing chain, the method comprising:
implementing, by the distributed system, a task coordination state machine corresponding to the processing chain, the task coordination state machine storing a set of states in memory;
discovering, using the task coordination state machine, an error caused by an event occurring during execution of the processing chain;
reporting the error by a failure report generator service; and
recovering, by a failure recovery service, from the error by:
calling, by the failure recovery service, a resolution plan selector for selection of one or more resolvers to recover from the error;
retrieving, by the resolution plan selector, one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
executing, by a resolution plan executor, the one or more resolvers to rehydrate one or more data fields in memory; and
transitioning, by the distributed system, the task coordination state machine to a particular working state that was in effect prior to occurrence of the error, thereby rehydrating the event that caused the error.
2. The method of claim 1 , wherein the task coordination state machine comprises:
the set of states, wherein the set of states includes the error state and a state corresponding to each unit of work in the processing chain;
a set of state transitions, each of which illustrates a logical connections between two states in the set of states; and
a set of events defining changes in a context of the processing chain that correspond to each state transition.
3. The method of claim 1 , wherein discovering the error using the task coordination state machine includes:
in response to occurrence of the error,
transitioning the task coordination state machine from the particular working state into an error state, and
generating a notification based on the error.
4. The method of claim 3 , wherein the notification identifies a reason code, a time stamp, a type of the error, a severity of the error, and a criticality of the error.
5. The method of claim 1 , wherein reporting the error by the failure report generator service includes:
generating an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
storing the incident in an incident repository; and
notifying applications affected by the incident.
6. The method of claim 5 , wherein reporting the error by the failure report generator service further includes:
causing presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
awaiting instructions, from the user device, to initiate a failure recovery service to recover from the incident.
7. The method of claim 5 , wherein the incident further identifies a correlation identifier that can be used to stitch together events in the system, and wherein the incident context identifies at least one of a source unit of work, a target unit of work, and an initiating system or user that initiated processing that caused the error.
8. An apparatus for providing for failure recovery in a distributed system during execution of a processing chain, the apparatus comprising at least one processor and at least one memory storing computer-executable instructions, that, when executed by the at least one processor, cause the apparatus to:
implement a task coordination state machine corresponding to the processing chain, the task coordination state machine storing a set of states in memory;
discover, using the task coordination state machine, an error caused by an event occurring during execution of the processing chain;
report the error; and
recover from the error by:
retrieving one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
executing the one or more resolvers to rehydrate one or more data fields in memory; and
transitioning the task coordination state machine to a particular working state that was in effect prior to occurrence of the error, thereby rehydrating the event that caused the error.
9. The apparatus of claim 8 , wherein the task coordination state machine comprises:
the set of states, wherein the set of states includes the error state and a state corresponding to each unit of work in the processing chain;
a set of state transitions, each of which illustrates a logical connections between two states in the set of states; and
a set of events defining changes in a context of the processing chain that correspond to each state transition.
10. The apparatus of claim 8 , wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to discover the error using the task coordination state machine by causing the apparatus to:
in response to occurrence of the error,
transition the task coordination state machine from the particular working state into an error state, and
generate a notification based on the error.
11. The apparatus of claim 10 , wherein the notification identifies a reason code, a time stamp, a type of the error, a severity of the error, and a criticality of the error.
12. The apparatus of claim 8 , wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to report the error by causing the apparatus to:
generate an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
store the incident in an incident repository; and
notify applications affected by the incident.
13. The apparatus of claim 12 , wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to report the error by causing the apparatus to:
cause presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
await instructions, from the user device, to initiate a failure recovery service to recover from the incident.
14. The apparatus of claim 12 , wherein the incident further identifies a correlation identifier that can be used to stitch together events in the system, and wherein the incident context identifies at least one of a source unit of work, a target unit of work, and an initiating system or user that initiated processing that caused the error.
15. A computer program product comprising at least one non-transitory computer-readable storage medium for providing for failure recovery in a distributed system during execution of a processing chain, the at least one non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause an apparatus to:
implement a task coordination state machine corresponding to the processing chain;
discover, using the task coordination state machine, an error caused by an event occurring during execution of the processing chain, the task coordination state machine storing a set of states in memory;
report the error; and
recover from the error by:
calling a resolution plan selector for selection of one or more resolvers to recover from the error;
retrieving, by the resolution plan selector, one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
executing, by a resolution plan executor, the one or more resolvers to rehydrate one or more data fields in memory; and
transitioning the task coordination state machine to a particular working state that was in effect prior to occurrence of the error, thereby rehydrating the event that caused the error.
16. The computer program product of claim 15 , wherein the computer-executable instructions, when executed, cause the apparatus to report the error by causing the apparatus to:
generate an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
store the incident in an incident repository; and
notify applications affected by the incident.
17. The computer program product of claim 16 , wherein the computer-executable instructions, when executed, cause the apparatus to report the error by causing the apparatus to:
cause presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
await instructions, from the user device, to initiate a failure recovery service to recover from the incident.
18. The computer program product of claim 15 , wherein the computer-executable instructions, when executed, cause the apparatus to recover from the error by causing the apparatus to:
retrieve one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
execute the one or more resolvers to rehydrate the event; and
transition the task coordination state machine to the particular working state that was in effect prior to occurrence of the error.
19. A method for providing for failure recovery in a distributed system during execution of a processing chain, the method comprising:
implementing, by the distributed system, a task coordination state machine corresponding to the processing chain, the task coordination state machine storing a set of states in memory;
discovering, using the task coordination state machine, an error caused by an event occurring during execution of the processing chain;
generating an incident based on a notification generated during the discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
storing the incident in an incident repository;
notifying applications affected by the incident;
reporting the error by a failure report generator service, wherein reporting the error comprises:
causing presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
awaiting instructions, from the user device, to initiate a failure recovery service to recover from the incident; and
recovering, by a failure recovery service, from the error by rehydrating one or more data fields in memory to transition the task coordination state machine to a particular working state that was in effect prior to occurrence of the error, thereby rehydrating the event that caused the error.
20. The method of claim 19 , wherein the incident further identifies a correlation identifier that can be used to stitch together events in the system, and wherein the incident context identifies at least one of a source unit of work, a target unit of work, or an initiating system or user that initiated processing that caused the error.