Automated system-level failure and recovery
Systems and methods for automated system-level failure and recovery are described. In some embodiments, an Information Handling System (IHS) includes a processor and a memory, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to execute a selected process configured to participate in an inter-process communication (IPC) with at least one other process, invoke an error handling process by simulating a fault in the IPC, and determine if the error handling process successfully handles the fault.
1. An Information Handling System (IHS), comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to:
execute a client component configured to participate in an inter-process communication (IPC) with a service component as coordinated by Operating System (OS) component;
invoke a fault injection component configured to cause the service component to crash in the absence of fault; and
determine that an error handling process successfully handles the crash in response to: (i) a request being sent by the client component to the service component through the OS component, (ii) the service component crashing in response to a command issued by the fault injection component, (iii) after a first timeout, the request being sent a second time by the client component to the service component through the OS component causing the service component to restart, (iv) the OS component notifying the fault injection component that the service component has restarted, and (v) after a second timeout expiring, the request being sent a third time by the client component to the service component through the OS component causing the service component to respond.
2. The IHS of claim 1 , wherein the OS component includes an IPC broker.
3. The IHS of claim 1 , wherein program instructions, upon execution, further cause the IHS to determine that the error handling process successfully handles the crash in response to the service component continuing to run despite the crash.
4. The IHS of claim 1 , wherein the inter-process communication includes an external network test.
5. The IHS of claim 4 , wherein the external network test involves testing Hypertext Transfer Protocol (HTTP) responses.
6. The IHS of claim 5 , wherein program instructions, upon execution, further cause the IHS to determine that the error handling process successfully handles the crash in response to the service component failing with a known error code.
7. The IHS of claim 6 , wherein program instructions, upon execution, further cause the IHS to determine that the error handling process successfully handles the crash in response to the service component recovering within a specified time period.
8. The IHS of claim 1 , wherein program instructions stored thereon that, upon execution, further cause the IHS to:
receive configuration parameters; and
test inter-process dependencies using the configuration parameters.
9. The IHS of claim 8 , wherein the configuration parameters are selected from the group consisting of: an identification of an IPC broker, an identification of one or more other processes configured to communicate with the IPC broker, a list of one or more faults, and selection criteria.
10. The IHS of claim 9 , wherein the one or more faults further comprise: a crash, a hang, a malloc failure, a message drop, or an out-of-order message.
11. The IHS of claim 9 , wherein the selection criteria further comprise:
a first criterion indicating how often a particular fault is to be simulated; and
a second criterion indicating for how long the particular fault is to be simulated.
12. A memory device having program instructions stored thereon that, upon execution by a processor of an Information Handling System (IHS), cause the IHS to:
execute a broker process configured to participate in an inter-process communication (IPC) between a client component and a service component;
invoke a fault injection component configured to cause an Out-of-Order (OOO) error in the absence of fault; and
determine that an error handling process successfully handles the OOO error in response to: (i) the fault injection component issuing a capture command to the broker process, (ii) the broker process receiving a request issued by the client component to the service component, (iii) the broker process re-routing the request to the fault injection component, (iv) after a timeout expires, the client component sending a resend request command to the service component via the broker process, (v) the broker process receiving a response from the service component, (vi) the broker process re-routing the response to the fault injection component, (vii) the fault injection component issuing a replay request command to the broker process, (viii) the broker process sending the replay request command to the service component, (ix) the service component discarding the replayed request; (x) the fault injection component sending a replay response command to the broker process, (xi) the broker process replaying the response to the client component, and (xii) the client component discarding the replayed response.
13. The memory device of claim 12 , wherein the program instructions, upon execution, further cause the IHS to determine that the error handling process successfully handles the OOO error in response to the service process: (a) continuing to run despite the OOO error; or (b) failing with a known error code and recovering within a specified time period.
14. The memory device of claim 12 , wherein the program instructions, upon execution, further cause the IHS to:
receive configuration parameters; and
test inter-process dependencies using the configuration parameters.
15. The memory device of claim 14 , wherein the configuration parameters are selected from the group consisting of: an identification of an IPC broker, an identification of one or more other processes configured to communicate with the IPC broker, a list of one or more faults, and selection criteria.
16. The memory device of claim 15 , wherein the one or more faults further comprise: a crash, a hang, a malloc failure, a message drop, or an out-of-order message.
17. The memory device of claim 15 , wherein the selection criteria further comprise:
a first criterion indicating how often a particular fault is to be simulated; and
a second criterion indicating for how long the particular fault is to be simulated.
18. In an Information Handling System (IHS), a method comprising:
executing a broker process configured to coordinate an inter-process communication (IPC) between a client component and a service component;
invoking a fault injection component configured to cause the service component to crash in the absence of fault; and
determining that an error handling process successfully handles the crash in response to: (i) a request being sent by the client component to the service component through the OS component, (ii) the service component crashing in response to a command issued by the fault injection component, (iii) after a first timeout expiring, the request being sent a second time by the client component to the service component through the OS component causing the service component to restart, (v) the OS component notifying the fault injection component that the service component has restarted, (vi) after a second timeout expiring, the request being sent a third time by the client component to the service component through the OS component and causing the service component to respond.
19. The method of claim 18 , wherein the error handling process successfully handles the crash in response to the service component: (a) continuing to run despite the crash; or (b) failing with a known error code and recovering within a specified time period.
20. The method of claim 18 , further comprising:
receiving configuration parameters selected from the group consisting of: an identification of an IPC broker, an identification of one or more other processes configured to communicate with the IPC broker, a list of one or more faults, and selection criteria;
wherein the one or more faults include: a crash, a hang, a malloc failure, a message drop, or an out-of-order message; and
wherein the selection criteria include: (a) a first criterion indicating how often a particular fault is to be simulated and (b) a second criterion indicating for how long the particular fault is to be simulated.