Fault definition and injection process to simulate timing based errors in a distributed system
Embodiments for simulating timing-related error conditions in a distributed system, by allowing a user to define a fault map specifying one or more faults to be committed by components in the distributed system. These generated fault events are to be executed in different components of the distributed system in a serialized distributed order. An event injection process delivers the fault map messages to the nodes in the distributed system, and the nodes then execute an operation sequence containing the fault events in the proper order as coordinated by the event injection process. The faults are then committed by the associated components in the nodes. Execution of these fault events occurs before, after or during a regular component procedure or action to simulate the desired timing-related error.
1. A method of simulating timing-related error conditions in a distributed system, comprising:
allowing a user to define a fault map specifying one or more faults to be committed by components in the distributed system, wherein the generated fault events are to be executed in different components including a plurality of nodes in a serialized distributed order, and comprising one or more compute nodes and a controller node executing the event injection process, wherein each node executes a fault injection daemon process controlled by the controller node;
delivering, by an event injection process, fault map messages to each node of the plurality of nodes;
executing in each node, an operation sequence containing the fault events in the proper order as coordinated by the event injection process; and
committing, by associated components in one or more nodes of the plurality of nodes, the fault map faults to simulate a desired timing-related error, the components communicating using a peer-to-peer messaging service of a data and control network coupling the compute nodes and the controller node in the system, wherein the messaging service includes a Notify_And_Wait_Before_Send command to cause the system to recognize fault messages and perform orderly execution of the fault events.
2. The method of claim 1 wherein fault events are injected in between component actions to be executed in a specified time order by the associated components, and wherein execution of the fault events occurs before, after or during a regular component procedure or action for the operation sequence.
3. The method of claim 2 wherein the component actions comprise at least one of: reading data from a disk into memory, writing data stored in memory to the disk, performing a computation, sending a network message to a same or a different set of components, and wherein a fault condition to be simulated occurs if a component fails or takes too long to complete an action, or if an exact timing or sequencing is missed.
4. The method of claim 1 wherein the fault map comprises directives to execute fault events that are sent as a series of fault event messages.
5. The method of claim 4 wherein the fault event messages comprise information including: an identification of any nodes that participate in the fault event, the action that each of the any nodes is to perform, and an order in which the actions are performed.
6. The method of claim 1 further comprising receiving a clear fault command from a network client to clear the injected faults and restore the system to an original state.
7. A method of simulating timing-related error conditions in a distributed system, comprising:
defining a fault map to be distributed from a controller node of the network to a plurality of other nodes in the network, the fault map comprising a global sequence of fault event messages, wherein execution of the sequence simulates a specific error condition of the network;
distributing the fault map from the controller node to the other nodes through an event injection process communicating among the nodes using a peer-to-peer messaging system and communicating between a fault injection daemon process in the controller node and a respective daemon process running on each of the other nodes, wherein event injection process is used to inject errors in a distributed key value store by modifying some of the associated components of a distributed key value store including a messaging layer; and
executing in the other nodes, the global sequence of fault event messages in a proper order as coordinated by the event injection process to simulate the specific error condition.
8. The method of claim 7 wherein fault events of the global sequence are injected in between component actions to be executed in a specified time order by the components, and wherein execution of the fault events occurs before, after or during a regular component procedure or action for the operation sequence.
9. The method of claim 8 wherein the component actions comprise at least one of: reading data from a disk into memory, writing data stored in memory to the disk, performing a computation, sending a network message to a same or a different set of components, and wherein a fault condition to be simulated occurs if a component fails or takes too long to complete an action, or if an exact timing or sequencing is missed.
10. The method of claim 7 wherein the fault event messages information including: an identification of the nodes that participate in this fault event, the action that each of node is to perform, and an order in which the actions are performed.
11. The method of claim 7 further comprising receiving a clear fault command from a network client to clear the injected faults and restore the system to an original state.
12. A system for simulating timing-related error conditions in a distributed system, comprising:
a controller node of a network defining a fault map to be distributed to a plurality of other nodes in the network, the fault map comprising a sequence of fault event messages, wherein execution of the sequence simulates a specific error condition of the network;
an event injection component distributing the fault map from the controller node to each node of the plurality of other nodes through communicating among the nodes using a peer-to-peer messaging system, and communicating between a fault injection daemon process in the controller node and a respective daemon process running on each of the other nodes, wherein event injection process is used to inject errors in a distributed key value store by modifying some of the associated components of a distributed key value store including a messaging layer; and
respective node components executing in the other nodes, the sequence of fault event messages in a proper order as coordinated by the event injection process to simulate the specific error condition.
13. The system of claim 12 wherein the plurality of nodes communicate with each other using a peer-to-peer messaging service of a data and control network coupling the controller node and the plurality of other nodes in the system.
14. The system of claim 13 wherein the component actions comprise at least one of: reading data from a disk into memory, writing data stored in memory to the disk, performing a computation, sending a network message to a same or a different set of components, and wherein a fault condition to be simulated occurs if a component fails or takes too long to complete an action, or if an exact timing or sequencing is missed, and wherein the fault event messages information including: an identification of the nodes that participate in this fault event, the action that each of node is to perform, and an order in which the actions are performed.