Effective high availability cluster management and effective state propagation for failure recovery in high availability clusters

1. A node availability management system comprising:

a president node comprising:

a message queue to store a message;

a decision engine comprising:

an operation manager configured to generate an operation based on said message, and wherein said operation includes a plurality of atomic actions arranged in a dependency graph to respond to the message,

a graph builder configured to build the dependency graph for the operation using the plurality of atomic actions,

a domain manager configured to:

determine an operation domain comprising at least one of a plurality of nodes affected by the operation and at least one of a plurality of components affected by the operation,

compare the operation domain of the operation to a second operation domain of a concurrently executing second operation,

determine whether the operation domain of the operation conflicts with the second operation domain of the concurrently executing second operation, wherein the domain manager locks the operation domain when no conflict exists, and

release the operation domain when the operation is completed;

an executor configured to execute said operation by executing the plurality of atomic actions in the dependency graph in breadth-first order,

wherein a parent atomic action of the plurality of atomic actions is completely executed before a child atomic action of the plurality of atomic actions is executed,

wherein the executor maintains a queue of parent atomic actions, and wherein the executor is further configured to:

add the parent atomic action to the queue,

execute the parent atomic action,

remove the parent atomic action from the queue,

add the dependent child atomic actions to the queue, and

tag the operation as complete when the queue is empty; and

a slave node operatively connected to the president node,

wherein the executor offloads the execution of at least one of the plurality of atomic actions to the slave node, and

wherein executing the plurality of atomic actions is suspended until one of a group consisting of the slave node replying to the atomic action and a timeout occurs.

2. The system of claim 1 , wherein the president node is configured to act as the slave node.

3. The system of claim 1 , wherein the president node further comprises:

a domain manager further configured to maintain a waiting list that stores the operation when the conflict exists, wherein the operation is stored until the conflict is resolved.

4. The system of claim 1 , wherein the president node further comprises:

a checkpoint module configured to checkpoint the dependency graph associated with the operation to a checkpoint domain, wherein the checkpoint domain comprises the at least one of a plurality of nodes in the operation domain, and wherein each time at least one of the plurality of atomic actions in the dependency graph is executed, the dependency graph is checkpointed asynchronously to the nodes in the checkpoint domain.

5. The system of claim 1 , wherein the executor is configured to execute multiple operations in parallel.

6. A distributed computing system for responding to an event comprising:

a president node configured to convert the event to a message; and

a slave node operatively connected to the president node,

wherein the president node generates an operation in response to the message comprising a dependency graph of atomic actions;

wherein the president node assigns a priority to the dependency graph wherein a higher-priority dependency graph is executed prior to a lower-priority dependency graph;

wherein the president node comprises an executor for executing the plurality of atomic actions of said operation in breadth-first order, wherein the executor maintains a queue of parent atomic actions, and wherein the executor is further configured to:

add the parent atomic action to the queue,

execute the parent atomic action,

remove the parent atomic action from the queue,

add the dependent child actions to the queue, and

tag the operation as complete when the queue is empty;

wherein the president node is configured to:

determine an operation domain comprising at least one of a plurality of nodes affected by the operation and at least one of a plurality of components affected by the operation,

compare the operation domain of the operation to a second operation domain of a concurrently executing second operation,

determine whether the operation domain of the operation the conflicts with the second operation domain of the concurrently executing second operation, wherein the domain manager locks the operation domain when no conflict exists, and

release the operation domain when the operation is completed; and

wherein the executor offloads the execution of at least one of the plurality of atomic actions to the slave node, and wherein executing the atomic actions is suspended until one of a group consisting of the slave node replying to the atomic action and a timeout occurs.

7. The system of claim 6 , wherein the president node is configured to act as the slave node.

8. The system of claim 6 , wherein the executor is configured to execute multiple operations in parallel.

9. The system of claim 6 , wherein the executor is configured to shrink the dependency graph if execution of at least one of the plurality of atomic actions fails.

10. The system of claim 6 , wherein the executor is configured to report the execution failure of the at least one of the plurality of atomic actions to a decision engine.