Recoverable asynchronous message driven processing in a multi-node system
An approach provides for recovery of message driven directors responsible for dynamically allocating resources within a multi-node database system. The receipt of a message, the action taken in response to it, and other messages sent in furtherance of the action, are performed within a single transaction. In this way, the messages and other data are left in a state consistent with the message and action taken in response being fully processed or not processed at all. A standby director, stepping in for a crashed director, inherits a state consistent with where a crashed director last successfully completed an action. The approach insulates directors from the special perils of performing high latency actions by delegating the performance of such actions to a job, (i.e. another process).
1. A method for performing actions in response to messages transmitted within a computer system, the method comprising steps of:
receiving within a queue a first message that notifies of a condition;
as a first transaction initiated by a first entity, performing first operations that include:
dequeueing the first message from the queue, and
performing a first operation related to the condition;
wherein said first transaction terminates without being committed;
as a second transaction initiated by a second entity, performing second operations that include:
dequeueing the first message from the queue, and
performing a second operation related to the condition;
committing said second transaction; and
wherein the steps are performed by one or more computing machines.
2. The method of claim 1 , wherein:
the first message notifies of a violation of service-level agreements associated with the computer system; and
the first operation attempts to adjust allocation of resources within the computer system to remedy the violation.
3. The method of claim 2 , wherein the first operations include requesting that a second message be transmitted.
4. The method of claim 3 , wherein:
the first operations include requesting that a third message be transmitted to a third entity, said third message requesting an action related to the condition; and
the second operations include requesting a fourth message to be transmitted to said third entity, said fourth message requesting said action related to the condition.
5. The method of claim 3 , wherein the first operations include causing a third message for a recipient to be added to the queue in a way that causes the third message to be received by the recipient after a specified period of time, wherein said third message notifies of said condition.
6. The method of claim 5 , after said specified period of time:
as a third transaction, performing third operations that include:
dequeueing the third message from the queue, and
performing a third operation related to the condition.
7. The method of claim 5 , the steps further including:
receiving within the queue a fourth message that notifies that said condition has abated;
supplying said fourth message to said first entity; and
wherein supplying said fourth message to said first entity causes dequeueing of said third message.
8. The method of claim 2 , wherein:
said first entity includes a first process executing on a node of said computer system; and
the first operations include initiating an asynchronous process executing on said computer system to perform said first operation.
9. The method of claim 8 , wherein the first operations include causing a third message for a recipient to be added to the queue in a way that causes the third message to be received by the recipient after a specified period of time, wherein said third message identifies said asynchronous process.
10. The method of claim 9 , the steps further including:
receiving said third message;
determining that said third message identifies said asynchronous process; and
in response to determining that said third message identifies said asynchronous process, ceasing execution of said asynchronous process.
11. The method of claim 9 , the steps further including:
receiving a fourth message that includes data that:
indicates a result of the first operation performed by said asynchronous process performing the action, and
identifies said asynchronous process; and
in response to receiving the fourth message, dequeueing the third message.
12. The method of claim 1 , wherein:
the computer system includes a first node and a second node;
the first entity is hosted by said first node;
the second entity is hosted by said second node; and
the second entity serves as standby to perform a responsibility of said first entity in response to ceasing execution of said first entity.
13. The method of claim 12 , the steps further including:
detecting expiration of a period of time; and
in response to detecting the expiration of the period of time, ceasing execution of said first entity.
14. A non-transitory computer-readable medium storing instructions encoded therewith which, when executed with one or more processors of a computer system, causes performance of steps of:
as a first transaction initiated by a first entity, performing first operations that include:
dequeueing the first message from the queue, and
performing a first operation related to the condition;
wherein said first transaction terminates without being committed;
as a second transaction initiated by a second entity, performing second operations that include:
dequeueing the first message from the queue, and
performing a second operation related to the condition; and
committing said second transaction.
15. The non-transitory computer-readable medium of claim 14 , wherein:
the first message notifies of a violation of service-level agreements associated with the computer system; and
the first operation attempts to adjust allocation of resources within the computer system to remedy the violation.
16. The non-transitory computer-readable medium of claim 15 , wherein the first operations include requesting that a second message be transmitted.
17. The non-transitory computer-readable medium of claim 16 , wherein:
the first operations include requesting that a third message be transmitted to a third entity, said third message requesting an action related to the condition; and
the second operations include requesting a fourth message to be transmitted to said third entity, said fourth message requesting said action related to the condition.
18. The non-transitory computer-readable medium of claim 16 , wherein the first operations include causing a third message for a recipient to be added to the queue in a way that causes the third message to be received by the recipient after a specified period of time, wherein said third message notifies of said condition.
19. The non-transitory computer-readable medium of claim 18 , after said specified period of time:
as a third transaction, performing third operations that include:
dequeueing the third message from the queue, and
performing a third operation related to the condition.
20. The non-transitory computer-readable medium of claim 18 , the steps further including:
receiving within the queue a fourth message that notifies that said condition has abated;
supplying said fourth message to said first entity; and
wherein supplying said fourth message to said first entity causes dequeueing of said third message.
21. The non-transitory computer-readable medium of claim 15 , wherein:
said first entity includes a first process executing on a node of said computer system; and
the first operations include initiating an asynchronous process executing on said computer system to perform said first operation.
22. The non-transitory computer-readable medium of claim 21 , wherein the first operations include causing a third message for a recipient to be added to the queue in a way that causes the third message to be received by the recipient after a specified period of time, wherein said third message identifies said asynchronous process.
23. The non-transitory computer-readable medium of claim 22 , the steps further including:
receiving said third message;
determining that said third message identifies said asynchronous process; and
in response to determining that said third message identifies said asynchronous process, ceasing execution of said asynchronous process.
24. The non-transitory computer-readable medium of claim 22 , the steps further including:
receiving a fourth message that includes data that:
indicates a result of the first operation performed by said asynchronous process performing the action, and
identifies said asynchronous process; and
in response to receiving the fourth message, dequeueing the third message.
25. The non-transitory computer-readable medium of claim 14 , wherein:
the computer system includes a first node and a second node;
the first entity is hosted by said first node;
the second entity is hosted by said second node; and
the second entity serves as standby to perform a responsibility of said first entity in response to ceasing execution of said first entity.
26. The non-transitory computer-readable medium of claim 25 , the steps further including:
detecting expiration of a period of time; and in response to detecting the expiration of the period of time, ceasing execution of said first entity.