Fault-tolerant and highly-available configuration of distributed services
Fault-tolerant and highly available configuration of distributed services including a computer-implemented method for distributed task orchestration. The method includes setting a watch on a first key in a first replica of a distributed configuration store and setting a value of a second key in a second replica of the distributed configuration store where the value of the second key indicates that a first sub-task of a distributed task is successfully complete. The value of the second key is received based on the watch set on the first key and execution of the second sub-tasks is initiated based on receiving the value of the second key. Also disclosed are techniques for service lifecycle management.
1. A method for distributed task orchestration, the method comprising:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.
2. The method of claim 1 , wherein the watch set of the first key is a forward watch.
3. The method of claim 1 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.
4. The method of claim 1 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.
5. The method of claim 1 , further comprising:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.
6. One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the watch set of the first key is a forward watch.
8. The one or more non-transitory computer-readable media of claim 6 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.
9. The one or more non-transitory computer-readable media of claim 6 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.
10. The one or more non-transitory computer-readable media of claim 6 , the one or more programs further comprising instructions for:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.
11. A system, comprising:
one or more processors;
one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.
12. The system of claim 11 , wherein the watch set of the first key is a forward watch.
13. The system of claim 11 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.
14. The system of claim 11 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.
15. The system of claim 11 , the one or more programs further comprising instructions for:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.
16. The system of claim 11 , the one or more programs further comprising instructions for:
setting a watch on a third key in the second replica;
based, at least in part, on successful completion of the second sub-task, setting a value of a fourth key in the first replica;
based, at least in part, on the watch set on the third key, receiving the value of the fourth key; and
based, at least in part, on the receiving the value of the fourth key, initiating execution of a third sub-task of the distributed task that is dependent on successful completion of the second sub-task.
17. The system of claim 11 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the one or more programs further comprise instructions for:
performing the setting the watch on the first key in the first replica at the first host;
performing the setting the value of the second key in the second replica at the second host;
performing the receiving the value of the second key at the first host; and
performing the initiating execution of the second sub-task at the first host.
18. The system of claim 11 , wherein the receiving the value of the second key is based, at least in part, on a consensus protocol.
19. The system of claim 11 , the one or more programs further comprising instructions for:
setting a value of a third key in the first replica based, at least in part, successful completion of the second sub-task, the value of the third key indicating that the second sub-task completed successfully.