Method, apparatus, and computer-readable medium for dynamic binding of tasks in a data exchange
Apparatus, computer-readable medium, and computer-implemented method for dynamic binding of tasks in a data exchange, including generating an execution plan for the data exchange comprising a plurality of streams, each stream corresponding to a pairing of a producer task and a consumer task and comprising a producer task identifier and a consumer task identifier, binding each producer task to a node at execution time by associating a corresponding producer task identifier with a corresponding node identifier of the node, binding each consumer task to a node at execution time by associating a corresponding consumer task identifier with a node identifier of the corresponding node, and routing an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream corresponding to the pairing of the producer task and the consumer task.
1. A method executed by one or more computing devices for dynamic binding of tasks in a data exchange, the method comprising:
generating, by at least one of the one or more computing devices, an execution plan for the data exchange, the execution plan comprising a plurality of streams, each stream corresponding to a pairing of a producer task and a consumer task and comprising a producer task identifier and a consumer task identifier;
binding, by at least one of the one or more computing devices, each producer task in a plurality of producer tasks to a node in a plurality of nodes at execution time by associating a corresponding producer task identifier with a corresponding node identifier of the node;
binding, by at least one of the one or more computing devices, each consumer task in a plurality of consumer tasks to a node in the plurality of nodes at execution time by associating a corresponding consumer task identifier with a node identifier of the corresponding node; and
routing, by at least one of the one or more computing devices, an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task.
2. The method of claim 1 , wherein each stream is configured to map the output of a corresponding producer task as input to a corresponding consumer task when the consumer task is executed.
3. The method of claim 1 , wherein the output of each producer task is written to a corresponding stream in the plurality of streams via a stream application programming interface (API).
4. The method of claim 1 , wherein routing an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task comprises:
writing the output of the producer task to the stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task;
detecting execution of the paired consumer task on the node bound to the paired consumer task; and
transmitting the output of the producer task from the corresponding stream to the node bound to the paired consumer task.
5. The method of claim 4 , wherein the output of the producer task is transmitted from the corresponding stream to the node bound to the paired consumer task without being stored in persistent storage.
6. The method of claim 4 , wherein detecting execution of the paired consumer task on the node bound to the paired consumer task comprises:
executing one or more tasks in the plurality of tasks on one or more nodes based at least in part on the execution plan;
transmitting one or more broadcasts indicating the one or more tasks which are being executed on the one or more nodes; and
identifying a node in the one or more nodes which is executing the paired consumer task based at least in part on the one or more broadcasts.
7. An apparatus for dynamic binding of tasks in a data exchange, the apparatus comprising:
one or more processors; and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
generate an execution plan for the data exchange, the execution plan comprising a plurality of streams, each stream corresponding to a pairing of a producer task and a consumer task and comprising a producer task identifier and a consumer task identifier;
bind each producer task in a plurality of producer tasks to a node in a plurality of nodes at execution time by associating a corresponding producer task identifier with a corresponding node identifier of the node;
bind each consumer task in a plurality of consumer tasks to a node in the plurality of nodes at execution time by associating a corresponding consumer task identifier with a node identifier of the corresponding node; and
route an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task.
8. The apparatus of claim 7 , wherein each stream is configured to map the output of a corresponding producer task as input to a corresponding consumer task when the consumer task is executed.
9. The apparatus of claim 7 , wherein the output of each producer task is written to a corresponding stream in the plurality of streams via a stream application programming interface (API).
10. The apparatus of claim 7 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to route an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task further cause at least one of the one or more processors to:
write the output of the producer task to the stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task;
detect execution of the paired consumer task on the node bound to the paired consumer task; and
transmit the output of the producer task from the corresponding stream to the node bound to the paired consumer task.
11. The apparatus of claim 10 , wherein the output of the producer task is transmitted from the corresponding stream to the node bound to the paired consumer task without being stored in persistent storage.
12. The apparatus of claim 10 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to detect execution of the paired consumer task on the node bound to the paired consumer task further cause at least one of the one or more processors to:
execute one or more tasks in the plurality of tasks on one or more nodes based at least in part on the execution plan;
transmit one or more broadcasts indicating the one or more tasks which are being executed on the one or more nodes; and
identify a node in the one or more nodes which is executing the paired consumer task based at least in part on the one or more broadcasts.
13. At least one non-transitory computer-readable medium storing computer-readable instructions for dynamic binding of tasks in a data exchange that, when executed by one or more computing devices, cause at least one of the one or more computing devices to:
generate an execution plan for the data exchange, the execution plan comprising a plurality of streams, each stream corresponding to a pairing of a producer task and a consumer task and comprising a producer task identifier and a consumer task identifier;
bind each producer task in a plurality of producer tasks to a node in a plurality of nodes at execution time by associating a corresponding producer task identifier with a corresponding node identifier of the node;
bind each consumer task in a plurality of consumer tasks to a node in the plurality of nodes at execution time by associating a corresponding consumer task identifier with a node identifier of the corresponding node; and
route an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task.
14. The apparatus of claim 13 , wherein each stream is configured to map the output of a corresponding producer task as input to a corresponding consumer task when the consumer task is executed.
15. The apparatus of claim 13 , wherein the output of each producer task is written to a corresponding stream in the plurality of streams via a stream application programming interface (API).
16. The apparatus of claim 13 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to route an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task further cause at least one of the one or more computing devices to:
write the output of the producer task to the stream in the plurality of streams that corresponds to the pairing of the producer task and the consumer task;
detect execution of the paired consumer task on the node bound to the paired consumer task; and
transmit the output of the producer task from the corresponding stream to the node bound to the paired consumer task.
17. The apparatus of claim 16 , wherein the output of the producer task is transmitted from the corresponding stream to the node bound to the paired consumer task without being stored in persistent storage.
18. The apparatus of claim 16 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to detect execution of the paired consumer task on the node bound to the paired consumer task further cause at least one of the one or more computing devices to:
execute one or more tasks in the plurality of tasks on one or more nodes based at least in part on the execution plan;
transmit one or more broadcasts indicating the one or more tasks which are being executed on the one or more nodes; and
identify a node in the one or more nodes which is executing the paired consumer task based at least in part on the one or more broadcasts.