Method, apparatus, and computer-readable medium for performing a data exchange on a data exchange framework

1. A method executed by one or more computing devices of a data exchange framework (DEF) for performing a data exchange, the method comprising:

receiving, by an orchestrator component of the DEF, a plurality of tasks for execution;

generating, by the orchestrator component of the DEF, an execution plan for executing the plurality of tasks on a plurality of nodes of the DEF, the execution plan comprising one or more data exchanges, each data exchange comprising at least one stream, and each stream including a logical identifier of a producer task in the plurality of tasks and a logical identifier of a consumer task in the plurality of tasks and being configured to map output of the producer task as input to the consumer task when the consumer task is executed;

executing, by one or more first nodes of the plurality of nodes of the DEF, one or more producer tasks of one or more streams of a data exchange in the one or more data exchanges based at least in part on the execution plan; and

transmitting, by the one or more first nodes, an output of the one or more producer tasks from the one or more first nodes to the one or more streams of the data exchange via a stream application programming interface (API), the stream API being managed by a stream API component of the DEF that is configured to map the output of the one or more producer tasks as input to one or more consumer tasks executing on one or more second nodes of the DEF based at least in part on detection of execution of the one or more consumer tasks on the one or more second nodes of the DEF.

2. The method of claim 1 , further comprising:

detecting, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the one or more second nodes; and

transmitting, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

3. The method of claim 2 , wherein the output of the at least one corresponding producer task is transmitted from the at least one corresponding stream to the at least one second node without being stored in persistent storage.

4. The method of claim 2 , wherein detecting execution of the at least one consumer task on at least one second node in the plurality of nodes comprises:

detecting, by the stream API component, one or more broadcasts from the one or second nodes, wherein each broadcast in the one or more broadcasts indicates a task in the one or more tasks which is being executed on a corresponding second node in the one or more second nodes; and

identifying, by the stream API component, at least one second node in the one or more second nodes which is executing the at least one consumer task based at least in part on the one or more broadcasts.

5. The method of claim 1 , further comprising:

identifying, by the stream API component, at least one consumer task in the one or more consumer tasks which is not scheduled for execution; and

writing, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to persistent storage via the stream API;

detecting, by the stream API component, execution of the at least one consumer task on at least one second node in the plurality of nodes; and

transmitting, by the stream API component, the output of the at least one producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one producer task is provided as input to the at least one consumer task.

6. The method of claim 1 , further comprising:

determining, by the stream API component, whether a quantity of the plurality of tasks is greater than a predetermined value;

writing, by the stream API component, the output of the one or more producer tasks from the one or more streams to persistent storage via the stream API based at least in part on a determination that the number of tasks in the plurality of tasks is greater than a predetermined value;

detecting, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the plurality of nodes; and

transmitting, by the stream API component, the output of at least one corresponding producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

7. The method of claim 6 , wherein the output of the one or more producer tasks from the one or more streams is grouped into separate files within persistent storage according to one or more consumer tasks corresponding to the one or more producer tasks.

8. The method of claim 1 , further comprising:

identifying, by the stream API component, at least one second stream of the data exchange which shares a producer task with at least one stream in the one or more streams, the shared producer task being executed on a first node in the one or more first nodes; and

transmitting, by the stream API component an output of the shared producer task from the first node to the at least one second stream via the stream API, wherein the output of the shared producer task is provided as input to at least one consumer task of the at least one second stream when the at least one consumer task is executed.

9. The method of claim 8 , wherein transmitting an output of the shared producer task from the first node to the at least one second stream via the stream API comprises:

packaging the at least one stream and the at least one second stream in a single package;

transmitting the package including the at least one first stream and the at least one second stream; and

multiplexing the package to identify at least one consumer task for the at least one first stream and identify the at least one consumer task for the at least one second stream.

10. The method of claim 1 , wherein the stream API component is resident on one or more of: the orchestrator component, the one or more first nodes, or the one or more second nodes.

11. An apparatus of a data exchange framework (DEF) for performing a data exchange, the apparatus comprising:

one or more processors; and

one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

receive, by an orchestrator component of the DEF, a plurality of tasks for execution;

generate, by the orchestrator component of the DEF, an execution plan for executing the plurality of tasks on a plurality of nodes of the DEF, the execution plan comprising one or more data exchanges, each data exchange comprising at least one stream, and each stream including a logical identifier of a producer task in the plurality of tasks and a logical identifier of a consumer task in the plurality of tasks and being configured to map output of the producer task as input to the consumer task when the consumer task is executed;

execute, by one or more first nodes of the plurality of nodes of the DEF, one or more producer tasks of one or more streams of a data exchange in the one or more data exchanges based at least in part on the execution plan; and

transmit, by the one or more first nodes, an output of the one or more producer tasks from the one or more first nodes to the one or more streams of the data exchange via a stream application programming interface (API), the stream API being managed by a stream API component of the DEF that is configured to map the output of the one or more producer tasks as input to one or more consumer tasks executing on one or more second nodes of the DEF based at least in part on detection of execution of the one or more consumer tasks on the one or more second nodes of the DEF.

12. The apparatus of claim 11 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

detect, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the one or more second nodes; and

transmit, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

13. The apparatus of claim 11 , wherein the output of the at least one corresponding producer task is transmitted from the at least one corresponding stream to the at least one second node without being stored in persistent storage.

14. The apparatus of claim 11 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to detect execution of the at least one consumer task on at least one second node in the plurality of nodes further cause at least one of the one or more processors to:

detect, by the stream API component, one or more broadcasts from the one or second nodes, wherein each broadcast in the one or more broadcasts indicates a task in the one or more tasks which is being executed on a corresponding second node in the one or more second nodes; and

identify, by the stream API component, at least one second node in the one or more second nodes which is executing the at least one consumer task based at least in part on the one or more broadcasts.

15. The apparatus of claim 11 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

identify, by the stream API component, at least one consumer task in the one or more consumer tasks which is not scheduled for execution; and

write, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to persistent storage via the stream API;

detect, by the stream API component, execution of the at least one consumer task on at least one second node in the plurality of nodes; and

transmit, by the stream API component, the output of the at least one producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one producer task is provided as input to the at least one consumer task.

16. The apparatus of claim 11 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

determine, by the stream API component, whether a quantity of the plurality of tasks is greater than a predetermined value;

write, by the stream API component, the output of the one or more producer tasks from the one or more streams to persistent storage via the stream API based at least in part on a determination that the number of tasks in the plurality of tasks is greater than a predetermined value;

detect, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the plurality of nodes; and

transmit, by the stream API component, the output of at least one corresponding producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

17. The apparatus of claim 11 , wherein the output of the one or more producer tasks from the one or more streams is grouped into separate files within persistent storage according to one or more consumer tasks corresponding to the one or more producer tasks.

18. The apparatus of claim 11 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

identify, by the stream API component, at least one second stream of the data exchange which shares a producer task with at least one stream in the one or more streams, the shared producer task being executed on a first node in the one or more first nodes; and

transmit, by the stream API component an output of the shared producer task from the first node to the at least one second stream via the stream API, wherein the output of the shared producer task is provided as input to at least one consumer task of the at least one second stream when the at least one consumer task is executed.

19. The apparatus of claim 11 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to transmit an output of the shared producer task from the first node to the at least one second stream via the stream API further cause at least one of the one or more processors to:

package the at least one stream and the at least one second stream in a single package;

transmit the package including the at least one first stream and the at least one second stream; and

multiplex the package to identify at least one consumer task for the at least one first stream and identify the at least one consumer task for the at least one second stream.

20. The apparatus of claim 11 , wherein the stream API component is resident on one or more of: the orchestrator component, the one or more first nodes, or the one or more second nodes.

21. At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by one or more computing devices of a data exchange framework (DEF), cause at least one of the one or more computing devices to:

receive, by an orchestrator component of the DEF, a plurality of tasks for execution;

generate, by the orchestrator component of the DEF, an execution plan for executing the plurality of tasks on a plurality of nodes of the DEF, the execution plan comprising one or more data exchanges, each data exchange comprising at least one stream, and each stream including a logical identifier of a producer task in the plurality of tasks and a logical identifier of a consumer task in the plurality of tasks and being configured to map output of the producer task as input to the consumer task when the consumer task is executed;

execute, by one or more first nodes of the plurality of nodes of the DEF, one or more producer tasks of one or more streams of a data exchange in the one or more data exchanges based at least in part on the execution plan; and

transmit, by the one or more first nodes, an output of the one or more producer tasks from the one or more first nodes to the one or more streams of the data exchange via a stream application programming interface (API), the stream API being managed by a stream API component of the DEF that is configured to map the output of the one or more producer tasks as input to one or more consumer tasks executing on one or more second nodes of the DEF based at least in part on detection of execution of the one or more consumer tasks on the one or more second nodes of the DEF.

22. The at least one non-transitory computer-readable medium of claim 21 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

detect, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the one or more second nodes; and

transmit, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

23. The at least one non-transitory computer-readable medium of claim 21 , wherein the output of the at least one corresponding producer task is transmitted from the at least one corresponding stream to the at least one second node without being stored in persistent storage.

24. The at least one non-transitory computer-readable medium of claim 21 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to detect execution of the at least one consumer task on at least one second node in the plurality of nodes further cause at least one of the one or more computing devices to:

detect, by the stream API component, one or more broadcasts from the one or second nodes, wherein each broadcast in the one or more broadcasts indicates a task in the one or more tasks which is being executed on a corresponding second node in the one or more second nodes; and

identify, by the stream API component, at least one second node in the one or more second nodes which is executing the at least one consumer task based at least in part on the one or more broadcasts.

25. The at least one non-transitory computer-readable medium of claim 21 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

identify, by the stream API component, at least one consumer task in the one or more consumer tasks which is not scheduled for execution; and

write, by the stream API component, output of at least one corresponding producer task in the one or more producer tasks from at least one corresponding stream in the one or more streams to persistent storage via the stream API;

detect, by the stream API component, execution of the at least one consumer task on at least one second node in the plurality of nodes; and

transmit, by the stream API component, the output of the at least one producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one producer task is provided as input to the at least one consumer task.

26. The at least one non-transitory computer-readable medium of claim 21 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

determine, by the stream API component, whether a quantity of the plurality of tasks is greater than a predetermined value;

write, by the stream API component, the output of the one or more producer tasks from the one or more streams to persistent storage via the stream API based at least in part on a determination that the number of tasks in the plurality of tasks is greater than a predetermined value;

detect, by the stream API component, execution of at least one consumer task in the one or more consumer tasks on at least one second node in the plurality of nodes; and

transmit, by the stream API component, the output of at least one corresponding producer task from persistent storage to the at least one second node via the stream API, wherein the output of the at least one corresponding producer task is provided as input to the at least one consumer task.

27. The at least one non-transitory computer-readable medium of claim 21 , wherein the output of the one or more producer tasks from the one or more streams is grouped into separate files within persistent storage according to one or more consumer tasks corresponding to the one or more producer tasks.

28. The at least one non-transitory computer-readable medium of claim 21 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

identify, by the stream API component, at least one second stream of the data exchange which shares a producer task with at least one stream in the one or more streams, the shared producer task being executed on a first node in the one or more first nodes; and

transmit, by the stream API component an output of the shared producer task from the first node to the at least one second stream via the stream API, wherein the output of the shared producer task is provided as input to at least one consumer task of the at least one second stream when the at least one consumer task is executed.

29. The at least one non-transitory computer-readable medium of claim 21 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to transmit an output of the shared producer task from the first node to the at least one second stream via the stream API further cause at least one of the one or more computing devices to:

package the at least one stream and the at least one second stream in a single package;

transmit the package including the at least one first stream and the at least one second stream; and

multiplex the package to identify at least one consumer task for the at least one first stream and identify the at least one consumer task for the at least one second stream.

30. The at least one non-transitory computer-readable medium of claim 21 , wherein the stream API component is resident on one or more of: the orchestrator component, the one or more first nodes, or the one or more second nodes.