Apparatus and method for dataflow execution in a distributed environment using directed acyclic graph and prioritization of sub-dataflow tasks

1. A non-transitory computer readable storage medium, comprising executable instructions to:

assess system cache resources, inter-process communication requirements and staging requirements to divide an extract, transform, load (ETL) dataflow task into a plurality of sub-tasks; and

execute the sub-tasks on distributed resources;

wherein a directed acyclic graph (DAG) is generated that represents execution order dependencies among the plurality of sub-tasks and is used to divide the ETL task based on the accessed system cache resources, inter-process communication requirements and staging requirements, each node of the DAG representing a group of sub-data flows that can be executed together, each of the sub-data flows representing a sub-task process at run time, arrows between two nodes of the DAG indicating a dependency between two groups of sub-data flows;

wherein:

the nodes of the DAG are sorted to generate a list of prioritized group of sub-data flows;

the sub-tasks are executed according to execution priorities specified by the list of prioritized groups of sub-data flows such that:

sub-data flows having different priorities are run in serial;

sub-data flows belonging to a group having a higher priority are executed prior to sub-data flows belonging to groups having a lower priority; and

sub-data flows having a same priority are run in serial or in parallel.

2. The non-transitory computer readable storage medium of claim 1 wherein at least one sub-task is a type-1 task using a single process mode with in-memory cache.

3. The non-transitory computer readable storage medium of claim 1 wherein at least one sub task is a type-2 task using a single process mode with a pageable cache.

4. The non-transitory computer readable storage medium of claim 1 wherein at least one sub task is a type-3 task using multiple process modes with inter-process communication.

5. The non-transitory computer readable storage medium of claim 1 wherein at least one sub task is a type-4 task using multiple process modes with staging between processes.

6. The non-transitory computer readable storage medium of claim 1 wherein at least one sub task is a type-5 task using multiple process modes with mixed staging and inter-process communications between sub-tasks.

7. The non-transitory computer readable storage medium of claim 1 wherein the executable instructions to assess include executable instructions to assess previous job execution statistics.

8. The non-transitory computer readable storage medium of claim 7 wherein the executable instructions to assess include executable instructions to identify pipelined sub-tasks to be run in parallel.

9. The non-transitory computer readable storage medium of claim 7 wherein the executable instructions to assess include executable instructions to identify staged sub-tasks to be run in serial.

10. The non-transitory computer readable storage medium of claim 1 wherein the executable instructions to assess include executable instructions to assess cluster level resources.

11. The non-transitory computer readable storage medium of claim 1 wherein the executable instructions to assess include executable instructions to assess current resource loads.

12. The non-transitory computer readable storage medium of claim 1 wherein the executable instructions to assess include executable instructions to assess locality of objects to be accessed by sub-tasks.

13. The non-transitory computer readable storage medium of claim 1 further comprising executable instructions to support named pipe communications between sub-tasks.

14. The non-transitory computer readable storage medium of claim 1 further comprising executable instructions to support peer-to-peer communications.

15. The non-transitory computer readable storage medium of claim 1 further comprising executable instructions to support centralized message broker communications.

16. The non-transitory computer readable storage medium of claim 1 , wherein:

the task types are selected from a group consisting of: a type-1 task, a type-2 task, a type 3-task, a type-4 task, and a type-5 task;

type-1 tasks use a single process mode with in-memory cache;

type-2 tasks use a single process mode with a paging cache;

type-3 tasks use multiple process modes with inter-process communication;

type-4 tasks use multiple process modes with staging between processes; and

type-5 tasks use multiple process modes with mixed staging and inter-process communications between sub-tasks.

17. A method for implementation by one or more data processors comprising:

assessing, by at least one data processor, system cache resources, inter-process communication requirements and staging requirements to divide an extract, transform, load (ETL) dataflow task into a plurality of sub-tasks; and

executing, by at least one data processor, the sub-tasks on distributed resources;

wherein a directed acyclic graph (DAG) is generated that represents execution order dependencies among the plurality of sub-tasks and is used to divide the ETL task based on the accessed system cache resources, inter-process communication requirements and staging requirements, each node of the DAG representing a group of sub-data flows that can be executed together, each of the sub-data flows representing a sub-task process at run time, arrows between two nodes of the DAG indicating a dependency between two groups of sub-data flows;

wherein:

the nodes of the DAG are sorted to generate a list of prioritized group of sub-data flows;

the sub-tasks are executed according to execution priorities specified by the list of prioritized groups of sub-data flows such that:

sub-data flows having different priorities are run in serial;

sub-data flows belonging to a group having a higher priority are executed prior to sub-data flows belonging to groups having a lower priority; and

sub-data flows having a same priority are run in serial or in parallel.

18. A system comprising:

at least one data processor;

memory storing instructions, which when executed, cause the at least one data processor to perform operations comprising:

assessing system cache resources, inter-process communication requirements and staging requirements to divide an extract, transform, load (ETL) dataflow task into a plurality of sub-tasks; and

executing the sub-tasks on distributed resources;

wherein a directed acyclic graph (DAG) is generated that represents execution order dependencies among the plurality of sub-tasks and is used to divide the ETL task based on the accessed system cache resources, inter-process communication requirements and staging requirements, each node of the DAG representing a group of sub-data flows that can be executed together, each of the sub-data flows representing a sub-task process at run time, arrows between two nodes of the DAG indicating a dependency between two groups of sub-data flows;

wherein:

the nodes of the DAG are sorted to generate a list of prioritized group of sub-data flows;

the sub-tasks are executed according to execution priorities specified by the list of prioritized groups of sub-data flows such that:

sub-data flows having different priorities are run in serial;

sub-data flows belonging to a group having a higher priority are executed prior to sub-data flows belonging to groups having a lower priority; and

sub-data flows having a same priority are run in serial or in parallel.