Dataflow graph processing
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.
1. A system, comprising:
a communication interface configured to receive an indication to generate a dataflow graph, wherein the indication includes a set of queries;
a processor coupled to the communication interface configured to:
determine dependencies of each query in the set of queries on another query;
determine a directed acyclic graph (DAG) of nodes based at least in part on the dependencies;
determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes, wherein the determining of the in-line expressions for tables of the dataflow graph comprises:
selecting a selected table of the tables of the dataflow graph;
determining upstream tables from the selected table by tracing the dataflow graph backwards from the table noting calculations of intermediate view nodes; and
determining the inline expressions by using the calculations of the intermediate view nodes from the tracing of the dataflow graph backwards from the table; and
provide the dataflow graph; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. A system of claim 1 , wherein the set of queries are expressed using a domain specific language.
3. A system of claim 2 , wherein the domain specific language comprises Structured Query Language (SQL).
4. A system of claim 1 , wherein a dependency of the dependencies is based at least in part on a primitive of a domain specific language.
5. A system of claim 4 , wherein the primitive of a domain specific language comprises a read of a table.
6. A system of claim 1 , wherein an input node of the DAG comprises an input node with no dependency.
7. A system of claim 1 , wherein an output node of the DAG comprises a node with no nodes that read from the node.
8. A system of claim 1 , wherein determining the dataflow graph comprises a depth first search of the DAG of nodes.
9. A system of claim 1 , wherein determining the dataflow graph comprises a breadth first search of the DAG of nodes.
10. A system of claim 1 , wherein the view nodes are designated by a user.
11. A system of claim 1 , wherein determining in-line expressions for tables of the dataflow graph further comprises inserting the inline expressions in the table of the dataflow graph.
12. A system of claim 1 , wherein the processor is further configured to verify the dataflow graph.
13. A system of claim 1 , wherein the processor is further configured to store the dataflow graph.
14. A system of claim 1 , wherein the processor is further configured to generate a report regarding the dataflow graph.
15. A method, comprising:
receiving an indication to generate a dataflow graph, wherein the indication includes a set of queries;
determining, using a processor, dependencies of each query in the set of queries on another query;
determining a directed acyclic graph (DAG) of nodes based at least in part on the dependencies;
determining the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes, wherein the determining of the in-line expressions for tables of the dataflow graph comprises:
selecting a selected table of the tables of the dataflow graph;
determining upstream tables from the selected table by tracing the dataflow graph backwards from the table noting calculations of intermediate view nodes; and
determining the inline expressions by using the calculations of the intermediate view nodes from the tracing of the dataflow graph backwards from the table; and
providing the dataflow graph.
16. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving an indication to generate a dataflow graph, wherein the indication includes a set of queries;
determining, using a processor, dependencies of each query in the set of queries on another query;
determining a directed acyclic graph (DAG) of nodes based at least in part on the dependencies;
determining the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes, wherein the determining of the in-line expressions for tables of the dataflow graph comprises:
selecting a selected table of the tables of the dataflow graph;
determining upstream tables from the selected table by tracing the dataflow graph backwards from the table noting calculations of intermediate view nodes; and
determining the inline expressions by using the calculations of the intermediate view nodes from the tracing of the dataflow graph backwards from the table; and
providing the dataflow graph.