Dataflow graph processing with expectations
A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.
1. A system, comprising:
a computer processor; and
a non-transitory computer-readable storage medium comprising instructions that when executed by the computer processor cause the system to perform actions comprising:
receiving, from a client device, an indication to generate an extract, transform and load (ETL) data processing pipeline defined with respect to a set of operations, the set of operations received via a communication interface of the client device and including one or more declarative structured query language (SQL) queries, each operation configured to read one or more input datasets stored in a storage location of a cloud platform and transform the one or more input datasets into one or more output datasets, at least one operation indicating an expectation describing one or more properties data in the input datasets or the output datasets of the at least one operation is expected to possess, wherein at least two operations are not in order;
generating an in-memory dataflow graph for the data processing pipeline,
comprising:
determining data dependencies between the set of operations of the data processing pipeline;
defining, based on the determined dependencies, an ordering in which the set of operations of the data processing pipeline is performed; and
generating the dataflow graph reflecting a correct ordering of the set of operations, the dataflow graph including a set of nodes and edges between the nodes, an edge in the dataflow graph representing a dependency between a node representing an input dataset and another node representing an output dataset for a respective operation;
testing the dataflow graph, comprising:
obtaining a sample set of data and storing the sample set of data in a storage device;
executing the at least one operation to the sample set of data;
determining whether the execution of the at least one operation meets the expectation; and
responsive the expectation being met, determining that the dataflow graph is validated;
processing the set of operations associated with the data processing pipeline such that the expectation for the at least one operation is enforced; and
providing a result of the processing to the client device.
2. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising inserting an expectation node in the dataflow graph at an insertion location to generate an updated dataflow graph, wherein inserting the expectation node in the dataflow graph comprises connecting an input of the expectation node to an output of a node prior to the insertion location.
3. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising inserting an expectation node in the dataflow graph at an insertion location to generate an updated dataflow graph, wherein inserting the expectation node in the dataflow graph comprises connecting an output of the expectation node to an input of a node subsequent to the insertion location.
4. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising inserting an expectation node in the dataflow graph at an insertion location to generate an updated dataflow graph, wherein inserting the expectation node in the dataflow graph comprises connecting an output of the expectation node to an input of a fork node.
5. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising inserting an expectation node in the dataflow graph at an insertion location to generate an updated dataflow graph, wherein inserting the expectation node in the dataflow graph comprises connecting an output of a fork node.
6. The system of claim 1 , wherein the one or more input datasets for an operation are stored at a different storage location from the one or more output datasets for the operation.
7. The system of claim 1 , wherein the instructions that cause the processor to perform actions of processing further comprises instructions that cause the processor to perform actions of checking a condition on the input datasets of the at least one operation.
8. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising stopping the processing in response to one or more records of the input datasets for the at least one operation not meeting a condition.
9. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising stopping the processing in response to one or more records of the input datasets for the at least one operation meeting a condition.
10. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising dropping the processing in response to one or more records of the input datasets for the at least one operation not meeting a condition.
11. The system of claim 1 , wherein the instructions cause the processor to perform further actions comprising dropping the processing in response to one or more records of the input datasets for the at least one operation meeting a condition.
12. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing further comprises instructions that cause the processor to perform actions comprising counting one or more records of the input datasets for the at least one operation in response to the one or more records of the input datasets not meeting a condition.
13. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing further comprises instructions that cause the processor to perform actions comprising counting one or more records of the input datasets for the at least one operation in response to the one or more records of the input datasets meeting a condition.
14. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing further comprises instructions that cause the processor to perform actions comprising quarantining one or more records of the input datasets of the at least one operation in response to the one or more records of the input datasets not meeting a condition.
15. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing further comprises instructions that cause the processor to perform actions comprising quarantining one or more records of the input datasets of the at least one operation in response to the one or more records of the input datasets meeting a condition.
16. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing further comprises instructions that cause the processor to perform actions comprising determining a percentage of records of the input datasets of the at least one operation not meeting a condition.
17. The system of claim 1 , wherein the instructions that cause the processor to perform actions for processing comprises instructions that cause the processor to perform actions comprising determining a percentage of records of the input datasets of the at least one operation meeting a condition.
18. The system of claim 1 , wherein the computer processor is further configured to store the dataflow graph.
19. The system of claim 1 , the instructions further causing the system to perform actions comprising:
verifying the in-memory dataflow graph to determine whether errors are present in the dataflow graph, wherein the verification is performed without materializing the input datasets and the output datasets of the set of operations of the data processing pipeline.
20. A method, comprising:
receiving, from a client device, an indication to generate an extract, transform and load (ETL) data processing pipeline defined with respect to a set of operations, the set of operations received via a communication interface of the client device and including one or more declarative structured query language (SQL) queries, each operation configured to read one or more input datasets and transform the one or more input datasets stored in a storage location of a cloud platform into one or more output datasets, at least one operation indicating an expectation describing one or more properties data in the input datasets or the output datasets of the at least one operation is expected to possess, wherein at least two operations are not in order;
generating an in-memory dataflow graph for the data processing pipeline, comprising:
determining data dependencies between the set of operations of the data processing pipeline;
defining, based on the determined dependencies, an ordering in which the set of operations of the data processing pipeline is performed; and
generating the dataflow graph reflecting a correct ordering of the set of operations, the dataflow graph including a set of nodes and edges between the nodes, an edge in the dataflow graph representing a dependency between a node representing an input dataset and another node representing an output dataset for a respective operation;
testing the dataflow graph, comprising:
obtaining a sample set of data and storing the sample set of data in a storage device;
executing the at least one operation to the sample set of data;
determining whether the execution of the at least one operation meets the expectation; and
responsive to the expectation being met, determining that the dataflow graph is validated;
processing the set of operations associated with the data processing pipeline such that the expectation for the at least one operation is enforced; and
providing a result of the processing to the client device.
21. The method of claim 20 , further comprising:
verifying the in-memory dataflow graph to determine whether errors are present in the dataflow graph, wherein the verification is performed without materializing the input datasets and the output datasets of the set of operations of the data processing pipeline.
22. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving, from a client device, an indication to generate an extract, transform and load (ETL) data processing pipeline defined with respect to a set of operations, the set of operations received via a communication interface of the client device and including one or more declarative structured query language (SQL) queries, each operation configured to read one or more input datasets and transform the one or more input datasets stored in a storage location of a cloud platform into one or more output datasets, at least one operation indicating an expectation describing one or more properties data in the input datasets or the output datasets of the at least one operation is expected to possess, wherein at least two operations are not in order;
generating an in-memory dataflow graph for the data processing pipeline, comprising:
determining data dependencies between the set of operations of the data processing pipeline;
defining, based on the determined dependencies, an ordering in which the set of operations of the data processing pipeline is performed; and
generating the dataflow graph reflecting a correct ordering of the set of operations, the dataflow graph including a set of nodes and edges between the nodes, an edge in the dataflow graph representing a dependency between a node representing an input dataset and another node representing an output dataset for a respective operation;
testing the dataflow graph, comprising:
obtaining a sample set of data and storing the sample set of data in a storage device:
executing the at least one operation to the sample set of data;
determining whether the execution of the at least one operation meets the expectation; and
responsive to the expectation being met, determining that the dataflow graph is validated;
processing the set of operations associated with the data processing pipeline such that the expectation for the at least one operation is enforced; and
providing a result of the processing to the client device.
23. The computer program product of claim 22 , further comprising computer instructions further for:
verifying the in-memory dataflow graph to determine whether errors are present in the dataflow graph, wherein the verification is performed without materializing the input datasets and the output datasets of the set of operations of the data processing pipeline.