Query generation and processing system

1. A system for processing queries on batch and streaming datasets, the system comprising:

a relational data store configured to ingest data from one or more data sources and store data in a first dataset and a second dataset;

a query generator configured to interpret a data expression in a simplified query language to generate a query in a structured query language based on a first quad corresponding to the first dataset and a second quad corresponding to the second dataset identified based on the data expression and determining an implicit join between the first quad and the second quad based on an unambiguous relationship obtainable from a schema of the first dataset and the second dataset, wherein the data expression does not expressly identify a join between the first quad and the second quad; and

a query processor configured to generate a query pipeline that uses the data of the first dataset and the second dataset stored by the relational data store to execute the query generated by the query processor,

wherein the query pipeline includes one or more compute nodes instantiated based on a query plan for a query, wherein the one or more compute nodes include alternating layers of faucets and turbines, and wherein an upstream faucet transmits pointers to the ingested data to one or more downstream turbines.

2. The system of claim 1 , wherein the relational data stores the data in the first dataset and the second data set directly into a memory buffer to enable compute nodes of the query pipeline to access to the data without making a copy of the data and without waiting for the data to be stored in a cold storage.

3. The system of claim 2 , wherein the relational data store stores data in blocks, and wherein the relational data store is configured to move a block of the blocks between the memory buffer, a local storage, and the cold storage based on a predicted relevance of data stored in the block.

4. The system of claim 1 , wherein the query generator is configured to parse the data expression to produce a tree of quads including the first quad and the second quad, generate a derivation graph including nodes representing the quads, and query the derivation graph based on a grain of the quads to generate the query.

5. The system of claim 4 , wherein the grain represents one or more dimensions of data stored by the relational data store.

6. The system of claim 4 , wherein each of the quads is represented by a data expression that identifies a subset of data stored by the relational data store.

7. The system of claim 1 , wherein the upstream faucet pulls new data ingested by the relational data store on a periodic basis independent of a time period used to aggregate the data.

8. The system of claim 1 , wherein the relational data store, the query generator, and the query processor are implemented on a common computing device.

9. The system of claim 1 , wherein a web application that allows a user to write the data expression and to configure the data ingestion from the one or more data sources outputs results of the executed query.

10. A method for processing queries on batch and streaming datasets, the method comprising:

storing data ingested from one or more data sources in a first dataset and a second dataset;

obtaining a data expression in a simplified query language;

identifying a first quad corresponding to the first dataset and a second quad corresponding to the second dataset based on the data expression, wherein the data expression does not expressly identify a join between the first quad and the second quad;

determining an implicit join between the first quad and the second quad based on an unambiguous relationship obtainable from a schema of the first dataset and the second dataset, including:

generating a derivation graph including nodes representing the quads, the generating including producing a tree including leaves corresponding to the quads based on an arrangement of the quads within the data expression and recursively iterating through leaves of the tree to add nodes representing the quads to the derivation graph, and

querying the derivation graph based on a grain of the quads; and

generating a query in a structured query language based on the implicit join between the first quad and the second quad; and

executing the query using a query pipeline that uses the data of the first dataset and the second dataset.

11. The method of claim 10 , wherein the data stored in the first dataset and the second data set are stored directly into a memory buffer to enable compute nodes of the query pipeline that executes the query to access to the data without making a copy of the data and without waiting for the data to be stored in a cold storage.

12. The method of claim 10 , wherein the query pipeline includes one or more compute nodes instantiated based on a query plan for a query, wherein the one or more compute nodes include alternating layers of faucets and turbines, and wherein an upstream faucet transmits pointers to the ingested data to one or more downstream turbines.

13. An apparatus for processing queries on batch and streaming datasets, the apparatus comprising:

a memory storing instructions; and

a processor configured to execute the instructions to:

store data ingested from one or more data sources in a first dataset and a second dataset;

obtain a data expression in a simplified query language;

generate a query in a structured query language based on an implicit join determined between a first quad corresponding to the first dataset and a second quad corresponding to the second dataset based on an unambiguous relationship obtainable from a schema of the first dataset and the second dataset, wherein the data expression does not expressly identify a join between the first quad and the second quad; and

execute the query using a query pipeline that uses the data of the first dataset and the second dataset,

wherein the data stored in the first dataset and the second data set are stored directly into a memory buffer to enable compute nodes of a query pipeline that executes the query to access to the data without making a copy of the data and without waiting for the data to be stored in a cold storage,

wherein the query pipeline includes one or more compute nodes instantiated based on a query plan for a query, wherein the one or more compute nodes include alternating layers of faucets and turbines, and wherein an upstream faucet transmits pointers to the ingested data within the memory buffer to one or more downstream turbines.

14. The apparatus of claim 13 , wherein the instructions to generate the query in the structured query language based on the implicit join include instructions to:

generate a derivation graph including nodes representing the quads; and

query the derivation graph based on a grain of the quads to determine the implicit join.

15. The apparatus of claim 13 , wherein a web application that allows a user to write the data expression and to configure the data ingestion from the one or more data sources outputs results of the executed query.