Managing data queries
In one aspect, in general, a method of generating a dataflow graph representing a database query includes receiving a query plan from a plan generator, the query plan representing operations for executing a database query on at least one input representing a source of data, producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan, and altering one or more components of the dataflow graph based on at least one characteristic of the at least one input representing the source of data.
1. A computer-implemented method of executing a query, the method including:
receiving a query;
identifying a first data source based on the query;
identifying a second data source based on the query, the second data source being associated with at least some data processing functionality that is different from at least some data processing functionality of the first data source;
generating a request to a query planner based on the query;
providing the request to the query planner;
receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;
generating a set of operations that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;
receiving data from the first data source and the second data source; and
executing, on an executing system other than a system managing a relational database, program code based on the set of operations, whereby executing the program code produces results of the query.
2. The method of claim 1 in which the query is expressed in accordance with a query language applicable to a relational database.
3. The method of claim 1 in which the first data source and second data source store data in tangible, non-transitory computer-readable media.
4. The method of claim 1 in which generating the set of operations includes:
generating, based on the query plan, a data structure instantiating a dataflow graph that includes:
a first node that represents at least one operation for execution,
the first node associated with information usable by an executing system to invoke executable program code to perform the operation,
the first node associated with information usable by an executing system to make data available to the program code, and
the operation being chosen based on a first step described by the query plan, and
at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph.
5. The method of claim 4 in which the dataflow graph includes a third node associated with information processed by an executing system to receive the data from the first data source, and a fourth node associated with information processed by an executing system to receive the data from the second data source.
6. The method of claim 5 in which the third node is associated with information processed by an executing system to invoke executable program code that causes a request to perform at least one operation chosen based on a second step described by the query plan, wherein the request is generated for transmission to a system managing the first data source.
7. The method of claim 6 including determining that the system managing the first data source has functionality for performing the at least one operation chosen based on the second step described by the query plan.
8. The method of claim 6 including analyzing information about the second data source, and determining that a system managing the second data source does not have functionality for performing the at least one operation chosen based on the second step described by the query plan.
9. The method of claim 4 including instantiating a link between a node of the dataflow graph and a node of a second dataflow graph, the second dataflow graph including at least one node that represents an operation for execution other than an operation chosen based on any step described by the query plan.
10. The method of claim 1 in which the first data source includes a relational database table.
11. The method of claim 10 in which the second data source includes a portion of data other than a relational database table.
12. The method of claim 10 in which the second data source includes a flat file.
13. The method of claim 1 in which the query includes SQL.
14. The method of claim 1 , including receiving, as output of the program code, results of the query.
15. The method of claim 1 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.
16. The method of claim 1 , wherein data processing functionality includes executable functionality configured to carry out at least one data processing instruction.
17. The method of claim 1 , wherein the combination operation includes a join operation.
18. The method of claim 1 , wherein the data summarization operation includes an aggregation operation.
19. A non-transitory computer-readable medium including computer-executable instructions for executing a query, the executable instructions causing a computer to carry out steps including:
receiving a query;
identifying a first data source based on the query;
identifying a second data source based on the query, the second data source being associated with at least some data processing functionality that is different from at least some data processing functionality of the first data source;
generating a request to a query planner based on the query;
providing the request to the query planner;
receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;
generating a set of operations that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;
receiving data from the first data source and the second data source; and
executing, on an executing system other than a system managing a relational database, program code based on the set of operations, whereby executing the program code produces results of the query.
20. A computer-implemented method of executing a query, the method including:
receiving a query plan corresponding to a query that is expressed in accordance with a query language applicable to a relational database, the query plan including a description of one or more steps for performance by a system managing a relational database;
generating a set of operations that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;
receiving data from a first data source and a second data source, the second data source being associated with at least some data processing functionality that is different from at least some data processing functionality of the first data source; and
executing, on an executing system other than a system managing a relational database, program code based on the set of executable instructions, whereby executing the program code produces results of the query.
21. The method of claim 20 in which the first data source and second data source store data in tangible, non-transitory computer-readable media.
22. The method of claim 20 in which generating the set of operations includes:
generating, based on the query plan, a data structure instantiating a dataflow graph that includes:
a first node that represents at least one operation for execution,
the first node associated with information usable by an executing system to invoke executable program code to perform the operation,
the first node associated with information usable by an executing system to make data available to the program code, and
the operation being chosen based on a first step described by the query plan, and
at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph.
23. The method of claim 22 in which the dataflow graph includes a third node associated with information processed by an executing system to receive the data from the first data source, and a fourth node associated with information processed by an executing system to receive the data from the second data source.
24. The method of claim 23 in which the third node is associated with information processed by an executing system to invoke executable program code that causes a request to perform at least one operation chosen based on a second step described by the query plan, wherein the request is generated for transmission to a system managing the first data source.
25. The method of claim 24 including determining that the system managing the first data source has functionality for performing the at least one operation chosen based on the second step described by the query plan.
26. The method of claim 24 including analyzing information about the second data source, and determining that a system managing the second data source does not have functionality for performing the at least one operation chosen based on the second step described by the query plan.
27. The method of claim 22 including instantiating a link between a node of the dataflow graph and a node of a second dataflow graph, the second dataflow graph including at least one node that represents an operation for execution other than an operation chosen based on any step described by the query plan.
28. The method of claim 20 in which the first data source includes a relational database table.
29. The method of claim 28 in which the second data source includes a portion of data other than a relational database table.
30. The method of claim 28 in which the second data source includes a flat file.
31. The method of claim 20 in which the query includes SQL.
32. The method of claim 1 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.
33. The method of claim 20 , wherein data processing functionality includes executable functionality configured to carry out at least one data processing instruction.
34. The method of claim 20 , wherein the combination operation includes a join operation.
35. The method of claim 20 , wherein the data summarization operation includes an aggregation operation.
36. The computer-readable medium of claim 19 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.
37. The computer-readable medium of claim 19 , wherein data processing functionality includes executable functionality configured to carry out at least one data processing instruction.