Query engine for graph databases and heterogeneous hardware
Database query processing techniques are disclosed. In various embodiments, a query associated with a database is received. A byte code representation of the query is generated, including by decomposing the query into a discrete set of streaming operators defined over associated data frames, wherein the byte code includes code defining for each operator in the discrete set of streaming operators the processing to be performed by that operator and further embodies a data flow graph that defines a flow of data to and through the discrete set of streaming operators. The byte code is executed by a query processing engine to generate and return a query result.
1 . A system, comprising:
a processor configured to:
receive at a client device a query associated with a database, wherein the query is expressed in one of a plurality of supported query languages;
generate a byte code representation of the query, including by decomposing the query into a discrete set of streaming operators defined over associated data frames, wherein the byte code includes code defining for each operator in the discrete set of streaming operators the processing to be performed by that operator and further embodies a data flow graph that defines a flow of data to and through the discrete set of streaming operators;
serialize the byte code and sending the serialized byte code to a remote database server;
receive from the remote server a response to the query; and
return the response; and
a memory coupled to the processor and configured to store the byte code representation of the query;
wherein the remote server is configured to execute the byte code to instantiate one or more instances of operator in the discrete set of streaming operators and use the data flow graph to manage the flow of data to and through the respective streaming operators and use data frames in an output queue of a final operator in the data flow graph to provide the result to the query.
2 . The system of claim 1 , wherein the processor is further configured to optimize the byte code representation of the query.
3 . The system of claim 2 , wherein the byte code is optimized based at least in part using metadata associated with the database.
4 . The system of claim 2 , wherein the byte code is optimized based at least in part using data read from the database.
5 . The system of claim 1 , further comprising a communications interface coupled to the processor and wherein the processor is further configured to send the byte code representation of the query to the database server with which the database is associated.
6 . The system of claim 5 , wherein the database server is configured to optimize or further optimize the byte code representation of the query.
7 . The system of claim 5 , wherein the database server comprises a virtual machine or other runtime configured to execute the byte code representation of the query.
8 . The system of claim 7 , wherein the virtual machine or other runtime is configured to compile the byte code representation of the query to generate executable machine code.
9 . The system of claim 7 , wherein execution of the byte code representation of the query causes the database server to create an instance of each of a plurality of operators comprising the discrete set of streaming operators and provide a scheduler configured to cause data frames to be provided to the respective input queues of each of the plurality of operators.
10 . The system of claim 9 , wherein the scheduler is configured to cause a data frame to be moved from the output queue of a first operator to the input queue of a second operator based at least in part on the data flow graph that defines the flow of data to and through the discrete set of streaming operators.
11 . The system of claim 7 , wherein the database server comprises a plurality of processing cores and the database server is configured to execute portions of the byte code representation of the query in parallel, across multiple of said cores, as permitted or required by the data flow graph that defines the flow of data to and through the discrete set of streaming operators.
12 . The system of claim 7 , wherein the database server comprises a distributed system comprising a plurality of physical machines and the database server is configured to execute portions of the byte code representation of the query across multiple of the physical machines.
13 . The system of claim 12 , wherein the database server implements a first subset of the discrete set of streaming operators at a first machine and a second subset of the discrete set of streaming operators at a second machine.
14 . The system of claim 13 , wherein the database server provides at the first machine a first operator proxy configured to send data frames via a communications channel to the second machine, and wherein the second machine is configured to place said data frames in an input queue of an instance of an associated operator running on the second machine.
15 . The system of claim 7 , wherein the database server is configured to distribute work among the plurality of machines according to a load balancing algorithm.
16 . The system of claim 7 , wherein the database server is configured to allocate work among the plurality of machines in response to back pressure.
17 . The system of claim 1 , wherein the processor is further configured to cache the byte code representation of the query.
18 . The system of claim 17 , wherein the query comprises a first query and wherein the processor is further configured to receive a second query, determine the second query is the same as the first query, and retrieve and use the previously cached byte code representation of the first query to process the second query.
19 . A method, comprising:
receiving at a client device a query associated with a database, wherein the query is expressed in one of a plurality of supported query languages;
generating a byte code representation of the query, including by decomposing the query into a discrete set of streaming operators defined over associated data frames, wherein the byte code includes code defining for each operator in the discrete set of streaming operators the processing to be performed by that operator and further embodies a data flow graph that defines a flow of data to and through the discrete set of streaming operators;
storing the byte code representation of the query;
serializing the byte code and sending the serialized byte code to a remote database server;
receiving from the remote server a response to the query; and
returning the response;
wherein the remote server is configured to execute the byte code to instantiate one or more instances of operator in the discrete set of streaming operators and use the data flow graph to manage the flow of data to and through the respective streaming operators and use data frames in an output queue of a final operator in the data flow graph to provide the result to the query.
20 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving at a client device a query associated with a database, wherein the query is expressed in one of a plurality of supported query languages;
generating a byte code representation of the query, including by decomposing the query into a discrete set of streaming operators defined over associated data frames, wherein the byte code includes code defining for each operator in the discrete set of streaming operators the processing to be performed by that operator and further embodies a data flow graph that defines a flow of data to and through the discrete set of streaming operators;
storing the byte code representation of the query
serializing the byte code and sending the serialized byte code to a remote database server;
receiving from the remote server a response to the query; and
returning the response;
wherein the remote server is configured to execute the byte code to instantiate one or more instances of operator in the discrete set of streaming operators and use the data flow graph to manage the flow of data to and through the respective streaming operators and use data frames in an output queue of a final operator in the data flow graph to provide the result to the query.