Query engine virtualization
Techniques for query engine virtualization are described. A query engine virtualization layer between clients and multiple query engines acts as a proxy to execute queries on behalf of the clients using ones of the multiple query engines selected for each query that are comparatively the best-suited to execute the query. The selection may be based on the workload pattern the query is issuing on the underlying data.
1. A computer-implemented method comprising:
receiving, at an endpoint of a service provider network, a first message originated by an electronic device of a user indicating a request to execute a query against a set of data;
sending a plurality of requests to a corresponding plurality of query engines implemented within the service provider network, each of the plurality of requests including the query or a rewritten query based on the query;
receiving a plurality of responses from the corresponding plurality of query engines, each of the plurality of responses indicating a predicted execution characteristic generated by the corresponding query engine based on the query or the rewritten query;
selecting, based on the plurality of responses, a query engine from the plurality of query engines having a best predicted performance characteristic to execute the query or the rewritten query;
transmitting a second message to the query engine to execute the query or the rewritten query;
receiving a third message from the query engine including a query result; and
sending a fourth message including the query result to the electronic device.
2. The computer-implemented method of claim 1 , wherein selecting the query engine based on the plurality of responses comprises:
identifying a minimum predicted execution time value from a plurality of predicted execution time values received in the plurality of responses, wherein the response of the plurality of responses sent by the selected query engine includes the minimum predicted execution time value.
3. The computer-implemented method of claim 2 , further comprising:
transmitting a fifth message destined to an electronic device of a user, the fifth message identifying at least the query engine and the minimum predicted execution time; and
receiving a sixth message originated by the electronic device indicating that the query or the rewritten query is to be executed using the query engine.
4. A computer-implemented method comprising:
receiving a first message indicating a request to execute a query against a set of data;
sending requests to a plurality of query engines, each of the requests including the query;
receiving a plurality of responses from the plurality of query engines, the responses indicating predicted performance characteristics generated by the plurality of query engines based on the query;
selecting, based at least in part on the responses, a query engine from the plurality of query engines having a best predicted performance characteristic corresponding to a workload needed to execute the query;
transmitting a second message to the query engine, the second message including the query;
receiving a third message from the query engine that includes a query result; and
sending a fourth message including the query result.
5. The computer-implemented method of claim 4 , wherein the fourth message is sent to a device from which the first message originated.
6. The computer-implemented method of claim 4 , wherein the first message is received at a query engine proxy of a query engine virtualization service implemented within a service provider network.
7. The computer-implemented method of claim 4 , wherein each of the predicted performance characteristics comprises a predicted execution time value, and wherein selecting the query engine from the plurality of query engines further comprises:
identifying the query engine from the plurality of query engines corresponding to the response of the plurality of responses having a minimum predicted execution time value of the plurality of predicted execution time values included in the plurality of responses.
8. The computer-implemented method of claim 7 , wherein selecting the query engine from the plurality of query engines further comprises:
transmitting a fifth message destined to an electronic device of a user, the fifth message identifying at least the query engine and the minimum predicted execution time; and
receiving a sixth message originated by the electronic device indicating a request to execute the query using the query engine.
9. The computer-implemented method of claim 4 , wherein selecting the query engine from the plurality of query engines comprises:
determining that the third message was received prior to receipt of any other message from any other query engine that includes a query result; and
identifying the query engine corresponding to the third message.
10. The computer-implemented method of claim 4 , wherein at least one of the predicted performance characteristics of the plurality of responses is one of:
a predicted execution time;
a predicted amount of data to be scanned to execute the query;
one or more operations to be performed as part of executing the query that have a highest resource cost; or
a predicted cost to execute the query.
11. The computer-implemented method of claim 4 , wherein selecting the query engine from the plurality of query engines comprises:
running a model, based at least on either the query or an identifier of an account associated with the first request, to identify the query engine.
12. The computer-implemented method of claim 4 , further comprising:
generating a rewritten query based on the query, wherein the rewritten query is compliant with a first query language definition, the query is compliant with a second query language definition, and the query and the rewritten query are both equivalent,
wherein the second message comprises the rewritten query.
13. The computer-implemented method of claim 4 , wherein:
the first message is received at an endpoint of a service provider network; and
each of the plurality of query engines is implemented within the service provider network.
14. The computer-implemented method of claim 4 , wherein the query involves data stored in a distributed manner at a plurality of different storage locations.
15. The computer-implemented method of claim 4 , wherein at least one of the plurality of query engines is a data warehouse service, an interactive query service, or a distributed processing service.
16. A system comprising:
a plurality of query engines implemented by a first one or more electronic devices having at least a first processor coupled to first memory; and
a query engine virtualization service implemented by a second one or more electronic devices having at least a second processor coupled to second memory, the query engine virtualization service including instructions stored in the second memory that upon execution by the at least a second processor cause the query engine virtualization service to:
receive a request to execute a query against a set of data;
send requests to the plurality of query engines, each of the requests including the query;
receive responses from the plurality of query engines, the responses indicating predicted performance characteristics generated by the plurality of query engines based on the query;
select, based on a query the responses, a query engine from the plurality of query engines that has a best predicted performance characteristic corresponding to a workload needed to execute the query;
transmit a first message to the query engine, the first message indicating the request to execute the query or a rewritten query generated based on the query;
receive a second message from the query engine including a query result; and
send a third message including the query result.
17. The system of claim 16 , wherein to select the query engine, the query engine virtualization service includes further instructions stored in the second memory that upon execution by the at least a second processor further cause the query engine virtualization service to:
identify a minimum predicted execution time value from a plurality of predicted execution time values corresponding to the plurality of query engines, wherein each of the plurality of predicted execution time values was generated by the corresponding query engine based on the query.
18. The system of claim 17 , wherein at least one of the plurality of query engines is a data warehouse service, an interactive query service, or a distributed processing service.
19. The system of claim 16 , wherein the query engine virtualization service includes further instructions stored in the second memory that upon execution by the at least a second processor further cause the query engine virtualization service to:
generate a rewritten query based on the query, wherein the rewritten query is compliant with a first query language definition, the query is compliant with a second query language definition, and the query and the rewritten query are both equivalent,
wherein the rewritten query is included in the first message.
20. The system of claim 16 , wherein the query engine virtualization service includes further instructions stored in the second memory that upon execution by the at least a second processor further cause the query engine virtualization service to:
transmit a fourth message destined to an electronic device of a user, the third message identifying at least the query engine and a minimum predicted execution time; and
receive a fifth message originated by the electronic device indicating a request to execute the query using the query engine.