Integrated native vectorized engine for computation
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
1. A system, comprising:
an interface configured to:
receive a query;
a processor configured to:
determine a set of nodes for the query;
determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second; engine, wherein the node of the set of nodes comprises the first engine node type in response to determining that the node is not able to be executed in the second engine;
generate a plan based at least in part on the set of nodes, comprising to:
determine whether a node of the set of nodes is a second engine node type; and
in response to a determination that the node of the set of nodes is the second engine node type:
add a transition node to the plan; and
add a first engine to second engine data converter and a contiguous group of second engine nodes to the transition node; and
execute the plan, comprising to:
determine whether a node of the plan is the transition node; and
in response to a determination that the node of the plan is the transition node:
convert data from a first engine data format to a second engine data format; and
execute the second engine node type; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the node of the set of nodes comprises the second engine node type in response to determining that the node is able to be executed in the second engine.
3. The system of claim 1 , wherein the node of the set of nodes comprises the second engine node type in response to determining that the node is able to be executed in the second engine and the node is in a contiguous sequence of nodes that all can executed in the second engine.
4. The system of claim 1 , wherein the node of the set of nodes comprises the first engine node type in response to not being in a contiguous sequence of nodes that all can be executed in the second engine.
5. The system of claim 1 , wherein generating the plan comprises generating instructions for executing the set of nodes.
6. The system of claim 5 , wherein executing the node of the set of nodes of the second engine node type comprises storing metrics data to a shared memory space.
7. The system of claim 6 , wherein metrics data is stored to the shared memory space by the first engine or by the second engine.
8. The system of claim 6 , wherein metrics data is retrieved from the shared memory space by the first engine.
9. The system of claim 5 , wherein executing the node of the set of nodes of the first engine node type after executing an other node of the second engine node type comprises translating data in memory from a second engine data format to a first engine data format.
10. The system of claim 5 , wherein executing the node of the set of nodes of the second engine node type after executing an other node of the second engine node type comprises transferring data from the other node to the node in a second engine data format.
11. The system of claim 5 , wherein executing the node of the set of nodes of the second engine node type comprises executing an expression using a first engine.
12. The system of claim 1 , wherein the node is executed using a first engine in response to the node having the first engine node type and is executed using the second engine in response to the node having the second engine node type.
13. The system of claim 1 , wherein a first engine or the second engine requests a memory allocation from a system memory manager or releases the memory allocation to the system memory manager.
14. The system of claim 1 , wherein the plan indicates cluster resources to execute a node of the set of nodes.
15. The system of claim 1 , wherein the plan indicates transfer of data between cluster resources prior to execution of a node of the set of nodes.
16. The system of claim 15 , wherein the data is translated after being transferred between cluster resources.
17. The system of claim 1 , wherein determining the node of the set of nodes further comprises determining whether the node comprises a third engine node type based at least in part on determining whether the node is able to be executed in a third engine.
18. A method comprising:
receiving a query;
determining, using a processor, a set of nodes for the query;
determining whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine, wherein the node of the set of nodes comprises the first engine node type in response to determining that the node is not able to be executed in the second engine;
generating a plan based at least in part on the set of nodes, comprising:
determining whether a node of the set of nodes is a second engine node type; and
in response to a determination that the node of the set of nodes is the second engine node type:
adding a transition node to the plan; and
adding a first engine to second engine data converter and a contiguous group of second engine nodes to the transition node; and
executing the plan, comprising:
determining whether a node of the plan is the transition node; and
in response to a determination that the node of the plan is the transition node:
converting data from a first engine data format to a second engine data format; and
executing the second engine node type.
19. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving a query;
determining, using a processor, a set of nodes for the query;
determining whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine, wherein the node of the set of nodes comprises the first engine node type in response to determining that the node is not able to be executed in the second engine;
generating a plan based at least in part on the set of nodes, comprising:
determining whether a node of the set of nodes is a second engine node type; and
in response to a determination that the node of the set of nodes is the second engine node type:
adding a transition node to the plan; and
adding a first engine to second ermine data converter and a contiguous group of second engine nodes to the transition node; and
executing the plan, comprising:
determining whether a node of the plan is the transition node; and
in response to a determination that the node of the plan is the transition node:
converting data from a first engine data format to a second engine data format; and
executing the second engine node type.