Integrated native vectorized engine for computation
A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.
1. A method, comprising:
receiving, at a cluster resource, a database query specified using a declarative database query language, the cluster resource including a first set of execution engines configured as a first engine type and a second set of execution engines configured as a second engine type, the second set of execution engines configured to perform one or more vectorized operations;
generating an execution plan representing the database query as a graph data structure including one or more nodes representing one or more operations to be performed for the database query;
identifying at least a subset of nodes in the graph data structure that are supported by execution engines of the second engine type;
generating a second execution plan for the second set of execution engines by transforming the subset of identified nodes to a sequence of vectorized operations that are supported by the second set of execution engines; and
executing the sequence of vectorized operations for the execution plan on the second set of execution engines based on one or more inputs.
2. The method of claim 1 , each vectorized operation in the one or more vectorized operations configured to receive one or more inputs in columnar format and generate one or more outputs in columnar format.
3. The method of claim 2 , the one or more inputs in columnar format, and elements of an input in the columnar format stored contiguously in memory.
4. The method of claim 1 , the cluster resource further configured with a memory pool that is shared by the first set of execution engines and the second set of execution engines.
5. The method of claim 1 , comprising:
identifying at least another subset of nodes in the graph data structure that are supported by execution engines of the first engine type;
generating code based on the execution plan for the another subset of nodes; and
executing the code on the first set of execution engines to generate one or more outputs for the another subset of nodes.
6. The method of claim 5 , the another subset of nodes placed before the subset of nodes and the one or more outputs generated in a row-wise format, and the method further comprising:
converting the one or more outputs in the row-wise format to a columnar format; and
providing the converted outputs to the second set of execution engines as the one or more inputs.
7. The method of claim 1 , the one or more vectorized operations including at least one of a scan operation, a filter operation, a project operation, a grouping aggregate operation.
8. A non-transitory computer-readable storage medium comprising stored instructions executable by a processor, the instructions when executed causing the processor to:
receive a database query specified using a declarative database query language;
access a cluster resource including a first set of execution engines configured as a first engine type and a second set of execution engines configured as a second engine type, the second set of execution engines configured to perform one or more vectorized operations;
generate an execution plan representing the database query as a graph data structure including one or more nodes representing one or more operations to be performed for the database query;
identify at least a subset of nodes in the graph data structure that are supported by execution engines of the second engine type;
generate a second execution plan for the second set of execution engines by transforming the subset of identified nodes to a sequence of vectorized operations that are supported by the second set of execution engines; and
execute the sequence of vectorized operations for the execution plan on the second set of execution engines based on one or more inputs.
9. The non-transitory computer-readable storage medium of claim 8 , each vectorized operation in the one or more vectorized operations configured to receive one or more inputs in columnar format and generate one or more outputs in columnar format.
10. The non-transitory computer-readable storage medium of claim 9 , the one or more inputs in columnar format, and elements of an input in the columnar format stored contiguously in memory.
11. The non-transitory computer-readable storage medium of claim 8 , the cluster resource further configured with a memory pool that is shared by the first set of execution engines and the second set of execution engines.
12. The non-transitory computer-readable storage medium of claim 8 , the instructions when executed causing the processor to:
identify at least another subset of nodes in the graph data structure that are supported by execution engines of the first engine type;
generate code based on the execution plan for the another subset of nodes; and
execute the code on the first set of execution engines to generate one or more outputs for the another subset of nodes.
13. The non-transitory computer-readable storage medium of claim 12 , the another subset of nodes placed before the subset of nodes and the one or more outputs generated in a row-wise format, and the instructions when executed causing the processor to:
convert the one or more outputs in the row-wise format to a columnar format; and
provide the converted outputs to the second set of execution engines as the one or more inputs.
14. The non-transitory computer-readable storage medium of claim 8 , the one or more vectorized operations including at least one of a scan operation, a filter operation, a project operation, a grouping aggregate operation.
15. A computer system, comprising:
a computer processor; and
a memory comprising stored instructions that when executed by the computer processor causes the computer system to:
receive, at a cluster resource, a database query specified using a declarative database query language, the cluster resource including a first set of execution engines configured as a first engine type and a second set of execution engines configured as a second engine type, the second set of execution engines configured to perform one or more vectorized operations;
generate an execution plan representing the database query as a graph data structure including one or more nodes representing one or more operations to be performed for the database query;
identify at least a subset of nodes in the graph data structure that are supported by execution engines of the second engine type;
generate a second execution plan for the second set of execution engines through transformation of the subset of identified nodes to a sequence of vectorized operations that are supported by the second set of execution engines; and
execute the sequence of vectorized operations for the execution plan on the second set of execution engines based on one or more inputs.
16. The computer system of claim 15 , further comprising stored instructions to receive, for each vectorized operation in the one or more vectorized operations, one or more inputs in columnar format and generate, one or more outputs in columnar format.
17. The computer system of claim 16 , the one or more inputs in columnar format, and elements of an input in the columnar format stored contiguously in memory.
18. The computer system of claim 15 , the cluster resource further configured with a memory pool that is shared by the first set of execution engines and the second set of execution engines.
19. The computer system of claim 15 , the stored instructions further comprising stored instructions that when executed by the processor cause the computer system to:
identify at least another subset of nodes in the graph data structure that are supported by execution engines of the first engine type;
generate code based on the execution plan for the another subset of nodes; and
execute the code on the first set of execution engines to generate one or more outputs for the another subset of nodes.
20. The computer system of claim 19 , wherein the another subset of nodes is placed before the subset of nodes and the one or more outputs are generated in a row-wise format, and the stored instructions further comprising stored instructions that when executed by the processor cause the computer system to:
convert the one or more outputs in the row-wise format to a columnar format; and
provide the converted outputs to the second set of execution engines as the one or more inputs.