Real time data replication for query execution in a massively parallel computer
View Patent ↗Embodiments of the invention may be used to increase query processing parallelism of an in-memory database stored on a parallel computing system. A group of compute nodes each store a portion of data as part of the in-memory database. Further, a pool of compute nodes may be reserved to create copies of data from the compute nodes of the in-memory database as part of query processing. When a query is received for execution, the query may be evaluated to determine whether portions of in-memory should be duplicated to allow multiple elements of the query (e.g., multiple query predicates) to be evaluated in parallel.
1. A method for processing a database query, comprising:
receiving a query of a database, wherein the database is stored on a plurality of compute nodes provided by a parallel computing system, wherein each compute node includes at least a processor and a memory;
identifying two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes;
copying the data records stored on the first compute node to a second compute node, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes provided by the parallel computing system;
transmitting a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and
receiving the respective query results from the first compute node and the second compute node.
2. The method of claim 1 , further comprising:
reconciling the results received from the first compute node and the second compute node; and
returning the reconciled results.
3. The method of claim 2 , wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.
4. The method of claim 1 , wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.
5. The method of claim 1 , wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.
6. The method of claim 1 , wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.
7. A computer readable storage medium containing a program which, when executed, performs an operation, comprising:
receiving a query of a database, wherein the database is stored on a plurality of compute nodes provided by a parallel computing system;
identifying two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes;
copying the data records stored on the first compute node to a second compute node, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes provided by the parallel computing system;
transmitting a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and
receiving the respective query results from the first compute node and the second compute node.
8. The computer readable storage medium of claim 7 , wherein the operation further comprises:
reconciling the results received from the first compute node and the second compute node; and
returning the reconciled results.
9. The computer readable storage medium of claim 8 , wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.
10. The computer readable storage medium of claim 9 , wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.
11. The computer readable storage medium of claim 7 , wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.
12. The computer readable storage medium of claim 7 , wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.
13. A parallel computing system, comprising:
a plurality of compute nodes, each having at least a processor and a memory, wherein each of the plurality of compute nodes stores a portion of an in-memory database; and
a master node having at least a processor and a memory and a database controller program configured to:
receive a query of the in-memory database,
identify two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes,
copy the data records stored on the first compute node to a second compute node, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes used to store the in-memory database,
transmit a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and
receive the respective query results from the first compute node and the second compute node.
14. The parallel computing system of claim 13 , further comprising:
reconciling the results received from the first compute node and the second compute node; and
returning the reconciled results.
15. The parallel computing system of claim 14 , wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.
16. The parallel computing system of claim 13 , wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.
17. The parallel computing system of claim 13 , wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.
18. The parallel computing system of claim 13 , wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.