Parallel database query processing for non-uniform data sources via buffered access
View Patent ↗An apparatus, program product and method utilize a dynamically-populated query buffer to facilitate the handling of at least a portion of a database query in parallel. A query is implemented using at least first and second portions, where the second portion of the query is executed in parallel using a plurality of threads. The first portion of the query is executed to dynamically populate a query buffer with records from a data source, and the plurality of threads that execute the second portion of the query are specified to the query buffer so that the effective data source for the second portion of the query comprises the records that are dynamically populated into the query buffer.
1. A method of executing a database query, the method comprising:
(a) executing a first portion of a query to dynamically populate a query buffer with records from a data source; and
(b) executing a second portion of the query in parallel using a plurality of threads specified to the query buffer;
wherein the query buffer includes a plurality of entries, wherein executing the first portion of the query includes storing a record in an entry in the query buffer, and wherein executing the second portion of the database query includes, in each thread, retrieving an entry from the query buffer and executing the second portion of the query on a record on the retrieved entry.
2. The method of claim 1 , wherein executing the first portion of the query dynamically populates the query buffer with records from the data source that match a first query criterion, and wherein executing the second portion of the query includes selecting those records among those populated in the query buffer that match a second query criterion.
3. The method of claim 1 , wherein executing the first portion of the query is performed serially.
4. The method of claim 1 , wherein executing the first portion of the query is performed by a thread that is separate from the plurality of threads that execute the second portion of the query.
5. The method of claim 1 , wherein executing the first portion of the query is performed by a thread among the plurality of threads.
6. The method of claim 5 , wherein executing the first portion of the query is performed by different threads among the plurality of threads at different times.
7. The method of claim 6 , wherein executing the first portion of the query includes determining whether a thread among the plurality of threads has been drafted as a producer, and if so, serially executing the first portion of the query in the producer thread.
8. The method of claim 1 , wherein executing the second portion of the query consumes records from the query buffer in parallel with dynamically populating the query buffer.
9. The method of claim 1 , wherein each entry is configured to store a plurality of records.
10. The method of claim 1 , wherein executing the first portion of the query includes populating at least one entry with a uniform set of records.
11. The method of claim 1 , wherein executing the first portion of the query includes populating at least one entry with a non-uniform set of records.
12. The method of claim 1 , wherein executing the first and second portions of the query each include executing an attribute operation list associated with a node defined in a query object, the attribute operation list configured to manipulate at least one attribute described in an attribute descriptor array.
13. An apparatus, comprising:
(a) a memory within which is resident at least a portion of a database; and
(b) program code configured to execute a query on the database, the program code configured to execute a first portion of the query to dynamically populate a query buffer with records from the database, and to execute a second portion of the query in parallel using a plurality of threads specified to the query buffers;
wherein the query buffer includes a plurality of entries, wherein the program code is configured to execute the first portion of the query by storing a record in an entry in the query buffer, and wherein the program code is configured to execute the second portion of the database query by, in each thread, retrieving an entry from the query buffer and executing the second portion of the query on a record on the retrieved entry.
14. The apparatus of claim 13 , wherein the program code is configured to execute the first portion of the query to dynamically populate the query buffer with records from the database that match a first query criterion, and wherein the program code is configured to execute the second portion of the query by selecting those records among those populated in the query buffer that match a second query criterion.
15. The apparatus of claim 13 , wherein the program code is configured to serially execute the first portion of the query.
16. The apparatus of claim 13 , wherein the program code is configured to execute the first portion of the query in a thread that is separate from the plurality of threads that execute the second portion of the query.
17. The apparatus of claim 13 , wherein the program code is configured to execute the first portion of the query in a thread among the plurality of threads.
18. The apparatus of claim 17 , wherein the program code is configured to execute the first portion of the query in different threads among the plurality of threads at different times.
19. The apparatus of claim 18 , wherein the program code is configured to execute the first portion of the query by determining whether a thread among the plurality of threads has been drafted as a producer, and if so, serially executing the first portion of the query in the producer thread.
20. The apparatus of claim 13 , wherein the program code is configured to execute the second portion of the query to consume records from the query buffer in parallel with dynamically populating the query buffer.
21. The apparatus of claim 13 , wherein each entry is configured to store a plurality of records.
22. The apparatus of claim 13 , wherein the program code is configured to execute the first portion of the query by populating at least one entry with a uniform set of records.
23. The apparatus of claim 13 , wherein the program code is configured to execute the first portion of the query by populating at least one entry with a non-uniform set of records.
24. The apparatus of claim 13 , wherein the program code is configured to execute the first and second portions of the query each by executing an attribute operation list associated with a node defined in a query object, the attribute operation list configured to manipulate at least one attribute described in an attribute descriptor array.
25. A program product, comprising:
(a) program code configured to execute a database query, the program code configured to execute a first portion of the query to dynamically populate a query buffer with records from a data source, and to execute a second portion of the query in parallel using a plurality of threads specified to the query buffer; and
(b) a signal bearing medium bearing the program code;
wherein the query buffer includes a plurality of entries, wherein the program code is configured to execute the first portion of the query by storing a record in an entry in the query buffer, and wherein the program code is configured to execute the second portion of the database query by, in each thread, retrieving an entry from the query buffer and executing the second portion of the query on a record on the retrieved entry.
26. The program product of claim 25 , wherein the signal bearing medium includes at least one of a transmission medium and a recordable medium.