Progressive query computation using streaming architectures
The described implementations relate to processing of electronic data. One implementation is manifest as a technique that can include obtaining a relational query that references one or more data items and associating progress intervals with the data items. The technique can also include converting the relational query into a corresponding streaming query, and providing the streaming query and the data items with the progress intervals to a stream engine that produces incremental results of the query. For example, the progress intervals can be based on row numbers of a relational database table. The progress intervals can be used to define event lifetimes of streaming events that are provided as inputs to the stream engine.
1. A method comprising:
obtaining a query that references relational data items from a relational data table;
adapting the relational data items for processing by a stream engine to obtain streaming events, wherein the adapting the relational data items for processing by the stream engine comprises associating different progress intervals with different relational data items;
processing the streaming events with the stream engine to obtain incremental results; and
outputting the incremental results in response to the query.
2. The method of claim 1 , further comprising:
displaying a graphical user interface with a user interface element for configuring a number of streaming events that contribute to each of the incremental results; and
adjusting a number of the streaming events that share the same progress interval based at least on input directed to the user interface element.
3. The method of claim 2 , wherein the relational data items lack explicit temporal data.
4. The method of claim 3 , further comprising:
outputting at least some of the incremental results before all of the relational data items from the relational data table have been processed.
5. The method of claim 4 , further comprising:
configuring the different progress intervals to cause different streaming events to begin at different times and to end concurrently,
wherein the configuring causes all of the relational data items to contribute to a final result produced by the stream engine.
6. The method of claim 5 , wherein the different progress intervals are configured to group multiple rows of relational data items together in individual progress intervals.
7. A system comprising:
one or more processors; and
one or more computer-readable storage media comprising instructions which, when executed by the one or more processors, cause the one or more processors to:
obtain relational data items that lack explicit temporal ordering;
derive streaming events from the relational data items, the streaming events having explicit temporal orderingbeing derived by associating different progress intervals with different relational data items;
execute a stream engine that performs incremental processing on the streaming events derived from the relational data items to obtain incremental results; and
output the incremental results.
8. The system of claim 7 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
send the incremental results over a computer network to a requesting device.
9. The system of claim 8 , wherein the incremental processing performed on the streaming events includes streaming selection operations, streaming projection operations, and streaming join operations.
10. The system of claim 7 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
derive the streaming events by associating monotonically-increasing values with the relational data items.
11. The system of claim 7 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
use row identifiers of the relational data items to derive the different progress intervals.
12. The system of claim 7 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
partition the relational data items using a particular attribute of the relational data items; and
assign the particular attribute of the relational data items to corresponding streaming events as a progress attribute.
13. The system of claim 12 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
manipulate the progress attribute of the streaming events to concurrently expire groups of relational data items that share a given value for the particular attribute.
14. The system of claim 13 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
manipulate another progress attribute of the incremental results so that the incremental results do not expire with the groups of relational data items.
15. A hardware computer-readable storage media storing instructions which, when executed by one or more hardware processors, cause the one or more hardware processors to perform acts comprising:
obtaining data items that lack explicit temporal ordering;
obtaining a query that references the data items;
processing the data items to obtain streaming events by associating different progress intervals with different relational data items;
executing a stream engine that performs incremental processing on the streaming events derived from the data items to obtain incremental results; and
outputting the incremental results in response to the query.
16. The hardware computer-readable storage media of claim 15 , wherein the data items are relational data items from a relational data table.
17. The hardware computer-readable storage media of claim 16 , the acts further comprising:
reading the data items in sequential order from corresponding physical locations on a storage device into main memory; and
performing the incremental processing with the stream engine entirely in main memory.
18. The hardware computer-readable storage media of claim 17 , the acts further comprising:
manipulating the different progress intervals to ensure that the incremental processing is performed entirely in main memory.
19. The hardware computer-readable storage media of claim 15 , the acts further comprising:
displaying a graphical user interface comprising a control element for configuring a specified number of data items to include in respective data groups; and
configuring the streaming events so that the specified number of data items are processed together within respective increments of the incremental processing.
20. The hardware computer-readable storage media of claim 19 , the acts further comprising:
updating the graphical user interface to reflect the incremental results as the incremental results are produced by the stream engine.