Open station as a stream analysis operator container
View Patent ↗An open station as a stream analysis operator container is disclosed. In an example, a method is stored as instructions on computer readable media and executable by a processor. The method includes packaging graph-structured stream transformations into a topology of a top-level dataflow process including stream transformation operators (STOs) on open stations. The method also includes sending a tuple from one of the STOs to every other STO in the topology subscribed to a data stream of the tuple.
1. A method of using an open station as a stream analysis operator container, the method is stored as instructions on a non-transitory computer readable media and executable by a processor for: packaging graph-structured stream transformations into a topology of a top-level dataflow process including stream transformation operators (STOs) on open stations, wherein the open stations each have a defined dataflow function having a higher level of abstraction with respect to the STOs contained in each open station such that the defined dataflow function of each open station encompass different types of specific STO application logic, facilitating the plugging in of specific types of STO application logic into the topology of the top-level dataflow process; and sending a tuple from one of the STOs to every other STO in the topology subscribed to a data stream of the tuple, and further comprising executing multiple STOs in parallel using a task property specifying a number of threads for running the multiple STOs.
2. The method of claim 1 , wherein the data stream is an unbounded sequence of tuples.
3. The method of claim 1 , further comprising executing an STO cycle-by-cycle to process the data stream chunk-by-chunk.
4. The method of claim 3 , wherein processing the data stream chunk-by-chunk is by cutting the data stream into a sequence of chunks, with each chunk representing a bounded data set on which an operation is defined, and after processing a chunk of the data stream, re-running the operation to process the next chunk of the data stream.
5. The method of claim 3 , wherein executing the STO cycle-by-cycle is as a Cycle-based STO (CSTO) to generate an unbounded sequence of results on each chunk of the data stream.
6. The method of claim 1 , further comprising grouping data streams to inform the topology how to send the tuple.
7. The method of claim 1 , wherein edges in the graph identify STOs subscribed to the data stream.
8. A data processing system with open station as a stream analysis operator container, comprising: a topology of graph-structured stream transformations representing a top-level dataflow process including stream transformation operators (STOs) on open stations, wherein the open stations each have a defined dataflow function having a higher level of abstraction with respect to the STOs contained in each open station such that the defined dataflow function of each open station encompass different types of specific STO application logic-facilitating the plugging in of specific types of STO application logic into the topology of the top-level dataflow process; and a processor for sending a tuple from one of the STOs to every other STO in the topology subscribed to a data stream of the tuple, and further comprising executing multiple STOs in parallel using data grouping characteristics indicating how results of predecessor STOs are grouped for sending to successor STOs.
9. The system of claim 8 , wherein characteristics of the STOs are properties of the corresponding station objects.
10. The system of claim 8 , wherein input/output characteristics of the STOs specify number of input tuples and output tuples the handled by the STOs.
11. The system of claim 8 , wherein the STOs support one input/one output, one input/multiple outputs, multiple inputs/one output, and multiple inputs/multiple outputs.
12. The system of claim 8 , wherein grouping characteristics of the STOs define how the topology sends tuples between STOs.
13. The system of claim 8 , wherein the grouping characteristics comprise a shuffle grouping which sends tuples to a random task to evenly distribute processing the tuples across all consecutive downstream tasks.
14. The system of claim 8 , wherein the grouping characteristics comprise a fields grouping which ensures tuples with same value go to a same task.
15. A data processing system with open station as a stream analysis operator container, comprising: a topology of graph-structured stream transformations into a topology of a top-level dataflow process including stream transformation operators (STOs) on open stations, wherein the open stations each have a defined dataflow function having a higher level of abstraction with respect to the STOs contained in each open station such that the defined dataflow function of each open station encompass different types of specific STO application logic, facilitating the plugging in of specific types of STO application logic into the topology of the top-level dataflow process; and a processor for sending a tuple from one of the STOs to every other STO in the topology subscribed to a data stream of the tuple, wherein invocation of the STOs follow a pattern for handling input and output, and further comprising executing multiple STOs in parallel using a task property specifying a number of threads for running the multiple STOs.
16. The system of claim 15 , further comprising a scalar STO invocation pattern.
17. The system of claim 15 , further comprising a table STO invocation pattern.
18. The system of claim 15 , further comprising a set-in/set-out STO invocation pattern.