Data parallelism and parallel operations in stream processing
A stream processing platform that provides fast execution of stream processing applications within a safe runtime environment. The platform includes a stream compiler that converts a representation of a stream processing application into executable program modules for a safe environment. The platform allows users to specify aspects of the program that contribute to generation of modules that execute as intended. A user may specify aspects to control a type of implementation for loops, order of execution for parallel paths, whether multiple instances of an operation can be performed in parallel or whether certain operations should be executed in separate threads. In addition, the stream compiler may generate executable modules in a way that cause a safe runtime environment to allocate memory or otherwise operate efficiently.
1. A method of generating machine-executable instructions for processing at least one stream of data values, the method comprising:
receiving:
an indication of at least one first operation to be performed on the at least one stream, the at least one stream is defined by a schema and comprises a plurality of tuples; and
an indication of at least one second operation to be performed on the at least one stream;
generating a first set of machine-executable instructions to perform the at least one first operation; and
generating a second set of machine-executable instructions to perform the at least one second operation, wherein:
the first set and the second set are generated based on user input such that, when the user input comprises an indication that the at least one second operation is to execute in a separate thread, the first set and the second set are encoded to, when executed, be separated by a thread boundary and, when the user input does not comprise the indication, the first set and the second set are encoded to be executed in a same thread.
2. The method of claim 1 , wherein the thread boundary also defines a process boundary.
3. The method of claim 1 , wherein the thread boundary also defines a machine boundary.
4. The method of claim 1 , wherein receiving the indication of at least one first operation comprises receiving the indication through a graphical user interface.
5. The method of claim 1 , wherein:
generating the first set of machine-executable instructions comprises producing a first compiled module; and
generating the second set of machine-executable instructions comprises producing a second compiled module.
6. The method of claim 1 , wherein:
generating the first set of machine-executable instructions comprises producing a plurality of modules adapted for execution under control of a scheduler.
7. The method of claim 1 , wherein:
generating the first set of machine-executable instructions comprises producing a plurality of modules adapted for execution by a virtual machine.
8. The method of claim 1 , wherein:
generating the first set of machine-executable instructions comprises producing a plurality of modules adapted for execution by a physical machine.
9. A method of generating machine-executable instructions for processing at least one stream of data values, the method comprising:
providing a user interface configured to receive from a user a representation of processing to be performed on the at least one stream;
receiving, via the user interface, a first indication, the first indication indicating a plurality of operations to be performed on the at least one stream;
receiving, via the user interface, based on user input, a second indication, the second indication indicating that each of the plurality of operations is to be performed on the at least one stream in a respective path of a plurality of parallel paths and that each of the plurality of operations is to be performed on the data values in the path of the plurality of parallel paths in an indicated order; and
based on the first indication, the second indication and the indicated order, generating machine-executable instructions to perform the plurality of operations in each of the plurality of parallel paths in the indicated order, wherein the machine-executable instructions are generated based on user input such that, when the user input comprises a third indication that a first operation of the plurality of operations is to execute in a separate thread from a second operation of the plurality of operations, the machine-executable instructions are encoded such that, when executed, the first operation and the second operation are separated by a thread boundary and, when user input does not comprise the third indication, the machine-executable instructions are encoded such that the first operation and the second operation execute in a same thread.
10. The method of claim 9 , wherein the stream comprises a plurality of sequential values and the method further comprises generating machine-executable instructions to direct each of the sequential values of the plurality of sequential values to a path of the plurality of paths.
11. The method of claim 9 , wherein generating machine-executable instructions comprises generating machine-executable instructions to execute a first portion of the plurality of threads on a first processor and to execute a second portion of the plurality of threads on a second processor.
12. The method of claim 9 , wherein receiving the second indication comprises receiving a numeric indication of a number of parallel paths in the plurality of paths.
13. The method of claim 9 , wherein:
generating machine-executable instructions to perform the plurality of operations in each of the plurality of parallel paths comprises generating multiple instances of a module executing the plurality of operations.
14. The method of claim 9 , wherein:
the representation of processing comprising the plurality of operations to be performed on the at least one input stream comprises a representation to produce at least one output stream, the representation of processing including a plurality of operators and data flow between the operators; and
receiving the second indication indicating that each of the plurality of operations is to be performed on the at least one stream in a respective path of the plurality of parallel paths comprises receiving through the user interface an identification of an operator of the plurality of operators.
15. A method of generating machine-executable instructions for processing a stream of data values, the method comprising:
receiving a representation of processing to be performed on the stream, the representation comprising an operation, and the stream is defined by a schema and comprises a plurality of tuples;
determining a manner of execution of the operation by determining whether an indication indicating that the operation is to be concurrently performed on the stream in a plurality of parallel paths such that each path of the plurality of parallel paths performs the operation in a separate thread of a plurality of threads is received; and
generating machine-executable instructions based on the determined manner of execution such that:
the generated machine-executable instructions perform the operation in each of the plurality of parallel paths in a separate thread of the plurality of threads when the indication is received; and
the generated machine-executable instructions perform the operation such that at least two of the plurality of parallel paths are executed in a same thread when the indication is not received.
16. The method of claim 15 , wherein the stream comprises a plurality of sequential values and the method further comprises generating machine-executable instructions to direct each of the sequential values of the plurality of sequential values to a path of the plurality of parallel paths.
17. The method of claim 15 , wherein generating machine-executable instructions comprises generating machine-executable instructions to execute a first portion of the plurality of threads on a first processor and to execute a second portion of the plurality of threads on a second processor.
18. The method of claim 15 , wherein receiving the indication that the operation is to be performed in the plurality of parallel paths comprises receiving a numeric indication of a number of parallel paths in the plurality of paths.
19. The method of claim 15 , wherein generating machine-executable instructions to perform the operation in each of the plurality of parallel paths comprises generating multiple instances of a module executing the operation.
20. The method of claim 15 , wherein:
the representation of processing comprising the operation to be performed on the stream comprises a representation to produce at least one output stream, the representation of processing including a plurality of operators and data flow between the operators; and
receiving the indication that the operation is to be performed on the stream in the plurality of parallel paths comprises receiving an identification of an operator of the plurality of operators.