Stream processor with compiled programs
A stream processing platform that provides fast execution of stream processing applications within a safe runtime environment. The platform includes a stream compiler that converts a representation of a stream processing application into executable program modules for a safe environment. The platform allows users to specify aspects of the program that contribute to generation of modules that execute as intended. A user may specify aspects to control a type of implementation for loops, order of execution for parallel paths, whether multiple instances of an operation can be performed in parallel or whether certain operations should be executed in separate threads. In addition, the stream compiler may generate executable modules in a way that cause a safe runtime environment to allocate memory or otherwise operate efficiently.
1. A method of generating machine-executable instructions for processing a stream of values, the method comprising:
generating machine-executable instructions to be executed in processing inputs of at least one input stream to generate outputs of at least one output stream, the machine-executable instructions defining at least one data path and each data path comprising at least one operation, wherein generating the machine-executable instructions comprises generating the instructions such that executing the machine-executable instructions to process each input of the at least one input stream comprises executing operations of the at least one data path in a same deterministic order for each input, the deterministic order incorporating each of the operations of the at least one data path, and wherein generating the machine-executable instructions comprises:
receiving a representation of processing to be performed on the at least one input stream to produce the at least one output stream, the representation including a plurality of operators and a data flow, wherein at least some of the plurality of operators are arranged in the data flow in at least one loop, wherein the data flow includes the at least one data path and the plurality of operators correspond to the operations of the at least one data path that are to be executed in the deterministic order for each input of the at least one input stream;
receiving a user input indicating at least one implementation type, corresponding to each of the at least one loop, that is to be used in compiling instructions for each of the at least one loop and in determining the deterministic order in which the plurality of operators, including operators of the at least one loop, will be executed for each input of the at least one input stream; and
compiling the representation into the machine-executable instructions, the compiling comprising at least (i) forming an intermediate representation of the plurality of operators, the data flow including the plurality of operators, and the at least one loop each having the corresponding implementation type indicated by the user input, (ii) forming an executable code model for stream processing based on the intermediate representation, and (iii) forming the machine-executable instructions based on the executable code model, wherein the executable code model defines the deterministic order in which the operations of the data flow will be executed for each input of the at least one input stream and forming the machine-executable instructions based on the executable code model comprises generating a sequence of instructions corresponding to the operations of one of the at least one data path, wherein the sequence specifies an order of execution of instructions of the sequence that matches the same deterministic order such that execution of the machine-executable instructions for each input of the at least one input data stream comprises executing the sequence of instructions corresponding to the operations in the same deterministic order, and wherein the sequence of instructions defines instructions for recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a first type and defines instructions for non-recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a second type.
2. The method of claim 1 , wherein receiving a representation comprises receiving the representation through a graphical user interface.
3. The method of claim 1 , wherein compiling the representation comprises:
generating a set of ordered statements in a source code for a computer programming language;
generating an executable representation of the ordered statements.
4. The method of claim 3 , wherein:
generating a set of ordered statements comprises generating a JAVA program; and
generating an executable representation comprises compiling the set of ordered statements in a JAVA compiler.
5. The method of claim 1 , wherein compiling the representation into machine-executable instructions comprises:
compiling the representation into machine-executable instructions that, in response to application of the input of an input data stream, perform the operations on the input of the input data stream in the same deterministic order, the operations producing a value of an output stream of the at least one output stream.
6. The method of claim 1 , wherein compiling the representation into machine-executable instructions comprises operating a plurality of code generators, each code generator associated with an operator of the plurality of operators.
7. The method of claim 1 , wherein the first type of loop is a recursive type and the second type of loop is a non-recursive type.
8. The method of claim 7 , wherein each of the at least one loop is one of the recursive type or the non-recursive type.
9. At least one non-transitory computer-readable storage medium having encoded thereon executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method of generating machine-executable instructions for processing a stream of values, the method comprising:
generating machine-executable instructions to be executed in processing inputs of at least one input stream to generate outputs of at least one output stream, the machine-executable instructions defining at least one data path and each data path comprising at least one operation, wherein generating the machine-executable instructions comprises generating the instructions such that executing the machine-executable instructions to process each input of the at least one input stream comprises executing operations of the at least one data path in a same deterministic order for each input, the deterministic order incorporating each of the operations of the at least one data path, and wherein generating the machine-executable instructions comprises:
receiving a representation of processing to be performed on the at least one input stream to produce the at least one output stream, the representation including a plurality of operators and a data flow, wherein at least some of the plurality of operators are arranged in the data flow in at least one loop, wherein the data flow includes the at least one data path and the plurality of operators correspond to the operations of the at least one data path that are to be executed in the deterministic order for each input of the at least one input stream;
receiving a user input indicating at least one implementation type, corresponding to each of the at least one loop, that is to be used in compiling instructions for each of the at least one loop and in determining the deterministic order in which the plurality of operators, including operators of the at least one loop, will be executed for each input of the at least one input stream, wherein each implementation type of the at least one implementation type is a recursive type or a non-recursive type; and
compiling the representation into the machine-executable instructions, the compiling comprising at least (i) forming an intermediate representation of the plurality of operators, the data flow including the plurality of operators, and the at least one loop each having the corresponding implementation type indicated by the user input, (ii) forming an executable code model for stream processing based on the intermediate representation, and (iii) forming the machine-executable instructions based on the executable code model, wherein the executable code model defines the deterministic order in which the operations of the data flow will be executed for each input of the at least one input stream and forming the machine-executable instructions based on the executable code model comprises generating a sequence of instructions corresponding to the operations of one of the at least one data path, wherein the sequence specifies an order of execution of instructions of the sequence that matches the same deterministic order such that execution of the machine-executable instructions for each input of the at least one input data stream comprises executing the sequence of instructions corresponding to the operations in the same deterministic order, and
wherein the sequence of instructions defines instructions for recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a recursive type and defines instructions for non-recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a non-recursive type.
10. An apparatus comprising:
at least one processor; and
at least one computer-readable storage medium having encoded thereon executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method of generating machine-executable instructions for processing a stream of values, the method comprising:
generating machine-executable instructions to be executed in processing inputs of at least one input stream to generate outputs of at least one output stream, the machine-executable instructions defining at least one data path and each data path comprising at least one operation, wherein generating the machine-executable instructions comprises generating the instructions such that executing the machine-executable instructions to process each input of the at least one input stream comprises executing operations of the at least one data path in a same deterministic order for each input, the deterministic order incorporating each of the operations of the at least one data path, and wherein generating the machine-executable instructions comprises:
receiving a representation of processing to be performed on the at least one input stream to produce the at least one output stream, the representation including a plurality of operators and a data flow, wherein at least some of the plurality of operators are arranged in the data flow in at least one loop, wherein the data flow includes the at least one data path and the plurality of operators correspond to the operations of the at least one data path that are to be executed in the deterministic order for each input of the at least one input stream;
receiving a user input indicating at least one implementation type, corresponding to each of the at least one loop, that is to be used in compiling instructions for each of the at least one loop and in determining the deterministic order in which the plurality of operators, including operators of the at least one loop, will be executed for each input of the at least one input stream, wherein each implementation type of the at least one implementation type is a recursive type or a non-recursive type; and
compiling the representation into the machine-executable instructions, the compiling comprising at least (i) forming an intermediate representation of the plurality of operators, the data flow including the plurality of operators, and the at least one loop each having the corresponding implementation type indicated by the user input, (ii) forming an executable code model for stream processing based on the intermediate representation, and (iii) forming the machine-executable instructions based on the executable code model, wherein the executable code model defines the deterministic order in which the operations of the data flow will be executed for each input of the at least one input stream and forming the machine-executable instructions based on the executable code model comprises generating a sequence of instructions corresponding to the operations of one of the at least one data path, wherein the sequence specifies an order of execution of instructions of the sequence that matches the same deterministic order such that execution of the machine-executable instructions for each input of the at least one input data stream comprises executing the sequence of instructions corresponding to the operations in the same deterministic order, and wherein the sequence of instructions defines instructions for recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a recursive type and defines instructions for non-recursively executing instructions corresponding to operators of a loop of the at least one loop when a corresponding implementation type of the loop is a non-recursive type.