Execution environment for data transformation applications
The execution environment provides for scalability where components will execute in parallel and exploit various patterns of parallelism. Dataflow applications are represented by reusable dataflow graphs called map components, while the executable version is called a prepared map. Using runtime properties the prepared map is executed in parallel with a thread allocated to each map process. The execution environment not only monitors threads, detects and corrects deadlocks, logs and controls program exceptions, but also data input and output ports of the map components are processed in parallel to take advantage of data partitioning schemes. Port implementation supports multi-state null value tokens to more accurately report exceptions. Data tokens are batched to minimize synchronization and transportation overhead and thread contention.
1. A method for executing a dataflow application comprising:
providing a dataflow application comprising a plurality of map components and data ports, wherein a number of the map components are linked between data ports, wherein the map components each comprise one or more map processes, and wherein some of the map components comprise composite components having a plurality of processes;
allocating a processing thread to each map process; and
executing each map process on a separate thread.
2. The method of claim 1 , including generating a prepared map of the dataflow application and assigning runtime properties to the prepared map prior to execution.
3. The method of claim 1 , certain map components including input ports and output ports and the executing step including reading data tokens from respective input ports and writing data tokens to respective output ports.
4. The method of claim 1 , including choosing port implementations to a specific data type prior to said executing step.
5. The method of claim 1 , some of the ports being hierarchical comprising a plurality of hierarchical ports and the data being partitioned for parallel processing.
6. The method of claim 1 , including transporting data tokens among the map processes.
7. The method of claim 6 , including batching the data tokens to regulate a length of time a map process may execute without synchronization.
8. The method of claim 1 , including ports associated with each map process for representing and transporting multi-state null value tokens.
9. The method of claim 8 , the null value tokens including an error null.