System and method for data transformation using dataflow graphs
A system and method for managing data, such as in a data warehousing, analysis, or similar applications, where dataflow graphs are expressed as reusable map components, at least some of which are selected from a library of components, and map components are assembled to create an integrated dataflow application. Composite map components encapsulate a dataflow pattern using other maps as subcomponents. Ports are used as link points to assemble map components and are hierarchical and composite allowing ports to contain other ports. The dataflow application may be executed in a parallel processing environment by recognizing the linked data processes within the map components and assigning threads to the linked data processes.
1. A method of transforming data in a multi-thread parallel processing environment comprising a single operating system and one or more processors wherein:
map components are assembled visually into an integrated dataflow application using compiler tools linking map processes within map components, wherein the map components are linked using ports;
the integrated dataflow application is executed in parallel by recognizing the linked map processes within the map components and allocating a separate thread to each of a plurality of linked map processes; and
each of said plurality of linked map process is executed on its allocated thread substantially in parallel as part of a single process, and said data resides in memory accessible to each of said plurality of linked map process.
2. The method of claim 1 , wherein said plurality of linked map processes read data tokens from input ports and write data tokens to output ports.
3. The method of claim 1 , communication between said processes executing in parallel being managed by an executor separate from the operating system.
4. The method of claim 1 , at least one map component being composite, thereby encapsulating a particular dataflow pattern.